Saved Search vs. Open Search Server Error in Query

[gearmstrong]

Good day Splunkers,

Today doing an audit of my Alerts, I opened one in "Open Search" and immediately got "Server Error" upon trying to run it.  Checked Scheduler Logs and Alert is Successfully running and issuing email as action.  It is a very lengthy beast for "PowerShell Command Execution" checking tons of IOC conditions. 

I 'Ctrl-X'd' my way through script eventually arriving at the conclusion that it appears to be breaking on one particular search term.  Now this worked previously (when originally authored in version 7.x) and currently working via Scheduler so I have questions around why running in Scheduler works vs. in Open Search.... do they run differently?  Is this 7.x 8.x (Python 2 vs. 3 syntax) difference maybe? 

We are running Enterprise 'All-in-one' Version:8.0.0 Build:1357bef0a7f6.  Below is first part of our query which is enough for you to test with... gist of it is the '-' in "Set-ExecutionPolicy" (worked before) now causes "Server Error".

For testing simply add a wildcard in place of dash - change *Set-ExecutionPolicy* to *Set*ExecutionPolicy*.  Other dashes in query do not seem to affect this.  This is very weird behavior and has hurt my brain a bit this morning.  Please advise....

index=security OR index=wineventlog source="WinEventLog:Microsoft-Windows-PowerShell/Operational"
EventCode=4103 OR EventCode=4104 OR EventCode=4688 OR EventCode=24577 Message=*Set-ExecutionPolicy*

Best regards,

Greg

[gearmstrong · Accepted solution]

Answer is query was being ran from 'un-trusted' browser which is routed through Application Gateway which has very very specific firewall rules such as to block "get-psdrive" or "set-executionpolicy" but allow "get-psadforestinfo" for example. 

Anyway... I eventually discovered that if I ran the query on localhost browser it worked.  This also explains why job created (and scheduled) in version 7.x still worked but I couldn't edit/save via 'Open in Search' option.

So, my fault for not catching this before now, but I did want to share with you in case you experience similar issues maybe you can investigate whether you have any App Gateways in play!

Best regards,

Greg

[gearmstrong]

Ok... more strange behavior from a second Alert which says is working in Scheduler Logs.  This one is our "Alert Suspicious/Administrative Processes" Alert.  I used Open Search and got Server Error.  Noticed an obvious typo with "sysprep.exe ORsysteminfo.exe ".  "OR" needs a space but when fixed I still see Server Error. So this means what is scheduled vs. what we open are different copies...  yes?

Same procedure followed to identify culprit.  "OR powershell.exe".  When removed the SPL compiles and executes properly!  I confirmed this is correct exe name and no other issues with SPL.  

I can provide code but at this point...

[gearmstrong]

Fixed my second issue by using "OR Powershell*" vs. "OR Powershell.exe".   (not as clean as I would like but it works again)

Can these two issues be related to some sort of 'Injection Protection' or similar?  Just odd that these SPL queries used to work just fine as Saved Alerts before.

Thanks,

Greg