My forehead is sore from banging it on my desk. Please help.
I cannot get scripts to run from an alert. The following is all the relevant info I have been using to trouble shoot this.
Implementation
To make it easy, I configured the alert to run "always" so the search term is irrelevant. The alert should be triggered every minute regardless. (I have also ran this with a search that I was manually triggering events in)
The powershell script
#!C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
"hello world" | Out-File -FilePath .\hello_ps.txt
The savedsearches.conf stanzas
[script_test_ps]
action.script = 1
action.script.filename = test.ps1
alert.digest_mode = True
alert.suppress = 0
alert.track = 0
auto_summarize.dispatch.earliest_time = -1d@h
cron_schedule = * * * * *
enableSched = 1
search = sourcetype=nothing
Verification
We know the search is running, we can see the search run and we can watch the script being called but it never actually executes.
From python.log
2014-10-02 18:19:03,171 Central Daylight Time INFO runshellscript:188 - runshellscript: ['C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe', 'D:\\Splunk\\bin\\scripts\\test.ps1', '0', 'index=testing sourcetype=script_test', 'index=testing sourcetype=script_test', 'script_test_ps', 'Saved Search [script_test_ps] always(0)', 'https://splunkweb.domain.tld/app/search/@go?sid=scheduler_bmVpbC5wZXRlcnNvbg__search__RMD59d824a49b6b738b2_at_1412291940_27935', '', 'D:\\Splunk\\var\\run\\splunk\\dispatch\\scheduler_bmVpbC5wZXRlcnNvbg__search__RMD59d824a49b6b738b2_at_1412291940_27935\\results.csv.gz']
From scheduler.log
10-02-2014 18:19:03.889 -0500 INFO SavedSplunker - savedsearch_id="neil.peterson;search;script_test_ps", user="neil.peterson", app="search", savedsearch_name="script_test_ps", status=success, digest_mode=1, scheduled_time=1412291940, dispatch_time=1412291942, run_time=0.562, result_count=0, alert_actions="script", sid="scheduler_bmVpbC5wZXRlcnNvbg__search__RMD59d824a49b6b738b2_at_1412291940_27935", suppressed=0, thread_id="AlertNotifierWorker-1"
Troubleshooting
Powershell execution policy is unrestricted
PS D:\Splunk\bin\scripts> Get-ExecutionPolicy
Unrestricted
Splunkd is running as a service account
PS D:\Splunk\bin\scripts> Get-WmiObject win32_service | Where-Object {$_.name -like "splunk*"} | Select-Object name, startname
name startname
---- ---------
Splunkd domain\svc.splunk
splunkweb domain\svc.splunk
The service account is in the Administrators groups
The troubleshooting steps from this wiki article: http://wiki.splunk.com/Community:TroubleshootingAlertScripts
Is my scheduled search running? YES. I see it in scheduler.log
Is my scheduled search generating the expected results? YES. I can watch the results come in on a real time search. I have also scheduled the alert to run "always" to make the search part of it irrelevant.
Is my alert action being triggered? YES. I have added email actions and I get those emails, as well as watching it in https://splunkweb.domain.tld/en-US/app/launcher/job_management?savedSearch=script_test
Is my alert script working? YES. I can run it from the command line, as the svc.splunk user, but when the alert triggers it, nothing happens.
Reference
I have been using the following documentation. I have read it and reread it.
http://docs.splunk.com/Documentation/Splunk/6.1.3/alert/ConfiguringScriptedAlerts
http://docs.splunk.com/Documentation/Splunk/6.1.3/Alert/Setupalertactions#Run_a_script
http://docs.splunk.com/Documentation/Splunk/6.1.3/Admin/Savedsearchesconf
http://wiki.splunk.com/Community:TroubleshootingAlertScripts
I am out of rope on this one. Any help is appreciated. I have also done all of the above with a batch script as well with the same disappointing results.
I have been testing and troubleshooting this from lots of different angles. I tried to break everything down to the simplest example, but my copy pasta above may still have gotten mixed up. If so please point it out and I will verify my testing results.
EDIT: For posterity sake, I will explain the (very very very elementary and obvious) solution here. The script was writing out to $splunk_home\bin not $splunk_home\bin\scripts where the script was located. If you browse to the scripts location and run it, it obviously works, but when Splunk runs it, those relative paths do not exist from bin/
The other quirk I have noticed is that when Splunkd is running as LocalSystem, it can execute batch scripts but not powershell. When Splunk is running as a service account the powershell scripts work fine.
To wrap your ps1 up into a bat you should put the folllowing in the batch script.
C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -File .\location\ofyour\script.ps1
The above will look for $splunk_home\bin\location\ofyour\script.ps1 which is what we want.
... View more