How do I create an alert which triggers a custom script when unique results are ingested into Splunk?

[MikaJustasACN]

Hello,

I am having trouble establishing a logic to cover the following.

Selected events (windows and some syslog) are being sent to an indexer by a universal forwarders (UF). For each unique event, I need to trigger a custom script which does some magic in the background. A savedsearch should do the trick, however, I am not confident how I can ensure that I won't hit the following problems. The initial plan was to run the saved search -15min@min to -5min@min every 10 minutes.

Problem 1. if an event was delayed to be indexed due to network connectivity and comes -30 minutes, I would never detect that.

Problem 2. savedsearch might fail to run and state in _internal still would still show success (Splunk 6.6.4). At least my testing showed that.

Problem 3. With savedsearch, even if the scheduled cron */10 ran might run with some window, even if I don't allow that. From experience, I have noticed it can start running a few seconds earlier/later. If it runs earlier, it might create a gap of 1 minute due to @min.

Thank you for advice!

[MikaJustasACN · Accepted solution]

I will post an answer which I believe is the correct one. I found it in a forum not related to Splunk but adopted to Splunk.

1. savedsearch1 runs to store events into a summary index to clean up the events and add a flag which will be used by the script. Run time is since last success time of the savedsearch1.
2. savedsearch2 runs on the summary index on events where flag is false (script not executed) and runs a script on results. The script writes back an event back saying true or false on the flag (if script succeeds, true, if fails, false).

Now the only thing is to figure out how to write an event back to Splunk with a python script 🙂

[mstjohn_splunk]

hi @MikaJustasACN

Did the answer below solve your problem? If so, please resolve this post by approving it! If your problem is still not solved, keep us updated so that someone else can help ya. Thanks for posting!

[mwdbhyat]

Hi,

Would something like this help out? It's condition based rather than schedule based:

https://answers.splunk.com/answers/100268/trigger-a-report-based-on-an-event.html