Splunk Search

Can one scheduled saved search trigger another saved search?

Lowell
Super Champion

Is is possible to setup an alerting condition on a scheduled saved search what would turn around and launch another saved search?

This may seem like a weird request, but here are some scenarios I've come across this week where I think such a function may be helpful. Or perhaps an better alternative is out there. Either way, I'm looking for ideas.

Scenarios

  1. A firewall is reporting a large number of errors from a PPTP session. Unfortunately, the repeating error messages only contain a pid value which has to be crossed referenced with other events to get other relevant fields (username, local_ip, remote_ip,...). What would be nice is to have one scheduled saved search that looks for this scenario, which would then trigger another more-detailed saved search that reports the full-detail via an email alert action. (This second saved search is not scheduled, it is only run on demand.) That way, the heavy-duty (more resource intensive) search only runs on an as-needed basis, which is triggered by a lighter-weight regularly scheduled search.
  2. A business critical process is preforming poorly. We have tons of alerts around this process already, so adding more isn't the solution. However, it would be helpful to send a PDF copy of a post-poor-performance-analysis view summarizing the problems. We could schedule delivery of this report, but we don't want the overhead of generating it when it's not needed, and even more importantly, if the report is emailed out daily (irregardless of whether there were performance problems or not) then the recipients will simply learn to ignore it. (It seems to be human nature). I would like to have one scheduled search that evaluates the overall performance and when the conditions are right, launch the PDF view sending mechanism. Perhaps this is currently possible with the existing view delivery mechanism I'm not sure.

I've thought of a few ways to jerry-rig this, but nothing stands out as a good idea at the moment. If there is a way to use an triggered action script to make an API call to splunkd to setup a scheduled savedsearch to run just once, or a "run-now" mode that could probably do the trick.

The thing I like about having one scheduled saved search trigger another saved search is that such a mechanism could be done from within the scheduler and could therefore be managed and controlled by it.

Any thoughts or ideas?


Update:

Since I really haven't received any helpful feedback on this, and I'm not very good at sitting still, I've started working on my own solution to this problem by attempting to create a custom alerting action which will run a custom search command, which in turn will trigger the execution of a secondary saved search.

I've run into an issue getting the custom alerting action working, but again I'm just trying to figure out what I can based on existing config files.

I've had some success getting a custom search command to launch a saved search, but I've run into an bug that prevents the authentication session key from being usable by a search command.

1 Solution

Lowell
Super Champion

I have published an app that gives me the functionality that I'm looking for:
http://www.splunkbase.com/apps/All/4.x/app:RunSavedSearch+alert+action

View solution in original post

Lowell
Super Champion

I have published an app that gives me the functionality that I'm looking for:
http://www.splunkbase.com/apps/All/4.x/app:RunSavedSearch+alert+action

jitsinha
Path Finder

It seems the link isnt working.

Simeon
Splunk Employee
Splunk Employee

Lowell
Super Champion

@Paolo, I was wondering that too. But the answer I received is that can't really abort a search like this. There is really no flow control mechanism provided by splunk short of the alerting condition of a saved search. See related: http://answers.splunk.com/questions/4472/can-a-search-be-terminated-prematurely-based-on-a-condition...

0 Karma

Paolo_Prigione
Builder

This is very interesting.

Whould there be a way to abort the search execution without raising an error?
Because you could then add a custom search command which, in case reads zero-input results, aborts the search. Something like this:

"lightsearch" | abortonnoresults | search "heavysearch" | ...

0 Karma

Lowell
Super Champion

Yes, that is correct. I do want to use conditional alerting. But I want to use it in a way that is not coved in the docs. I would like to use a conditional alert that runs a secondary saved search and I want that secondary search to actually be the search that launches the alerting action. Perhaps my question was unclear.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...