Alerting

Scheduling alerts via Cron

ppurokit
Path Finder

Hi All,

I have a requirement where i have around 80 saved searches which needs to run in an interval of every 5 min.

Scheduling all the 80 saved searches to run at the same moment will be an issue and it will hog the CPU also.

So i planned to run all the 80 queries in a batched way like.. For Example : 20 seaches at 1st min , another 20 at 2nd min etc... So that i will cover all the 80 searches in the 5 min window.

Using the Splunk Cron "* * * * *" I see only option to provide only at the minutes level. Like schedule to run for every 5 min?

How can i achieve it to run all in the 5 min window but , 1 min 20 , 2min 20 etc..

Please suggest.

Thanks
Prasanna Kumar

rriegert
New Member

The Problem with the 1-59/5 * * * * answer is that once you get to the 20th alert and you have the alert set to look at something like 5-10 min in the past for an event, you will miss alerts. For example, if you set the alert to kick off at 20 past the hour, i.e. 20-59/5 * * * *, it will run at 20, 25, 30, ..., 55 looking back at 5-10 min in the past, i.e. at 55 it will look back at 45-50 past the hour. However, the next time it will run after 55 in hour one, is 20 after hour 2. So, you'd miss a total of 20 minutes of events that occurred from 50 minutes past hour 1 to 10 minutes past hour 2. The preferred way that I'd found to do a cron syntax that does 5 minutes starting at 20 past the hour originally but then continuing every 5 minutes to eternity is */5+20 * * * *. However, it appears that splunk does not support the +x syntax. So, the next "best" option I have found is by doing this:

alert1=0,5,10,15,20,25,30,35,40,45,50,55 * * * *
alert2=1,6,11,16,21,26,31,36,41,46,51,56 * * * *
alert3=2,7,12,17,22,27,32,37,42,47,52,57 * * * *
alert4=3,8,13,18,23,28,33,38,43,48,53,58 * * * *
alert5=4,9,14,19,24,29,34,39,44,49,54,59 * * * *

And you'd start back over at 0, 5, 10, etc. after alert 10. It's a bit messy, but given Splunk's apparent cron syntax limitation, it appears to be the only thing that will do the trick. Any suggestions to how to clean up this cron syntax given Splunk's limitations? Any comments appreciated.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

As suggested by Lukejadamec, every search will its own cron.
Try this

Search 1 ----     */5 * * * *                 runs @ 0,5,10,15....
Search 2 ----     1-59/5 * * * *              runs @ 1,6,11,16....
Search 3 ----     2-59/5 * * * *              runs @ 2,7,12,17....
Search 4 ----     3-59/5 * * * *              runs @ 3,8,13,18....
Search 5 ----     4-59/5 * * * *              runs @ 4,9,14,19....

lukejadamec
Super Champion

Each search will need its own schedule.

To run a cron schedule every 5 minutes starting at a particular minute use this:

EDITED:

1-59/5 * * * * for minutes 1,6,11,16,etc

2-59/5 * * * * for minutes 2,7,12,17,etc

and so on.

lukejadamec
Super Champion

Let me see if I understand before I answer.
Lets say you have 3 searches. You want them to run at different times(search 1 at minute 6, search 2 at minute 7, and search 3 at minute 8), but you want them to search over the same time frame (0-5 minutes)?

0 Karma

ppurokit
Path Finder

Thanks for the inputs.

So all these will run for every 5 min. I do have search window on earliest =-6m@m and latest=-1m@m.

If i have this for all my queries, and i run search1 at 5th min, search2 at 6th min, search3 at 7th min...

I hope I will not be able to get all the data for the previous 5 min.

How can i over come this issue?

Your suggestions will help me a lot.

0 Karma

lukejadamec
Super Champion

My bad. Turns out Splunk does not accept all forms of cron syntax. I updated the answer with a syntax that should work in Splunk.

0 Karma

ppurokit
Path Finder

Hi Lukejadamec,

I tried as you said and im getting the below error.

Encountered the following error while trying to update: In handler 'savedsearch': Invalid cron_schedule= * /15+3 * * * *

Can you please let me know if I'm missing something here?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...