Splunk Search

Scheduling multiple searches

mansel_scheffel
Explorer

Hi,

I am trying to schedule 60 saved searches with summery indexing. There are for 5 different searches, each with 4 schedules - 10min, 1h, 1d, 1w. The initial 10min search will summarise, from there I want the 1h search to create an SI from the 10min SI, the 1d to create an SI from the 1h SI, etc.

-What would be the best way to do this in terms of setting schedule windows
-How can I catch up for delayed data, populate the latest period and verify the previous one if possible?

Thanks!

0 Karma
1 Solution

somesoni2
SplunkTrust
SplunkTrust

To take care of indexing latency, your time range should go little earlier then the current time.

E.g. For 10 min searches, the time range can be -15m@m to 5m@m, allowing 5 min extra for new events to become searchable. The cron can be this 3-59/10 * * * * (running on 3,13,23.. min of every hour).
For this same 10 min search, the data for last hour will get completed after 3rd min execution of 10 min search, so the time range for 1 hour this can will be -1h@h to @h with cron as 15 * * * *
The daily search's time range will be -1d@d to @d, with cron 11 1 * * * (allowing adding 1 hour 11 min for hourly data to be completed)
The weekly search's time range will be -1w@w to @w, with cron 22 2 * * 1

View solution in original post

0 Karma

somesoni2
SplunkTrust
SplunkTrust

To take care of indexing latency, your time range should go little earlier then the current time.

E.g. For 10 min searches, the time range can be -15m@m to 5m@m, allowing 5 min extra for new events to become searchable. The cron can be this 3-59/10 * * * * (running on 3,13,23.. min of every hour).
For this same 10 min search, the data for last hour will get completed after 3rd min execution of 10 min search, so the time range for 1 hour this can will be -1h@h to @h with cron as 15 * * * *
The daily search's time range will be -1d@d to @d, with cron 11 1 * * * (allowing adding 1 hour 11 min for hourly data to be completed)
The weekly search's time range will be -1w@w to @w, with cron 22 2 * * 1

0 Karma

mansel_scheffel
Explorer

With regards to the above - if I dont want to have buckets that are say 09.48 -0 9.58 can I change the cron jobs to run at 5-59/10 * * * * and so on, so that my time will be on 09.50 - 10.00 etc?

0 Karma

mansel_scheffel
Explorer

Thanks for the help! You answered another one of my questions on tstats as well which was very helpful. As mentioned in the above post, I am creating a kinda search inception and am wondering how my 2nd,3rd,4th search string would be written?

EG - (Event count) - The initial 10min search to populate the SI is - | tstats count WHERE index=test* earliest=-10m@m latest=@m by _time span=10m

How would I shape that search for the other 3 SI's populating searches, based on the first SI, so that I can ultimately use anyone of the 4 I choose as a single value with trendline on my dashboard?

0 Karma

somesoni2
SplunkTrust
SplunkTrust

After you set your 10 min search using above tstats query, your 10 min SI will have (listing main fields only) fields _time and count, with field count not being an indexed fields (fields gets ingested as kv pair in SI).

So your other SI searches would use regular stats/timechart command, like this

Hourly

index=your10MinSI source="your10MinSISearchName" earliest=-1h@h latest=@h | timechart span=1h sum(count) as count

Daily

index=your1HourSI source="your1HourSISearchName" earliest=-1d@d latest=@d | timechart span=1d sum(count) as count

Weekly

index=your1DaySI source="your1DaySISearchName" earliest=-1w@w latest=@w | timechart span=1w sum(count) as count
0 Karma

mansel_scheffel
Explorer

Awesome thanks a lot !!

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...