Splunk Search

How to find all searches that are scheduled to run every hour?

hartfoml
Motivator

I have this search

host=MyIndeders sourcetype=cpu | multikv fields CPU pctUser | timechart span=5m avg(pctUser) AS "Avg % CPU" by host

I can create a chart showing the CPU pegs at 100% for 10 minutes every hour.

I want to find all scheduled jobs that are scheduled to run every hour to find the ones that I can stop.

Anyone know how I can search for jobs in all apps that might run every hour?

Tags (2)
1 Solution

somesoni2
SplunkTrust
SplunkTrust

You can get the historic run for scheduled searches from _internal index. Below search will give you list of scheduled search running every 3600 secs/1 hour, along with their run_time and result_count (which may help you identify expensive queries.

index=_internal sourcetype=scheduler  | table _time user savedsearch_name status scheduled_time run_time result_count | sort savedsearch_name _time  | streamstats current=f window=1 first(scheduled_time) as prevSched by savedsearch_name | eval schedulingTime=scheduled_time-prevSched  | where schedulingTime=3600

View solution in original post

somesoni2
SplunkTrust
SplunkTrust

You can get the historic run for scheduled searches from _internal index. Below search will give you list of scheduled search running every 3600 secs/1 hour, along with their run_time and result_count (which may help you identify expensive queries.

index=_internal sourcetype=scheduler  | table _time user savedsearch_name status scheduled_time run_time result_count | sort savedsearch_name _time  | streamstats current=f window=1 first(scheduled_time) as prevSched by savedsearch_name | eval schedulingTime=scheduled_time-prevSched  | where schedulingTime=3600

rahulroy_splunk
Path Finder

Do you capture Process information on the server? (may be using TA apps). If yes, then you might be able to track the process which is causing those peak CPU usage.

0 Karma

hartfoml
Motivator

Thanks @Somesoni2 this helps but I still can't find what is causing the 100 CPU spick every hour for 10 minutes. At least I know how to find the scheduled jobs thanks.

0 Karma

strive
Influencer

You have to use the RESTAPI for this

http://docs.splunk.com/Documentation/Splunk/6.1.2/RESTAPI/RESTsearch#saved.2Fsearches

The returned values from the RESTAPI will help you to build your algorithm to stop the searches.

0 Karma

hartfoml
Motivator

Yes I did thanks for the help, I looked as suggested by @somesoni above "|rest /services/saved/searches" and only found 30 saved searches and I know that I have way more than that. Also I only found 12 scheduled as I said above. So fare I have not been able to find the full list of scheduled jobs with there perspective cron schedule. Jo Joy just yet. thanks for the help. I will keep looking till I find the answer. Maybe I will get support to help; ARGG

0 Karma

cphair
Builder

You're probably only looking at the current app context. You need to wildcard your query to force it to look at all apps:
| rest /servicesNS/-/-/saved/searches

Add a filter for is_scheduled=1 to see only the scheduled jobs. See https://answers.splunk.com/answers/221242/is-there-a-way-to-identify-all-scheduled-searches.html for details.

0 Karma

strive
Influencer

we have been using this rest API for quite sometime and it works fine.
Did you get a chance to login to SplunkWeb and check how many saved searches are there and indeed how many of them are scheduled?

0 Karma

hartfoml
Motivator

Thanks, this did show me about 30 saved searches. I know I have many more than that. of the 30 only 12 have something in the "cron_schedule" column. four of which are running every 5 minutes "*/5 * * * *". I am looking for expensive jobs that run every hour.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Scheduled views (simple xml dashboards scheduled to run) are different from Scheduled searches (saved search scheduled to run). You should try using "|rest /services/saved/searches".

0 Karma

hartfoml
Motivator

Thanks I have been looking for about an hour now but have not been able to find the info useing rest api.

I used this command | rest /services/autherization/scheduledviews

and was able to only see one scheduled view. I have not found the scheduled searches. any idea where to look?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...