I have this search
host=MyIndeders sourcetype=cpu | multikv fields CPU pctUser | timechart span=5m avg(pctUser) AS "Avg % CPU" by host
I can create a chart showing the CPU pegs at 100% for 10 minutes every hour.
I want to find all scheduled jobs that are scheduled to run every hour to find the ones that I can stop.
Anyone know how I can search for jobs in all apps that might run every hour?
You can get the historic run for scheduled searches from _internal index. Below search will give you list of scheduled search running every 3600 secs/1 hour, along with their run_time and result_count (which may help you identify expensive queries.
index=_internal sourcetype=scheduler | table _time user savedsearch_name status scheduled_time run_time result_count | sort savedsearch_name _time | streamstats current=f window=1 first(scheduled_time) as prevSched by savedsearch_name | eval schedulingTime=scheduled_time-prevSched | where schedulingTime=3600
You can get the historic run for scheduled searches from _internal index. Below search will give you list of scheduled search running every 3600 secs/1 hour, along with their run_time and result_count (which may help you identify expensive queries.
index=_internal sourcetype=scheduler | table _time user savedsearch_name status scheduled_time run_time result_count | sort savedsearch_name _time | streamstats current=f window=1 first(scheduled_time) as prevSched by savedsearch_name | eval schedulingTime=scheduled_time-prevSched | where schedulingTime=3600
Do you capture Process information on the server? (may be using TA apps). If yes, then you might be able to track the process which is causing those peak CPU usage.
Thanks @Somesoni2 this helps but I still can't find what is causing the 100 CPU spick every hour for 10 minutes. At least I know how to find the scheduled jobs thanks.
You have to use the RESTAPI for this
http://docs.splunk.com/Documentation/Splunk/6.1.2/RESTAPI/RESTsearch#saved.2Fsearches
The returned values from the RESTAPI will help you to build your algorithm to stop the searches.
Yes I did thanks for the help, I looked as suggested by @somesoni above "|rest /services/saved/searches" and only found 30 saved searches and I know that I have way more than that. Also I only found 12 scheduled as I said above. So fare I have not been able to find the full list of scheduled jobs with there perspective cron schedule. Jo Joy just yet. thanks for the help. I will keep looking till I find the answer. Maybe I will get support to help; ARGG
You're probably only looking at the current app context. You need to wildcard your query to force it to look at all apps:
| rest /servicesNS/-/-/saved/searches
Add a filter for is_scheduled=1 to see only the scheduled jobs. See https://answers.splunk.com/answers/221242/is-there-a-way-to-identify-all-scheduled-searches.html for details.
we have been using this rest API for quite sometime and it works fine.
Did you get a chance to login to SplunkWeb and check how many saved searches are there and indeed how many of them are scheduled?
Thanks, this did show me about 30 saved searches. I know I have many more than that. of the 30 only 12 have something in the "cron_schedule" column. four of which are running every 5 minutes "*/5 * * * *". I am looking for expensive jobs that run every hour.
Scheduled views (simple xml dashboards scheduled to run) are different from Scheduled searches (saved search scheduled to run). You should try using "|rest /services/saved/searches".
Thanks I have been looking for about an hour now but have not been able to find the info useing rest api.
I used this command | rest /services/autherization/scheduledviews
and was able to only see one scheduled view. I have not found the scheduled searches. any idea where to look?