I would like to get a total count of the number of scheduled searches and ad-hoc (UI based) searches that are run per day.
In pre-5.x Splunk I could use:
index=_internal (sourcetype=searches OR sourcetype=scheduler) | timechart span="1d" count(host) by sourcetype
Apparently the sourcetype=searches no longer returns results.
If you are collecting process-level information for Splunk processes using the S.o.S app's ps_sos.sh
scripted input, you can break down your daily search workload between scheduled and ad-hoc searches like so:
`set_sos_index` sourcetype=ps host=<indexer or search-head host>
| multikv
| `get_splunk_process_type`
| search type="searches"
| rex field=ARGS "_--user=(?<search_user>.*?)_--"
| rex field=ARGS "--id=(?<sid>.*?)_--"
| rex field=sid "remote_(?<search_head>[^_]*?)_"
| eval is_remote=if(like(sid,"%remote%"),"remote","local")
| eval is_scheduled=if(like(sid,"%scheduler_%"),"scheduled","ad-hoc")
| eval is_realtime=if(like(sid,"%rt_%"),"real-time","historical")
| eval is_subsearch=if(like(sid,"%subsearch_%"),"subsearch","generic")
| eval search_type=is_remote.", ".is_scheduled.", ".is_realtime
| timechart span=1d dc(sid) AS "Search count" by is_scheduled
Note that you'l need to run this search from within the context of the S.o.S app for the macros it uses to be available. You will also need for the ps_sos.sh
scripted input to have been running for several days on the instance that you are targeting the search against.
Number of scheduled searches per day:
index=_internal sourcetype="scheduler" | timechart span=1d count(savedsearch_name) by host | addtotals
Also tried this but the numbers don't look right:
index=_audit user!=splunk-system-user | stats count by action