How can I find duplicate scheduled searches running in a search head clustering environment?

[sat94541]

I have a three Node Search Head Cluster environment and I suspect that some of the scheduled searches are running multiple times. How can I find these duplicate Scheduled searches running in an SHC environment.

[rbal_splunk · Accepted solution]

If you have a DMC that searches across all SHC member on Scheduler.log - you can proceed with the following steps to debug, or else you need to index Scheduler.log from all SHC members somewhere.

1)Run the search below to check if any scheduled search was run multiple times:

( host=  OR host=) )  source=*scheduler.log  status=success 
| rex field=sid "(?\w+_\w+_\w+_\w+_at_\d+)_" 
| eval secDiff=dispatch_time-scheduled_time 
| eval schedT=strftime(scheduled_time, "%F %H:%M:%S") 
| eval dispatchT=strftime(dispatch_time, "%F %H:%M:%S") 
| transaction shortID keepevicted=t 
| search linecount>1
|table savedsearch_name, sid, schedT, dispatchT, secDiff, alert_actions

Sample Result:

2) Next you can focus on duplicate sid of one saved search like below to get more detail on the sid like and use the search below to get more details

 (
  host=  OR host=
 )
 AND
 (
  (source=*scheduler.log*   AND status=success   AND ( sid=scheduler__admin_dWlfc21zcw__RMD5ae47099b8f1c50d5_at_1460388900_111_96EB1F29-E71E-49E0-982C-767B6E64BE32 OR sid=scheduler__admin_dWlfc21zcw__RMD5ae47099b8f1c50d5_at_1460388900_122_181ABE0B-D122-42D2-A0C1-BACD9B46F50A ))
 OR 
 ( source=*splunkd.log* AND "Making node the captain" )

) | table _raw host

Result: In this case it shows duplicate were caused by the captain switch.

In this case duplicate were caused due to switch Captain- this is from Splunk version 6.2.6.