"Too many search jobs found in the dispatch directory" - Can we run this command on our clustered search heads to clean it?

[sim_tcr]

We are on search head clustering with 4 search heads and version 6.3.3.
Recently started seeing WARNING:

Too many search jobs found in the dispatch directory (found=3186, warning level=2000). This could negatively impact Splunk's performance, consider removing some of the old search jobs.

Can we run below command on all our search heads to clean it?

splunk cmd splunkd clean-dispatch /apps/old-splunk_dispatch -1d

Will this only clean 1 day old jobs, or can we safely run it with -7d?

[sowings]

While it's not a direct answer to your question, consider WHY these jobs may be piling up. See my post on this answer for more information.

Regarding @SloshBurch's comment: if the jobs are adhoc in nature (the SID begins with a 10-digit number), then cleaning from a single member of the SHC will be OK for that member. There's no replication of adhoc search jobs. If, however, the pile up of dispatch directories is from scheduled search activity, it might make sense to evaluate the TTLs of those jobs to understand why they're piling up. The threshold of 2000 entries is a warning level, but given the way SHC replicates dispatch directories, it might be a bit low. If you're routinely up in the 3000 range, it may be reasonable (with the guidance of support) to raise the configured limit.

[sloshburch]

Extra details: http://docs.splunk.com/Documentation/Splunk/6.4.1/Troubleshooting/SOOOmanydispatchdirectories

[sim_tcr]

I ran the command only few got moved. Getting below error on some

Could not move /apps/splunk/var/run/splunk/dispatch/_splunktemps to /apps/old-splunk_dispatch/_splunktemps. Directory not empty
total: 2014, moved: 0, failed: 1, remaining: 2014 job directories from /apps/splunk/var/run/splunk/dispatch to /apps/old-splunk_dispatch

[sloshburch]

My guess is that the "Directory not empty" was similar to a folder currently being used (open file handles). Would you confirm that you stopped that host's splunk instance - just in case it was working on the FS at the same time? I'm also wondering if cleaning dispatch with a SHC will be sufficient at cleaning the other SHC members as well (since file system changes in a SHC are not synchronized).

[ddrillic]

You can safely clean up a week of data with -7d.

More about it at What causes "Too many search jobs found in the dispatch directory" and should Splunk be handling thi...