What is meant to clean up dispatch?

[djce]

Splunk recently fell over because the dispatch directory (on an ext2 filesystem) hit 32000 directory entries, so the OS would not let Splunk create any more, and all searches failed. The jobs dated back over 6 months or so - it looked like no jobs had ever been removed.

I removed old jobs by hand (find, xargs, rm etc) to free up directory slots, restarted Splunk, and searches started working again. Almost all of the jobs I had to remove did not have "save" set.

My question is: surely something (in Splunk) is normally meant to automatically delete old jobs, perhaps once their TTL has expired? What is meant to do that, and how might I debug the cause of it failing to clean up?

Sure, I can add a cron to do the find|xargs|rm dance to delete old jobs, but that feels very wrong. Splunk is meant to take care of this, no?

This seems to be related to
http://splunk-base.splunk.com/answers/28390/minimum-free-disk-space-1000mb-reached-for-optsplunkvarr...
and http://splunk-base.splunk.com/answers/29551/too-many-search-jobs-found-in-the-dispatch-directory
but they don't seem to answer it.

[nmaiorana]

What is the root cause of the directory filling up? Going in once a week to clean it manually seems ridiculous to me.

[d3]

I've got a problem with dispatch filling up but what is provided here is NOT a solution. My dispatch directory keeps getting 5-10 new directories each minute. My only saved searches are scheduled for once/day yet I keep getting dispatch is full. I'm currently at 1400 just today and rising. Running splunkd clean-dispatch or some convoluted 'rm' process is NOT solving the root issue of why it fills up in the first place. Can anyone point to what to look for why so many directories are being created and why there is no automatic reaper to clean this out properly?

[rgcurry]

What version are you running? We just started seeing this message after upgrading the 4.2.4 which we did to help resolve a Web UI stability problem (which it did resolve, thank you.) And what is your 'find' command structure? I have tried the following but it does not work if I use anything less than 5 for the -mtime parameter:

find /$SPLUNK_HOME/var/run/splunk/dispatch -mtime +2 -exec rm -R {} \;

There are also a lot of "session-*" files in the $SPLUNK_HOME/var/run/splunk/ directory, many with a matching, empty ".lock" file. What are these and can I get rid of these?

[yannK]

see this answer http://splunk-base.splunk.com/answers/29551/too-many-search-jobs-found-in-the-dispatch-directory

./splunk cmd splunkd clean-dispatch

usage: splunkd clean-dispatch '' ''
example: splunkd clean-dispatch /tmp/old-dispatch-jobs/ -1month
example: splunkd clean-dispatch /tmp/old-dispatch-jobs/ -10d@d
example: splunkd clean-dispatch /tmp/old-dispatch-jobs/ 2011-06-01T12:34:56.000-07:00

[djce]

We're on version 4.2.1.

The "find" I used to manually zap these jobs was as follows:

cd var/run/splunk/dispatch/
find . -maxdepth 1 -name '[0-9]*' -mtime +7 | while read job ; do if [ ! -e "$job/save" ] ; then rm -rfv $job ; fi ; done