Monitoring Splunk

Search not executed: The minimum free disk space (50000MB) reached for /home/username/splunk/var/run/splunk/dispatch. user=admin.

splunkfly
New Member

Even After increasing the size in the settings-->general settings-->Pause indexing if free disk space (in MB) falls below-->50000MB.

It is still not resolved the issue. Does anyone have solution get rid of this issue.

0 Karma

ddrillic
Ultra Champion

We ended up having a cronjob which clears the older directories. Something like -

find /opt/splunk/var/run/splunk/dispatch -type d -mtime +30 | xargs rm -rf

Not sure if it's the optimal solution, but it saves us lots of headaches ; - )

0 Karma

lguinn2
Legend

That setting ("Pause Indexing") means "if there is not 'this much' free space left on disk, stop indexing until there is more free space." This setting is also used for searches. When Splunk runs a search, it needs disk space for saving the search results. The error message says "hey, I don't have enough free disk for saving the search results." The same setting is used for both.

Increasing this number is NOT what you want!

Search results are usually only temporary, but if you have a lot of people saving their search results, they could be using a lot of space.

Or maybe you just don't have enough disk space for your indexes AND your search results.

You can temporarily set the number lower - but you need to find a more permanent resolution to the problem.

Figure out how much indexed data you have on disk and see if you need (1) more disk or (2) to limit the size of the indexers.
Decide if you need to set disk quotas for users to limit the amount of disk they can use for search results. (But be careful, because you want them to still be able to run searches and dashboards!)

0 Karma

lguinn2
Legend

Well, actually you can permanently set the number lower. But you still need to figure out what is consuming the disk, because it is probably just going to keep growing until the problem occurs again.

0 Karma
Get Updates on the Splunk Community!

Join Us for Splunk University and Get Your Bootcamp Game On!

If you know, you know! Splunk University is the vibe this summer so register today for bootcamps galore ...

.conf24 | Learning Tracks for Security, Observability, Platform, and Developers!

.conf24 is taking place at The Venetian in Las Vegas from June 11 - 14. Continue reading to learn about the ...

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...