Monitoring Splunk

Search not executed: The minimum free disk space (50000MB) reached for /home/username/splunk/var/run/splunk/dispatch. user=admin.

splunkfly
New Member

Even After increasing the size in the settings-->general settings-->Pause indexing if free disk space (in MB) falls below-->50000MB.

It is still not resolved the issue. Does anyone have solution get rid of this issue.

0 Karma

ddrillic
Ultra Champion

We ended up having a cronjob which clears the older directories. Something like -

find /opt/splunk/var/run/splunk/dispatch -type d -mtime +30 | xargs rm -rf

Not sure if it's the optimal solution, but it saves us lots of headaches ; - )

0 Karma

lguinn2
Legend

That setting ("Pause Indexing") means "if there is not 'this much' free space left on disk, stop indexing until there is more free space." This setting is also used for searches. When Splunk runs a search, it needs disk space for saving the search results. The error message says "hey, I don't have enough free disk for saving the search results." The same setting is used for both.

Increasing this number is NOT what you want!

Search results are usually only temporary, but if you have a lot of people saving their search results, they could be using a lot of space.

Or maybe you just don't have enough disk space for your indexes AND your search results.

You can temporarily set the number lower - but you need to find a more permanent resolution to the problem.

Figure out how much indexed data you have on disk and see if you need (1) more disk or (2) to limit the size of the indexers.
Decide if you need to set disk quotas for users to limit the amount of disk they can use for search results. (But be careful, because you want them to still be able to run searches and dashboards!)

0 Karma

lguinn2
Legend

Well, actually you can permanently set the number lower. But you still need to figure out what is consuming the disk, because it is probably just going to keep growing until the problem occurs again.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...