Monitoring Splunk

What do you do if the minimum free disk space is reached on a fresh Splunk installation?

derejekifle
New Member

I have a fresh install of Splunk on a CentOS VM that has 15GB of disk..
I'm getting the following message... what do I need to do?

  1. Dispatch Command: The minimum free disk space (2000MB) reached for /opt/splunk/var/run/splunk/dispatch.
  2. Failed to start KV Store process. See mongod.log and splunkd.log for details.
  3. Disk Monitor: The index processor has paused data flow. Current free disk space on partition '/' has fallen to 347MB, below the minimum of 2000MB. Data writes to index path '/opt/splunk/var/lib/splunk/audit/db'cannot safely proceed. Increase free disk space on partition '/' by removing or relocating data.
  4. KV Store changed status to failed. KVStore process terminated
  5. KV Store process terminated abnormally (exit code 100, status exited with code 100). See mongod.log and splunkd.log for details.
0 Karma

codebuilder
SplunkTrust
SplunkTrust

Configure logrotate and/or manually purge Splunk log files.

These are unfortunately located at /opt/splunk/var/log/splunk and /opt/splunk/var/log/introspection, which obviously count against your available space on /opt (which is usually small on a standard Linux install).

I generally symlink those directories to /var/log/splunk and /var/log/introspection, with /var/log being on it's own disk, VG, and LV
/dev/mapper/varlogvg01-varloglv01 e.g.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

hardikJsheth
Motivator

You are indexing more data than available space. Best would be to increase the disk size, if you are going to keep indexing at same rate.

Just to make it work you can reduce minimum recommended free space to 1GB by logging into your Splunk UI and goto Settings --> General Settings. Change the size for "Pause indexing if free disk space (in MB) falls below ". After changing this values you will have to restart your machine.

For KVStore if it doesn't work after restart check for error message in $SPLUNK_HOME/var/log/splunk/mongod.log.

0 Karma

tmarlette
Motivator

check your free disk space on CentOs under the /opt partition (assuming this is a stand alone instance). You've likely hit your max. if this is machine is a search head and an indexer, you're going to blow through 15GB almost after install.

keep in mind that all default indexes, as well as new indexes default to storing 500GB of data. This means that the index won't rotate out old data until each index hit's 500GB. I'm not sure how many indexes you have, but in your case, 1 is too many at 500GB. adjust the sizes to compensate, and get about 120GB or so on the /opt partition and you should be Ok for awhile.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...