Monitoring Splunk

Can I change the path of the dispatch directory?

banders2112
Engager

Hi,

My question is related to this one: http://answers.splunk.com/questions/1140/how-can-i-resolve-the-error-the-minimum-free-disk-space-204...

Namely, I am running into disk space issues. Splunk tries to store searches in /opt/splunk/var/run/splunk/dispatch, but can I change this path? I only have about 1.5 GB on the partition on which Splunk is installed. I set the indexing path to a NFS share with terabytes of storage (/mnt/splunk/var/lib/splunk). I'd like to use this path for search storage as well. Is this possible? Is there a setting in a config file somewhere that I can change?

Thanks! Brian

Tags (2)
1 Solution

jrodman
Splunk Employee
Splunk Employee

Nope, you can't specify an alternate location for the dispatch directory.

Options:

  • You could relocate your whole splunk install. If you have less than 1.5GB available, this would be my first suggestion.
  • On Unix, you could try making this directory into a symbolic link. You might discover new misbehaviors, because we don't test this way, but generally Splunk doesn't care about symlinks too much.

Sidenote: Indexing to NFS can be perilous. Be certain your NFS environment is going to work all the time without issues. If it stops operating for any length of time, you'll either get a service interruption, or lost data. I'd rather index to a higher reliability datapath and then quickly migrate the data to NFS, myself.

View solution in original post

shockman
Engager

I'm running into the same problem running Splunk on AWS (with a small root volume).
Previously moved the indexes to a non root volume. (specified a new splunkdb location in etc/splunk-launch.conf)
I moved the data (don't know if the first step is entirely necessary), symlinked to the external disk and restarted
NOTE: done as the splunk user, and assuming your external volume is mounted at /mnt/data

mkdir /mnt/data/dispatch
mv var/run/splunk/dispatch/* /mnt/data/dispatch/ && rmdir var/run/splunk/dispatch && ln -s /mnt/data/dispatch/ var/run/splunk/dispatch
~/bin/splunk restart

This seems to work for me. There is some report above that the symlink causes problems in previous versions, but I'm not seeing any errors of misbehavior.

BrendanCO
Path Finder

Shockman. Thank you. I had the exact scenario on AWS, added a drive and your solution here of just linking the old output folder with the new worked like a charm!

0 Karma

jrodman
Splunk Employee
Splunk Employee

Nope, you can't specify an alternate location for the dispatch directory.

Options:

  • You could relocate your whole splunk install. If you have less than 1.5GB available, this would be my first suggestion.
  • On Unix, you could try making this directory into a symbolic link. You might discover new misbehaviors, because we don't test this way, but generally Splunk doesn't care about symlinks too much.

Sidenote: Indexing to NFS can be perilous. Be certain your NFS environment is going to work all the time without issues. If it stops operating for any length of time, you'll either get a service interruption, or lost data. I'd rather index to a higher reliability datapath and then quickly migrate the data to NFS, myself.

David
Splunk Employee
Splunk Employee

There has also recently been at least one instance where the disk space size check did not follow a symlink. Having looked into this, my general consensus is that I wouldn't recommend symlinking as: 1. Most customers don't symlink and your experience will be better the less unique you are. 2. We do create large files and expect fast performance on other paths other than dispatch. 3. Anecdotal evidence exists about disk space size check not following symlinks.

0 Karma

jaredlaney
Contributor

@David - Is this comment still true?

0 Karma

waechtler
Path Finder

the issue with the outputlookup command seems to be resolved in 4.1.4:

The "outputlookup" search command doesn't work if var/run/ is on a different volume from etc/apps. (SPL-31765, SPL-31130)

gkanapathy
Splunk Employee
Splunk Employee

There is at least one problem still in version 4.1.3 if the dispatch directory is on a (symlinked or sub-mounted) volumed separate from the Splunk config directory, that the outputlookup command will not work. this is scheduled to be fixed in a subsequent maintenance release.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...