Splunk Dev

Why do I keep getting warning messages of 5GB even though dispatch is less than 1GB?

kamal_jagga
Contributor

Hello,

I keep getting warning messages that my dispatch directory is full (5GB) even though the dispatch dir size is less than 1 GB. And also, my queries stop running, hence I have to clean up the dispatch dir to make Splunk run again.

Kindly advise.

0 Karma

woodcock
Esteemed Legend

It is not how big the volume is, it is how much free space is left. Try this command:

df /opt/splunk/var/run/splunk/dispatch/

It will show you how big the hosting volume is, how much space is used and how much is available. The Available will be > 500M if you are getting that warning/symptoms (unless you have changed the default).

0 Karma

chrisyounger
SplunkTrust
SplunkTrust

Ah of course you are right @richgalloway ! The limit can be changed here in limits.conf if you cant make more space available dispatch_dir_warning_size = <int> https://docs.splunk.com/Documentation/Splunk/7.2.3/Admin/Limitsconf

0 Karma

richgalloway
SplunkTrust
SplunkTrust

The 5GB warning is not about size, it's about free space. It's possible for the dispatch directory to be completely empty and still get this warning because the disk doesn't have at least 5GB available.

The fix is to free up files elsewhere on the disk, not just the dispatch directory.

---
If this reply helps you, Karma would be appreciated.
0 Karma

chrisyounger
SplunkTrust
SplunkTrust

Hi @kamal_jagga

Interesting problem

  • is is possible that a large schedule job is filling the dispatch directory but then it fails so it clears the dispatch directory before you can see its true size?
  • Is it possible that the dispatch being filled is different to the server you assume it is? Maybe post the exact error message?

Good luck with your hunt

0 Karma

kamal_jagga
Contributor

Thanks for the reply. I had already checked for these things but wasn't of any help.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...