Diskmon warning regarding available disk space for index partition

[noahzstahl]

I'm receiving several warning events per minute in my splunkd log as follows:

---

WARN Diskmon - Potential performance issue: getting available disk space for partition "C:\Program Files\Splunk\var\lib\splunk/summary_indexers/db" from the file system directly.

WARN Diskmon - Potential performance issue: getting available disk space for partition "C:\Program Files\Splunk\var\lib\splunk/summary_fowarders/db" from the file system directly.

---

The indexer and forwarder summary indexes are the only two mentioned. The size of their directory structures is small, and the disk has 130 GB free. I have run a fsck repair.

Any ideas on what this means and how to resolve?

Thanks,

Noah

[hexx]

This is a bug and has been filed under references SPL-44955/SPL-45990. It is fixed in release 4.3.1.

This message indicates that we will make an extra system call that asks for available disk space on a partition but it is a bit too alarming and shouldn't spam splunkd.log at the currently reported rate.

Based on investigation, the repeated "Potential performance issue" message occurs most of the time when block signature is used.

In conclusion : If you are seeing this error on a version of Splunk < 4.3.1, you can safely ignore and get rid of it by upgrading to 4.3.1.

[hexx]

The bugs I mentioned are indeed fixed in 4.3.1 and we haven't heard of re-occurrences since. You might want to have a support case opened to look at these errors if they persist beyond 4.3.1.

[cphair]

@hexx, was this actually resolved in 4.3.1? I still see similarly spammy messages in 4.3.4 (136012).

[hexx]

As of right now, the tentative release date for 4.3.1 is late February/mid-March.

[tmeader]

Looking forward to getting this fixed. Is there any ETA on the 4.3.1 release at all?

[sdwilkerson]

Hirsts,
This original question was noahzstahl's so you probably should have started a new question.
However, I will attempt to give you some quick direction here.
All of the erors you show reference the directories/files used by the Splunk indexes mostly those used for the Splunk Deployment Monitor. I think issue is Splunk has issue with the Filesystem/OS. I have used Splunk for 4 years on dozens of sites and never seen this AFAIK. If you have DeploymentServer then you are Enterprise customer. I suggest you call Support and open a ticket.

[hirsts]

The characteristics are as below:

Physical Server:

Intel Xeon
L5420 @ 2.50 GHz
8GB Ram

Windows Server 2003
Enterprise X64 Edition
Service Pack 2

Splunk Version = Splunk 4.2.4 (build 110225)

The one server is acting as Indexer, Search Head and Deployment Server however this is a development environment and so theres very little traffic.

The Windows 2003 performance tools show very little IOPS

Splunk on Splunk highlights that there are 670 messages of this type every hour. Below is a sample:

b" from the file system directly.
11-24-2011 18:52:37.718 +1100 WARN DiskMon - Potential performance issue: getting available disk space for partition "C:\Program Files\Splunk\var\lib\splunk/summary_sources/db" from the file system directly.
11-24-2011 18:52:40.734 +1100 WARN DiskMon - Potential performance issue: getting available disk space for partition "C:\Program Files\Splunk\var\lib\splunk/summary_hosts/db" from the file system directly.
11-24-2011 18:52:40.734 +1100 WARN DiskMon - Potential performance issue: getting available disk space for partition "C:\Program Files\Splunk\var\lib\splunk/summary_pools/db" from the file system directly.
11-24-2011 18:52:42.718 +1100 WARN DiskMon - Potential performance issue: getting available disk space for partition "C:\Program Files\Splunk\var\lib\splunk/summary_sourcetypes/db" from the file system directly.
11-24-2011 18:52:47.625 +1100 WARN DiskMon - Potential performance issue: getting available disk space for partition "C:\Program Files\Splunk\var\lib\splunk/os/db" from the file system directly.
11-24-2011 18:52:53.468 +1100 WARN DiskMon - Potential performance issue: getting available disk space for partition "C:\Program Files\Splunk\var\lib\splunk/summary_forwarders/db" from the file system directly.
11-24-2011 18:53:03.468 +1100 WARN DiskMon - Potential performance issue: getting available disk space for partition "C:\Program Files\Splunk\var\lib\splunk/summary_indexers/db" from the file system directly.
11-24-2011 18:53:08.375 +1100 WARN DiskMon - Potential performance issue: getting available disk space for partition "C:\Program Files\Splunk\var\lib\splunk/os/db" from the file system directly.
11-24-2011 18:53:19.218 +1100 WARN DiskMon - Potential performance issue: getting available disk space for partition "C:\Program Files\Splunk\var\lib\splunk/summary_sources/db" from the file system directly.
11-24-2011 18:53:22.218 +1100 WARN DiskMon - Potential performance issue: getting available disk space for partition "C:\Program Files\Splunk\var\lib\splunk/summary_hosts/db" from the file system directly.
11-24-2011 18:53:22.234 +1100 WARN DiskMon - Potential performance issue: getting available disk space for partition "C:\Program Files\Splunk\var\lib\splunk/summary_pools/db" from the file system directly.
11-24-2011 18:53:24.218 +1100 WARN DiskMon - Potential performance issue: getting available disk space for partition "C:\Program Files\Splunk\var\lib\splunk/summary_sourcetypes/db" from the file system directly.
11-24-2011 18:53:35.218 +1100 WARN DiskMon - Potential performance issue: getting available disk space for partition "C:\Program Files\Splunk\var\lib\splunk/summary_forwarders/db" from the file system directly.
11-24-2011 18:53:45.218 +1100 WARN DiskMon - Potential performance issue: getting available disk space for partition "C:\Program Files\Splunk\var\lib\splunk/summary_indexers/db" from the file system directly.
11-24-2011 18:53:49.265 +1100 WARN DiskMon - Potential performance issue: getting available disk space for partition "C:\Program Files\Splunk\var\lib\splunk/os/db" from the file system directly.
11-24-2011 18:54:01.218 +1100 WARN DiskMon - Potential performance issue: getting available disk space for partition "C:\Program Files\Splunk\var\lib\splunk/summary_sources/db" from the file system directly.
11-24-2011 18:54:04.218 +1100 WARN DiskMon - Potential performance issue: getting available disk space for partition "C:\Program Files\Splunk\var\lib\splunk/summary_hosts/db" from the file system directly.
11-24-2011 18:54:04.218 +1100 WARN DiskMon - Potential performance issue: getting available disk space for partition "C:\Program Files\Splunk\var\lib\splunk/summary_pools/db" from the file system directly.
11-24-2011 18:54:06.218 +1100 WARN DiskMon - Potential performance issue: getting available disk space for partition "C:\Program Files\Splunk\var\lib\splunk/summary_sourcetypes/db" from the file system directly.
11-24-2011 18:54:10.312 +1100 WARN DiskMon - Potential performance issue: getting available disk space for partition "C:\Program Files\Splunk\var\lib\splunk/os/db" from the file system directly.
11-24-2011 18:54:17.218 +1100 WARN DiskMon - Potential performance issue: getting available disk space for partition "C:\Program Files\Splunk\var\lib\splunk/summary_forwarders/db" from the file system directly.

The only input.conf entries that I can find that might initiate local disk monitoring are in $SPLUNK_HOME\etc\system\default\inputs.conf:

[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.path]
disabled = 0
interval = 10000000
source = wmi
sourcetype = wmi
queue = winparsing
persistentQueueSize=200MB

[script://$SPLUNK_HOME\bin\scripts\splunk-regmon.path]
disabled = 0
interval = 10000000
source = WinRegistry
sourcetype=WinRegistry
queue = winparsing
persistentQueueSize=50MB

[script://$SPLUNK_HOME\bin\scripts\splunk-admon.path]
interval = 10000000
source = ActiveDirectory
sourcetype = ActiveDirectory
disabled = 0
queue = winparsing
persistentQueueSize=50MB

[script://$SPLUNK_HOME\bin\scripts\splunk-perfmon.path]
interval = 10000000
source = PerformanceMonitor
sourcetype = PerformanceMonitor
disabled = 0
queue = winparsing
persistentQueueSize=50MB

Apps that I've added to my implementation are:

*nix 4.5 (I've disabled all inputs)
JMX (not used it yet)
S.o.S (To try and resolve this issue)
Sideview Utils (needed by S.o.S)

Is a shame the splunkd.log does not reference the source that triggered the event.

Any ideas ?

[sdwilkerson]

Can you edit your question and provide the following:
- Version of Splunk (e.g. 4.2.4 64bit)
- Windows version and arch?
- Storage type (e.g. Internal, DAS, SAN, NAS)
- Have you run any utilities (e.g. sysinternals) that measure and report on system resources specifically storage IOPS (Input/Output Operations per Second)?

[sdwilkerson]

This should not have been posted as an "Answer" but should have been a comment of the original question.

[hirsts]

Did you resolve this, I have the same issue and struggling to fix.