Monitoring Splunk

Why is Distributed Management Console (DMC) not displaying accurate DISK usage values?

nawazns5038
Builder

Hi,

I see that DMC is unable to give the right volume usage for a particular partition in the servers. It is showing the wrong partition value in every instance for that particular partition name.

Is there any specific permission that needs to be checked or how can this be fixed ?

   | rest splunk_server_group=dmc_group_* /services/server/status/partitions-space 

The above query results in false values than that of df -h in the server

Thanks

splunkoptimus
Path Finder

Facing the same problem in version 8.2.4, disk usage on the frontend is 92% and on the backend its 86%.

0 Karma

pellegrini
Path Finder

Same issue in version 8.0.3 and Red Hat Enterprise Linux Server release 7.9 (Maipo).

Looks like some customers have this issue but not all.

The free/available figures in the REST call are incorrect but the capacity is correct.

| rest splunk_server=* /services/server/status/partitions-space

 

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Hi @nawazns5038,

| rest splunk_server_group=dmc_group_* /services/server/status/partitions-space provide values in MB, if you run df -BM the values are similar. If you want to convert those values to GB then you can use below query.

| rest splunk_server_group=dmc_group_* /services/server/status/partitions-space | eval available=round(available/1024,2), capacity=round(capacity/1024,2), free=round(free/1024,2)

I hope this helps.

Thanks,
Harshil

0 Karma

nawazns5038
Builder

Hi @harsmarvania57 ,

The DMC alert for the CRITICAL Disk usage uses the same query and calculation,

| rest splunk_server_group=dmc_group_* /services/server/status/partitions-space 
| eval free = if(isnotnull(available), available, free) 
| eval usage = capacity - free 
| eval pct_usage = floor(usage / capacity * 100) 
| where pct_usage > 80 
| stats first(fs_type) as fs_type first(capacity) AS capacity first(usage) AS usage first(pct_usage) AS pct_usage by splunk_server, mount_point 
| eval usage = round(usage / 1024, 2) 
| eval capacity = round(capacity / 1024, 2) 
| rename splunk_server AS Instance mount_point as "Mount Point", fs_type as "File System Type", usage as "Usage (GB)", capacity as "Capacity (GB)", pct_usage as "Usage (%)"

I see almost a terabyte of difference in the values and even if the backend space in the partition gets changed, the query isn't showing the updated values.

Thanks

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Which version of splunk are you running so that I will try to replicate this issue.

0 Karma

nawazns5038
Builder

version 6.5.3

0 Karma

thambisetty
SplunkTrust
SplunkTrust

facing same issue in 6.6.4 also.

————————————
If this helps, give a like below.
0 Karma

nawazns5038
Builder

Did you raise a support case ?
They have suggested me to upgrade Splunk to 6.6+

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...