Deployment Architecture

Splunk indexers diskpace mismatch in monitoring console

DarshanBK
Explorer

We have  indexer cluster deployment. We recently added additional memory to all the indexers.

In splunk monitoring console the new diskspace details are not updated.

In the rest URI path '/services/server/status/partitions-space' the correct diskspace details are not updated.

We are using below SPL to fetch diskspace details in monitoring console dashboard:

 

| rest splunk_server=* /services/server/status/partitions-space
| eval free = if(isnotnull(available), available, free)
| eval usage = round((capacity - free) / 1024, 2)
| eval capacity = round(capacity / 1024, 2)
| eval compare_usage = usage." / ".capacity
| eval pct_usage = round(usage / capacity * 100, 2)
| stats first(fs_type) as fs_type first(compare_usage) AS compare_usage first(pct_usage) as pct_usage by mount_point
| rename mount_point as "Mount Point", fs_type as "File System Type", compare_usage as "Disk Usage (GB)", pct_usage as "Disk Usage (%)"

Labels (2)
0 Karma

DarshanBK
Explorer

When I run the "df" command in indexer server exact diskspace details are displayed as below. But when we run the above SPL, it is displaying the old values(2.5TB) which was before increasing, not 4TB as in indexer server .

df -h|grep splunk
 4.0T 1.6T 2.4T 41% /opt/splunk

0 Karma

richgalloway
SplunkTrust
SplunkTrust

This may be a dumb question, but did you restart the indexers after adding storage?

---
If this reply helps you, Karma would be appreciated.
0 Karma

DarshanBK
Explorer

Not sure why it seems to be dumb, but it is certainly taking our effort in understanding why the correct details are updated in rest uri path.

I think the indexer cluster was not restarted after diskpace expansion. It was before I joined the organization.

Does restating the indexer cluster helps in solving this and we would be able to see correct data in monitoring console?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

When something is not working correctly, turning it off and on again often is the fix.  Try restarting one indexer to see if it then starts reporting the right diskspace.

---
If this reply helps you, Karma would be appreciated.

richgalloway
SplunkTrust
SplunkTrust

A change in memory would not affect the diskspace measurements.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Your Guide to SPL2 at .conf24!

So, you’re headed to .conf24? You’re in for a good time. Las Vegas weather is just *chef’s kiss* beautiful in ...

Get ready to show some Splunk Certification swagger at .conf24!

Dive into the deep end of data by earning a Splunk Certification at .conf24. We're enticing you again this ...

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Now On-Demand Join us to learn more about how you can leverage Service Level Objectives (SLOs) and the new ...