Why am I near critical disk alert on Indexer nodes?

[narenpalepu]

Hi ,

I am using following( default) query for near critical disk alert on Indexer nodes. The daily results are showing 99% where as actual disk usage is much lower. Can you help clarify. I will submit the actual support contract later.

Thanks,

Naren

| rest splunk_server_group=dmc_group_* /services/server/status/partitions-space 
| eval free = if(isnotnull(available), available, free) 
| eval usage = capacity - free 
| eval pct_usage = floor(usage / capacity * 100) 
| where pct_usage > 92 
| stats first(fs_type) as fs_type first(capacity) AS capacity first(usage) AS usage first(pct_usage) AS pct_usage by splunk_server, mount_point 
| eval usage = round(usage / 1024, 2) 
| eval capacity = round(capacity / 1024, 2) 
| rename splunk_server AS Instance mount_point as "Mount Point", fs_type as "File System Type", usage as "Usage (GB)", capacity as "Capacity (GB)", pct_usage as "Usage (%)"

Alert search Results:
Instance        Mount Point     File System Type        Capacity (GB)   Usage (GB)      Usage (%)
prd-sjc-splunk-indexer-1 /opt/colddb ext4 14881.80 14239.33 95
prd-sjc-splunk-indexer-2 /opt/colddb ext4 14881.80 14523.47 97 
prd-sjc-splunk-indexer-3 /opt/colddb ext4 14881.80 14664.28 98
prd-sjc-splunk-indexer-4 /opt/colddb ext4 14881.80 14845.24 99
prd-sjc-splunk-indexer-5 /opt/colddb ext4 14881.80 14612.96 98
prd-sjc-splunk-indexer-6 /opt/colddb ext4 14881.80 14744.09 99

Actual Disk space:
Processing on prd-sjc-splunk-indexer-2 

/dev/mapper/hot-hot                 10403135808  7814205760 2064642256  80% /opt/splunk
/dev/mapper/cold-cold               15604702004 10252296568 4565973644  70% /opt/colddb


Processing on prd-sjc-splunk-indexer-1:

/dev/mapper/hot-hot                 10403135808 7956960136 1921887880  81% /opt/splunk
/dev/mapper/cold-cold               15604702004 9749420004 5068850208  66% /opt/colddb


Processing on prd-sjc-splunk-indexer-5

/dev/xvdg                           10403139904 7912240516 1966611388  81% /opt/splunk
/dev/mapper/colddb-colddb           15604697908 9752163196 5066103124  66% /opt/colddb

Processing on prd-sjc-splunk-indexer-3:

/dev/mapper/hot-hot                 10403135808 7865249624 2013598392  80% /opt/splunk
/dev/mapper/cold-cold               15604702004 9997688028 4820582184  68% /opt/colddb

Processing on prd-sjc-splunk-indexer-4

/dev/mapper/cold-colddb             15604697908 10681093532 4137236996  73% /opt/colddb
/dev/mapper/hot-hotdb               10403135808  7779513904 2099334112  79% /opt/splunk

Processing on prd-sjc-splunk-indexer-6

/dev/xvdg                           10321219904  7766226312 2030705592  80% /opt/splunk
/dev/mapper/colddb-colddb           15604697908 10138947556 4679318764  69% /opt/colddb

Processing on prd-sjc-splunk-indexer-7

/dev/xvdg                           10403139904 7783278992  2095572912  79% /opt/splunk
/dev/xvdh                           17111506844 1754748380 14497765008  11% /opt/colddb

---

updated to mark code and data as code

[pmalcakdoj]

Splunk reports the available space as: Total Size - (Used + Reserved)
By default, ext3/4 filesystem dedicates 5% of total space to be reserved.
So if your partition total size is 1TB and the partition is completely empty, Splunk (DMC) will report it as 50GB out of 1000GB used.

[DalJeanis]

The first thing I note is that the issues are only on your colddb.

Run this to see if the available vs free numbers look different on the cold vs the hot mountpoints.

 | rest splunk_server_group=dmc_group_* /services/server/status/partitions-space 
 | eval usage = capacity - coalesce(available,free) 
 | eval pct_usage = floor(100*usage/capacity) 
 | where pct_usage > 80 
 | stats first(fs_type) as fs_type first(capacity) AS capacity first(usage) AS usage first(pct_usage) AS pct_usage 
first(free) as free first(available) as available by splunk_server, mount_point 
 | eval usage = round(usage / 1024, 2) 
 | eval capacity = round(capacity / 1024, 2) 
 | eval free = round(free / 1024, 2) 
 | eval available = round(available / 1024, 2) 
 | rename splunk_server AS Instance mount_point as "Mount Point", fs_type as "File System Type", usage as "Usage (GB)", capacity as "Capacity (GB)", pct_usage as "Usage (%)", free as "free (GB)", available as "available(GB)"

[narenpalepu]

Thanks . Results are same as before.

[narenpalepu]

Is there any other query we can run?. I am trying to access the license info from my dmc . Can you point me to a doc to establish support account. So that I can convert this to support ticket.