Is it possible to monitor file system /opt disk space and trigger an alert when it reached above 75 % of the total size of the file system.

[Hemnaath]

Hi All, We have an requirement to monitor the disk space of the file system /opt and /splogs in all the splunk servers and trigger an alert if the file system reached 75 % of the total disk space.

Currently we are doing it manually by login into all the splunk servers and executing UNIX command to find out the space and clearing the top 50 directories which is consuming more space.

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_splunk01-optvol
39G 29G 8.3G 78% /opt

df -h /splogs
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_splunk01_san-splogs
5.3T 4.0T 1.1T 79% /splogs

Kindly guide me in setting up this monitoring from Splunk.

thanks in advance.

[somesoni2]

You'd probably need to install Splunk TA for Linux app https://splunkbase.splunk.com/app/833/ OR create your own data input to first capture disk space at regular interval and then setup alert based on that data.

[guilmxm]

Hi,

The Nmon performance application can do this for you. (it does this by default for all file systems already)
This is part of the JFSFILE, for file systems percent of utilisation.

https://splunkbase.splunk.com/app/1753/

Guilhem

[Hemnaath]

thanks Guilhem, we are using 6.0.3 version so is there a way to get this information by executing splunk query.

thanks in advance.

[guilmxm]

Hi Hemnaath,

The TA-nmon will work on any 6.x Universal forwarder.
The nmon core application has multiple versions for each main Splunk release (the current version is tagged for 6.5 only but there are older release compatible for each Splunk)

Only the search heads will have restrictions for the application release.
This is all explained here:

http://nmon-for-splunk.readthedocs.io/en/latest/about.html#splunk-versions

Nmon does what you want (and much more off course), there are already alerts pre-built for file system utilisation)

[gcusello]

See in Distributed Management Console alert "DMC Alert - Near Critical Disk Usage"
You can access it at the URL
https://xxx.xxx.xxx.xxx/splunk/en-EN/app/splunk_management_console/alerts

Bye.
Giuseppe

[Hemnaath]

thanks Giuseppe, but we are using splunk 6.0.3 version and I did not find the DMC console, so is there a way to write a query to fetch this information (disk space) from all splunk servers.

thanks in advance.

[gcusello]

try the search

| rest splunk_server_group=dmc_group_* /services/server/status/partitions-space 
| eval free = if(isnotnull(available), available, free) 
| eval usage = capacity - free 
| eval pct_usage = floor(usage / capacity * 100) 
| where pct_usage > 80 
| stats first(fs_type) as fs_type first(capacity) AS capacity first(usage) AS usage first(pct_usage) AS pct_usage by splunk_server, mount_point 
| eval usage = round(usage / 1024, 2) 
| eval capacity = round(capacity / 1024, 2) 
| rename splunk_server AS Instance mount_point as "Mount Point", fs_type as "File System Type", usage as "Usage (GB)", capacity as "Capacity (GB)", pct_usage as "Usage (%)"

Bye.
Giuseppe

[Hemnaath]

Hi Giuseppe, I ran the above query and got no result. Kept the duration for an hour / day, but no luck. Kindly guide me in getting the required details. thanks in advance.

[gcusello]

modify in this way

| rest  /services/server/status/partitions-space 
 | eval free = if(isnotnull(available), available, free) 
 | eval usage = capacity - free 
 | eval pct_usage = floor(usage / capacity * 100) 
 | where pct_usage > 80 
 | stats first(fs_type) as fs_type first(capacity) AS capacity first(usage) AS usage first(pct_usage) AS pct_usage by splunk_server, mount_point 
 | eval usage = round(usage / 1024, 2) 
 | eval capacity = round(capacity / 1024, 2) 
 | rename splunk_server AS Instance mount_point as "Mount Point", fs_type as "File System Type", usage as "Usage (GB)", capacity as "Capacity (GB)", pct_usage as "Usage (%)"

Bye.
Giuseppe

[Hemnaath]

thanks Giuseppe for providing the query but when I executed the above query I am getting result only for splunk component related to Index instances, not for the other components like search head, Heavy Forwarder. Is it possible to get those details in the same query.

thanks in advance.

[somesoni2]

These REST Api endpoints are for Instrospection and they were introduced in Splunk 6.2 onwards. Rest query to this enpoint will not work in lower versions.