Knowledge Management

Best practices for writing log files that have variable number of fields

burwell
SplunkTrust
SplunkTrust

We are writing our own logs for disk usage and we are using key value pairs. The issue is that each host has a different number of disk partitions. So my logs look like the below.

We are not sure what we will do with the data yet. Maybe alert on conditions and maybe collect trending data.
What do people typically do in this case? Thanks.

2017-01-27 02:48:00 db_dt="2017-01-27 02:12:00" hostname=myhost1 vol1 = "/dev/sda1" capacity1 = "706G" percentfull1 = "9%" vol2 = "tmpfs" capacity2 = "7.6G" percentfull2 = "1%" 
2017-01-27 02:48:00 db_dt="2017-01-27 02:12:00" hostname=myhost2 vol1 = "/dev/sda1" capacity1 = "2.4G" percentfull1 = "84%" vol2 = "tmpfs" capacity2 = "24G" percentfull2 = "1%" vol3 = "/dev/sda3" capacity3 = "1.6T" percentfull3 = "1%" 
2017-01-27 02:48:00 db_dt="2017-01-27 02:12:00" hostname=myhost3 vol1 = "/dev/sda1" capacity1 = "474G" percentfull1 = "8%" vol2 = "tmpfs" capacity2 = "12G" percentfull2 = "4%"  vol4=/foo capacity4="3G" percentfull4="17%"
Tags (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi burwell,
I created for own customer a dashboard (with related alert) that shows all the servers disks, highlighting free space percentage, sending an alert when a disk has a free space percentage less that 10%.
I displayed each disk in a row, reporting in every row also servers informations (hostname, IP, etc...).
In this way I con see all disks, I can filter them and display the ones of only one server, and using colors, I can immediately see the ones in alert.
Putting every disk in a row, I haven't problems for different disks number in my server.
Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...