What are these Interesting Fields returned from the search "index=os sourcetype=iostats"?

[landen99]

Looking at:

index=os sourcetype=iostats

I come across many fields, but what do they mean?:

Interesting Fields
# avgSvcMillis 100+
# avgWaitMillis 100+
# bandwUtilPct 100+
a Device 31
# rKB_PS 100+
# rReq_PS 100+
# wKB_PS 100+
# wReq_PS 100+

After KV_MODE= multi, the raw events look like:

sdb      0.00       0.00        0.00        0.00        0.00        0.00        0.00

I am guessing that device is the mount partition, but what are the rest of the fields?

[hsesterhenn_spl]

Hi,

old question but things have changed the last couple of years and I add this just for reference:

We do have this kind of information now in the Monitoring Console and _introspection if you are running Splunk on the right OS.

http://docs.splunk.com/Documentation/Splunk/latest/DMC/ResourceusageDeployment

http://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Abouttheplatforminstrumentationfr...
http://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Whatdatagetslogged
http://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Whatdatagetslogged#I.2FO_statisti...

And because it's written in JSON format you are able to use "tstats" to get statistics fast.

Would be great to mark this question/answers as "solved"

Happy splunking,

Holger

[masonmorales · Accepted solution]

In order from left to right, the Splunk Field to iostat equivalent (and it's meaning) are:

Splunk Field = iostat column (definition)
Device = Device (This column gives the device (or partition) name.)
rReq_PS = r/s (The number of read requests that were issued to the device per second.)
wReq_PS = w/s (The number of write requests that were issued to the device per second.)
rKB_PS = rkB/s (The number of kilobytes read from the device per second.)
wKB_PS = wkB/s (The number of kilobytes written to the device per second.)
avgWaitMillis = await (The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them.)
avgSvcMillis = svctm (The average service time (in milliseconds) for I/O requests that were issued to the device.)
bandwUtilPct = %util (Percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%.)

So, in Splunk, if you have an output of:

sdb              243.00        16.00       6600.00        512.00           3.54           3.37          87.40

The fields from the output match up to:

 Device=sdb
 rReq_PS=243.00
 wReq_PS=16.00
 rKB_PS=6600.00
 wKB_PS=512.00
 avgWaitMillis=3.54
 avgSvcMillis=3.37
 bandwUtilPct=87.40

Note: I matched up the fields by looking at the TA's iostat.sh script and iostat's output.

More information about iostat output can be found at: http://perumal.org/analyzing-database-server-io-bottlenecks-using-iostat/

[woodcock]

This is the scripted input that shows you how to map the fields to the OS that you are using:

https://github.com/logscape/unixapp/blob/master/src/bin/iostat.sh

Let's say it is "linux", then you go read the man page for that command like this:

http://linux.die.net/man/1/iostat

I cannot give a definitive answer because we do not know your forwarder host OS.

[landen99]

Descriptions of those fields are not found at either link.

[jiaqya]

are these fields related to disk i/o ?

[woodcock]

I am trying to get you to do some work for yourself, because you (still) did not declare your forwarder host OS. If you go to the first link and search for avgSvcMillis, you will see which output arguments from which commands (based on which OS) are used to create this field. Once you know this, you go to the documentation for the command that matches your OS and get the documentation for it and look at the output argument that matches.