Splunk Search

Average CPU Usage

mark_chuman
Path Finder

Trying to create a useful CPU usage search, but coming up short. This search does not reflect what is actually being used for some reason:

index=perfstats host=servername | timechart avg("%_Privileged_Time") by host

For example, when I compare to what the server is actually using to what the search is reporting, the search is coming back extremely low and doesn't match up with actual usage.

Tags (3)
0 Karma

MuS
SplunkTrust
SplunkTrust

Hi mark.chuman,

timechart will do aggregation on the event values, if you don't need/want that to happen use stats or chart like this:

index=perfstats host=servername | stats avg("%_Privileged_Time") AS avg_PrivTime max("%_Privileged_Time") AS max_PrivTime min("%_Privileged_Time") AS min_PrivTime by host, _time

or

index=perfstats host=servername | chart avg("%_Privileged_Time") AS avg_PrivTime max("%_Privileged_Time") AS max_PrivTime min("%_Privileged_Time") AS min_PrivTime over _time by host

Both results are basically the same like using timechart, but without the aggregation.

hope this helps ...

cheers, MuS

0 Karma

mark_chuman
Path Finder

many thanks for this and it's a big help. I'm able to use this search to effectively track the average CPU consumption for the servers, but when I say trend it out over time (7+ days for example), it doesn't look very clean as in the trend lines are actually lines, they appear more as bars.

search - index=perfstats host=servername | chart avg("%_Processor_Time") over _time by host

I'll post an image..

0 Karma

MuS
SplunkTrust
SplunkTrust

sure this is why chartdoes not aggregate like timechart does and as another pointer, did your try other formats for your chart? You can also change the behavior of the lines for example no results (connect, treat as zero ...)

0 Karma

mark_chuman
Path Finder

Almost there I think.

So, I adjusted the visualization (format, connect the dots basically), but it's weird as there is what looks to be a true CPU metric followed by a very low, inaccurate metric which makes the trend line extremely spiky and doesn't really resemble what the CPU is doing.

So, the search is pretty simple - index=perfstats host=eivc201vwin | chart avg("%_Processor_Time") over _time by host

This image shows how spiky the trend line is - http://pasteboard.co/2mtXDwFl.png

This image shows the actual sample data where you see this "piggy back" type metric - http://pbrd.co/16bamV9

You'll see a reading of say 45 followed immediately by a real low reading of 3.

Thanks again for your time!

0 Karma

MuS
SplunkTrust
SplunkTrust

You're running the script to get the cpu stats every 5min right? Your spiky graph caused by this if you decrease the interval and get more results the smoother the graph will be

0 Karma

mark_chuman
Path Finder

It is grabbing the stats every 5 minutes, but it's weird as there are two stats for every 5 minute pull. They are a second apart and these are the metrics that appear to be making the trend spiky as the second metric is always much lower than the first. For example, a metric comes in at say 9:01:25 for 45% (or 45.87326) and then a metric comes in at 9:01:26 for say 2% (or 2.34344) and then the cycle starts again at 9:06:25.

0 Karma

mark_chuman
Path Finder

not enough karma to upload image 🙂

0 Karma

MuS
SplunkTrust
SplunkTrust

you can link any picture web page like tiny url, no need to upload a picture 😉

0 Karma

mark_chuman
Path Finder

but for example, this search comes back very clean in the display

index=perfstats host=servername | timechart avg("%_Idle_Time") by host

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...