Monitoring Splunk

Why is the CPU usage showing 80-100% (falsely)? -- Search help

lbalaur
Explorer

I am trying to get CPU usage for a specific process in windows. My search looks like this:

host=host1 AND sourcetype="Perfmon:Process"  AND  counter="% Processor Time" AND process_name="server*" | table _time, counter, process_name, Value

My result is showing mostly 100 for Value which is not really true. Windows runs on VM.

Result looks like this:

2017-09-22T14:40:28.000-0400    % Processor Time    server  100
2017-09-22T14:39:43.000-0400    % Processor Time    server  100
2017-09-22T14:37:28.000-0400    % Processor Time    server  100
2017-09-22T14:32:58.000-0400    % Processor Time    server#1    100
2017-09-22T14:32:13.000-0400    % Processor Time    server  100
2017-09-22T14:38:13.000-0400    % Processor Time    server  100
2017-09-22T14:31:28.000-0400    % Processor Time    server#1    11.30968265
2017-09-22T14:21:43.000-0400    % Processor Time    server  100
2017-09-22T14:18:43.000-0400    % Processor Time    server#1    0.105369743
2017-09-22T14:36:43.000-0400    % Processor Time    server  0.034732856
2017-09-22T14:35:58.000-0400    % Processor Time    server#1    0.14049302
2017-09-22T14:29:13.000-0400    % Processor Time    server  100
2017-09-22T14:28:28.000-0400    % Processor Time    server#1    84.84122861
2017-09-22T14:20:58.000-0400    % Processor Time    server#1    100
2017-09-22T14:16:28.000-0400    % Processor Time    server  100
2017-09-22T14:14:58.000-0400    % Processor Time    server#1    100

What should I do? Why is it pulling all the 100s? 80% of events show a 100. Is it an agent config issue?

0 Karma

tmarlette
Motivator

Perfmon is not very intuitive when watching processor utilization per process. I'm going to assume you're watching process utilization on a multi-processor machine. you may want to use 'System: %Total Processor Time' instead.

Here's some documentation:
https://docs.microsoft.com/en-us/sql/relational-databases/performance-monitor/monitor-cpu-usage

If you are monitoring a multi-processor machine, and if you're using this counter to monitor CPU utilization, then you're only watching a single thread. generally speaking, 1 thread = 1 core of the machine, so on a 16 core machine 1 core at 100%, is quite normal. When you have 10 instances of 'server#1' running at 100%, you would likely see performance degradation.

natalienguyen
Explorer

thanks @tmarlette, I was having the same issue and the %Total Processor Time helped!

0 Karma

lbalaur
Explorer

If System: %Total Processor Time is not showing up in splunk - I need to ask my admins to add this statistic then?

0 Karma

tmarlette
Motivator

Yeah, you would need to as your Splunk admins to add that counter in the Splunk_TA_windows app.

0 Karma

Sukisen1981
Champion

hi - cpu usage (or utilization) has to be over time, so what happens if you try something like this-
host=host1 AND sourcetype="Perfmon:Process" AND counter="% Processor Time" AND process_name="server*" | timechart avg(value)
OR
host=host1 AND sourcetype="Perfmon:Process" AND counter="% Processor Time" AND process_name="server*" | timechart avg(value) by process_name

if this works and you just need other fields,....you just need to eval them AFTER the timechart command or use a simple appendcols in this case

0 Karma

lbalaur
Explorer

If you do a timechart how are all those false 100s going to affect your average? My concern is not building the chart but underlying data. Based on my observations usage never reached a 100% but splunk keeps bringing it in.
I tested the timechart, btw, and it's staying above 50% on average when in fact it should be around 10%

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...