Need help in writing a query in Splunk to display the CPU Utilization of the Splunk forwarder (Universal) resources/process running on a Windows Machine

[dnvs007]

Logs i am receiving from the Windows machine are like below: (Getting just the "instance=_Total" logs)

02/09/2018 23:07:04.086 +1000
collection=CPUTime
object=Processor
counter="% User Time"
instance=_Total
Value=1.8358905012031694

Please help me write a query to address the issue ASAP.

[manish_singh_77 · Accepted solution]

Hi dnvs007,

Please try the below query and let me know if it works or not. Since we have the same index so we can use this search query..

index=os| search COMMAND="splunk*" OR process_name="splunk*" | eval CPU_Percent=coalesce(PercentProcessorTime,pctCPU) | timechart span=15m max(CPU_Percent) by host useother=f | sort _time

[manish_singh_77]

index=os sourcetype=foo OR sourcetype=coo host=$hostname$ | multikv fields pctCPU COMMAND PercentProcessorTime Name |eval pctCPU="PercentProcessorTime" | bucket span=1d _time | stats avg(Value) as pctCPU by _time host | where pctCPU>75.0| timechart span=1d avg(avgcpu) as "CPU Processor Time" by host limit=0

try this and let me know...

[dnvs007]

Thanks Manish.. Will definitely try and get back... Also just to clarify, will the multisearch usage not work?

[manish_singh_77]

It is not required..

[dnvs007]

Thanks again Manish! All i want from the query is for it to display on my dashboard the usage of Splunk resource utilization of the CPU(splunkd) on each server(Windows/Linux) in just one query...

[dnvs007]

Hi Manish,

That did not work... maybe i should have explained better..

I have multiple Linux hosts and Multiple Windows hosts..

Linux hosts come with index=os and fields pctCPU and COMMAND

However, windows hosts come with fields PercentProcessorTime and Name(which has splunkd info)

i have tokens created for all these hosts as you already know by "$hostname$

I now need a query which will show how much CPU Splunk is utilizing on these servers and display a graph on my dashboard.

Please help. Quick help much appreciated...

[dnvs007]

To Add i am using the query:
index=os sourcetype=foo OR sourcetype=coo host=$hostname$ | multikv fields pctCPU COMMAND PercentProcessorTime Name | multisearch [ search host=$hostname$ COMMAND="splunk*"] [search host=$hostname$ Name="splunk*" | eval pctCPU="PercentProcessorTime"] | timechart span=15m max(pctCPU) by host | sort _time

As posted earlier, gives me results for Linux hosts but not for Windows hosts.

Please help

[manish_singh_77]

what information you have in the below fields, could you please share..

pctCPU
COMMAND
PercentProcessorTime
Name

[richgalloway]

What have you tried so far and what results did you get?

[dnvs007]

Hello there!

Tried below and variations of below:

earliest=-15m sourcetype="Perfmon:CPU" counter="% Processor Time" | where (instance!="Idle" AND instance!="System") | stats avg(Value) by host,instance

even though does not exactly meet my requirement.

Nothing works because, i am getting logs only of "instance=_Total"

Please let me know if you need more details.

Thanks a ton! Greatly appreciate a quick response.

[richgalloway]

No SPL will help you find what is not indexed. If you need something other than instance=_Total then you should take steps to get it.

Based on what you have, you may get results with this query.

index=foo sourcetype="Perfmon:CPU" instance=_Total | stats avg(Value) by host

[dnvs007]

Thank you Rich Galloway! Sorry for the late response...

I got the processes for the Windows servers indexed. Could you please help me merge a query for Linux with Windows ?

The query i use is as below:
index=os sourcetype=foo OR sourcetype=coo host=$hostname$ | multikv fields pctCPU COMMAND | search host=$hostname$ COMMAND="splunk*" | timechart span=15m max(pctCPU) by host | sort _time

The indexed fields for windows is: PercentTime for pctCPU and Name for Command.

[dnvs007]

To add on i will need the graph to either show max values for Linux or for windows at a time in a consolidated query

[dnvs007]

Came up with this query:

index=os sourcetype=foo OR sourcetype=coo host=$hostname$ | multikv fields pctCPU COMMAND PercentProcessorTime Name | multisearch [ search host=$hostname$ COMMAND="splunk*"] [search host=$hostname$ Name="splunk*" | eval pctCPU="PercentProcessorTime"] | timechart span=15m max(pctCPU) by host | sort _time

Could someone please help in letting me know if this is the right query to get results as i need it to be?