Splunk Search

Monitor specific set of Processes from WMI

joshhenderson
Explorer

Hi,

What I'm attempting to do is monitor a specific set of processes on a machine. For this, I am obtaining data from WMI (where wmi_type=LocalProcesses,) and checking how many instances of that a specific process has occurred within a specified time frame (which I choose depening on how often the data from that server gets updated.) If it is ever zero, then there is an error.

Below is a search query that I am using on a dashboard:

host=[HOST] wmi_type=LocalProcesses earliest=-5m Name="[PROCESS]" | stats count AS TimesDetected | rangemap field=TimesDetected severe=0-0 default=low

(Where [HOST] is the name of the particular machine I am checking for, and [PROCESS] is the name of the process. An example would be Server01 and CcmExec respectively.)

This works quite well, and I can add text only panels to my dashboard (within the search app) to monitor each process.

The problem with this is that, once more than a handful of processes are monitored, the dashboard get's very cluttered (since you can only have 2 panels per row, and each process takes up a panel.) I am looking for a way to display the data in a table, for only certain processes that I have specified.

In an attempt to do this, I have come up with the following query:

host=HTVMBI02 wmi_type=LocalProcesses earliest=-5m Name="System" OR "CcmExec" | stats count AS TimesDetected by Name

It works well when viewed as a table, and I can add as many processes as I want to monitor to the table. However, should the process not have any data, it will not be shown in the resulting table. I can understand why this is (there is no data for it,) but I am looking for a way to force it to say '0' for processes I've specified that it can't find data for. Is there a way to do this?

Thanks.

1 Solution

dwaddle
SplunkTrust
SplunkTrust

One approach (an obviously not optimal one, but perhaps it will scale sufficiently) might be to use a lookup table and | append.

Suppose you set up a lookup table like this (call it processes.csv):

Name,TimesDetected
System,0
CcmExec,0

Now, you can make your search as follows:

host=HTVMBI02 wmi_type=LocalProcesses earliest=-5m 
[ | inputlookup processes.csv | fields Name ] 
| stats count AS TimesDetected by Name
| append [ | inputlookup processes.csv ]
| stats max(TimesDetected) as TimesDetected by Name

In theory, the lookup not only populates the base search but provides sentinel values to make it so you can be sure that every possible row has a "TimesDetected=0" value for display purposes.

View solution in original post

dwaddle
SplunkTrust
SplunkTrust

One approach (an obviously not optimal one, but perhaps it will scale sufficiently) might be to use a lookup table and | append.

Suppose you set up a lookup table like this (call it processes.csv):

Name,TimesDetected
System,0
CcmExec,0

Now, you can make your search as follows:

host=HTVMBI02 wmi_type=LocalProcesses earliest=-5m 
[ | inputlookup processes.csv | fields Name ] 
| stats count AS TimesDetected by Name
| append [ | inputlookup processes.csv ]
| stats max(TimesDetected) as TimesDetected by Name

In theory, the lookup not only populates the base search but provides sentinel values to make it so you can be sure that every possible row has a "TimesDetected=0" value for display purposes.

joshhenderson
Explorer

Thanks, worked really well. I was on the right track (very close actually) as I was looking at lookup tables, but had issues uploading it (which I've since resolved) and I wasn't putting a 0 value in the second column. Thanks for the help.

0 Karma
Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...