Splunk Search

Alert based on comparison between two results of the same search

JordanPeterson
Path Finder

I have the WinHostMon service reporting in every 5 minutes. I want to be alerted if a State changes from one 5 minute windows to the next. Right now I have a search that looks like this:

index="pod-os" sourcetype=WinHostMon source=service StartMode=Auto earliest=-11m 
    [ inputlookup svc.csv] 
| table host Name DisplayName Description State _time
| dedup 2 host Name
| sort host Name

Which will get me the last two reported states. Now what I want to do from here is alert if the previous State was "Stopped" and the new State is "Running". How can I compare where each host and name are the same but the state is different? I'd love to return it as a singular result to make alerting easier if possible as well.

If I can get the result like this it would be great:

|host | Name | DisplayName | Description | PreviousState | NewState |

0 Karma
1 Solution

JordanPeterson
Path Finder

I was able to get what I wanted using the dedup sortby and join commands.

index="pod-os" sourcetype=WinHostMon source=service StartMode=Auto earliest=-5m 
    [ inputlookup svc.csv] 
| table host Name DisplayName Description State 
| dedup host Name sortby -_time 
| join host Name 
   [ search index="pod-os" sourcetype=WinHostMon source=service StartMode=Auto earliest=-10m 
        [ inputlookup svc.csv] 
    | table host Name DisplayName Description State _time 
    | dedup host Name sortby +_time 
    | rename State as PrevState 
    | fields host Name PrevState ] 
| where PrevState!="Running" AND State="Running"

View solution in original post

0 Karma

JordanPeterson
Path Finder

I was able to get what I wanted using the dedup sortby and join commands.

index="pod-os" sourcetype=WinHostMon source=service StartMode=Auto earliest=-5m 
    [ inputlookup svc.csv] 
| table host Name DisplayName Description State 
| dedup host Name sortby -_time 
| join host Name 
   [ search index="pod-os" sourcetype=WinHostMon source=service StartMode=Auto earliest=-10m 
        [ inputlookup svc.csv] 
    | table host Name DisplayName Description State _time 
    | dedup host Name sortby +_time 
    | rename State as PrevState 
    | fields host Name PrevState ] 
| where PrevState!="Running" AND State="Running"
0 Karma
Get Updates on the Splunk Community!

Introducing Splunk Enterprise 9.2

WATCH HERE! Watch this Tech Talk to learn about the latest features and enhancements shipped in the new Splunk ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...