Splunk Search

Alert based on comparison between two results of the same search

JordanPeterson
Path Finder

I have the WinHostMon service reporting in every 5 minutes. I want to be alerted if a State changes from one 5 minute windows to the next. Right now I have a search that looks like this:

index="pod-os" sourcetype=WinHostMon source=service StartMode=Auto earliest=-11m 
    [ inputlookup svc.csv] 
| table host Name DisplayName Description State _time
| dedup 2 host Name
| sort host Name

Which will get me the last two reported states. Now what I want to do from here is alert if the previous State was "Stopped" and the new State is "Running". How can I compare where each host and name are the same but the state is different? I'd love to return it as a singular result to make alerting easier if possible as well.

If I can get the result like this it would be great:

|host | Name | DisplayName | Description | PreviousState | NewState |

0 Karma
1 Solution

JordanPeterson
Path Finder

I was able to get what I wanted using the dedup sortby and join commands.

index="pod-os" sourcetype=WinHostMon source=service StartMode=Auto earliest=-5m 
    [ inputlookup svc.csv] 
| table host Name DisplayName Description State 
| dedup host Name sortby -_time 
| join host Name 
   [ search index="pod-os" sourcetype=WinHostMon source=service StartMode=Auto earliest=-10m 
        [ inputlookup svc.csv] 
    | table host Name DisplayName Description State _time 
    | dedup host Name sortby +_time 
    | rename State as PrevState 
    | fields host Name PrevState ] 
| where PrevState!="Running" AND State="Running"

View solution in original post

0 Karma

JordanPeterson
Path Finder

I was able to get what I wanted using the dedup sortby and join commands.

index="pod-os" sourcetype=WinHostMon source=service StartMode=Auto earliest=-5m 
    [ inputlookup svc.csv] 
| table host Name DisplayName Description State 
| dedup host Name sortby -_time 
| join host Name 
   [ search index="pod-os" sourcetype=WinHostMon source=service StartMode=Auto earliest=-10m 
        [ inputlookup svc.csv] 
    | table host Name DisplayName Description State _time 
    | dedup host Name sortby +_time 
    | rename State as PrevState 
    | fields host Name PrevState ] 
| where PrevState!="Running" AND State="Running"
0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...