Splunk Search

How can I compare 2 latest events and search for the second latest event by host?

packland
Path Finder

Hi,

I'd like to create a search that detects a failover, i.e. it would compare the two latest events by host and when the second last is true, but the last is false it would appear in the results.

For example if I have data coming from 5 hosts, each of which has a single true or false value sent every 2 - 3 minutes, in my case indicating connectivity, I'd like to know pretty much as soon as the first false message comes through. The closest I've come is using the "latest" function to find the latest event and then maybe using a subsearch to find the second latest and compare the two, however as far as I can find there isn't a way to search for the second latest event (like a "latest(up) - 1" kind of thing).

I'm open to doing it in a different way if this isn't the right way to go about it. I know I could use transactions and search for a "false" transaction that is only 1 - 2 events long, however I feel like that's an expensive way of accomplishing something that might be much simpler.

Any suggestions would be great, thanks.

0 Karma
1 Solution

micahkemp
Champion

One potential option is to dedup by host, and keep 2 events per host. These should be the most recent event. Once you have those, you can use earliest/latest against the state, and then search to find the hosts that changed from active to standby:

| makeresults
| append [| makeresults | eval _time=1002, host="A", state="standby"]
| append [| makeresults | eval _time=1001, host="B", state="active"]
| append [| makeresults | eval _time=1000, host="A", state="active"]
| append [| makeresults | eval _time=999, host="B", state="active"]
| append [| makeresults | eval _time=998, host="A", state="standby"]
| append [| makeresults | eval _time=997, host="A", state="active"]
| append [| makeresults | eval _time=996, host="A", state="active"]
| append [| makeresults | eval _time=995, host="B", state="standby"]
| append [| makeresults | eval _time=994, host="B", state="standby"]
| dedup 2 host
| stats earliest(state) AS earliest_state, latest(state) AS latest_state BY host
| search earliest_state=active latest_state=standby

View solution in original post

micahkemp
Champion

One potential option is to dedup by host, and keep 2 events per host. These should be the most recent event. Once you have those, you can use earliest/latest against the state, and then search to find the hosts that changed from active to standby:

| makeresults
| append [| makeresults | eval _time=1002, host="A", state="standby"]
| append [| makeresults | eval _time=1001, host="B", state="active"]
| append [| makeresults | eval _time=1000, host="A", state="active"]
| append [| makeresults | eval _time=999, host="B", state="active"]
| append [| makeresults | eval _time=998, host="A", state="standby"]
| append [| makeresults | eval _time=997, host="A", state="active"]
| append [| makeresults | eval _time=996, host="A", state="active"]
| append [| makeresults | eval _time=995, host="B", state="standby"]
| append [| makeresults | eval _time=994, host="B", state="standby"]
| dedup 2 host
| stats earliest(state) AS earliest_state, latest(state) AS latest_state BY host
| search earliest_state=active latest_state=standby

packland
Path Finder

Did not realise you could use dedup like that. Thanks!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...