Splunk Search

Calculate difference in event counts and trigger alert if > 1

ryanchavez
Engager

Hello,

I am trying to devise a search that will basically count the number of two different log statements, and then from that search alert if their difference is > 1.

e.g. the search that returns the list of events is "Sending msg" OR "Received msg". I want to basically count the number of times "Sending msg" is found as one count, "Received msg" count as another, and alert if count( "Sending msg" ) - count( "Received msg" ) > 0 over a 5 minute window.

However there aren't any fields that I can use to count, it's just a free-text log statement. I'm struggling with the splunk syntax to accomplish this.

Any advice/help is appreciated.

Thanks,
Ryan

Tags (3)
1 Solution

ftk
Motivator

Without some sample data it's hard to tell what the best way to accomplish this would be, however I assume your events look like this:

[2011-05-05 12:12:12] Sending msg
[2011-05-05 12:13:10] Received msg

So you could create a search like the following:

[your search terms] | rex "(?<action>Sending|Received)" | stats count(eval(action="Sending")) AS SendingCount, count(eval(action="Receiving")) AS ReceivingCount | eval difference=SendingCount-ReceivingCount

And then when you schedule your search you can specify an advanced alert condition:

search difference > 0

View solution in original post

ftk
Motivator

Without some sample data it's hard to tell what the best way to accomplish this would be, however I assume your events look like this:

[2011-05-05 12:12:12] Sending msg
[2011-05-05 12:13:10] Received msg

So you could create a search like the following:

[your search terms] | rex "(?<action>Sending|Received)" | stats count(eval(action="Sending")) AS SendingCount, count(eval(action="Receiving")) AS ReceivingCount | eval difference=SendingCount-ReceivingCount

And then when you schedule your search you can specify an advanced alert condition:

search difference > 0

ryanchavez
Engager

That is exactly what I'm after. Thanks. Sorry I didn't include the log formats in my original post, but your inference was correct. I'm also reformatting these log statements so that I can use the transaction function to identify the individual transactions. Basically adding an identifier to correlate the sending & receiving log events. But this is an excellent first step. Thanks!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...