Calculate difference in event counts and trigger alert if > 1

[ryanchavez]

Hello,

I am trying to devise a search that will basically count the number of two different log statements, and then from that search alert if their difference is > 1.

e.g. the search that returns the list of events is "Sending msg" OR "Received msg". I want to basically count the number of times "Sending msg" is found as one count, "Received msg" count as another, and alert if count( "Sending msg" ) - count( "Received msg" ) > 0 over a 5 minute window.

However there aren't any fields that I can use to count, it's just a free-text log statement. I'm struggling with the splunk syntax to accomplish this.

Any advice/help is appreciated.

Thanks,
Ryan

[ftk · Accepted solution]

Without some sample data it's hard to tell what the best way to accomplish this would be, however I assume your events look like this:

[2011-05-05 12:12:12] Sending msg
[2011-05-05 12:13:10] Received msg

So you could create a search like the following:

[your search terms] | rex "(?<action>Sending|Received)" | stats count(eval(action="Sending")) AS SendingCount, count(eval(action="Receiving")) AS ReceivingCount | eval difference=SendingCount-ReceivingCount

And then when you schedule your search you can specify an advanced alert condition:

search difference > 0

[ryanchavez]

That is exactly what I'm after. Thanks. Sorry I didn't include the log formats in my original post, but your inference was correct. I'm also reformatting these log statements so that I can use the transaction function to identify the individual transactions. Basically adding an identifier to correlate the sending & receiving log events. But this is an excellent first step. Thanks!