First time posting! --using splunk 4.2.4--
I noticed similar questions on here that were either unanswered or didn't quite meet my needs. I had quite the time getting this far so thought I'd share and also solicit feedback that could improve on this. I'll also note that the logs I'm working with are really messy and inconsistent.
There are really 3 parts:
1) get 2 unique values from a search so you can do some math with them
2) in that same search, calculate a percentage from these 2 values
3) using this search trigger an alert with a 3rd condition
For my needs, a static number of errors to alert on just wouldn't cut it due to large fluctuations in the number of users. Using a ratio of errors/users (in the last X hrs) was preferable.
sourcetype="myAppHosts" "SOME SPECIFIC ERROR EVENT STRING"
| stats count as error_total
| eval user_total = [search "SOME UNIQUE USER LOGIN EVENT STRING" | stats count as search]
| eval percent = (error_total/user_total)*100
| fields percent error_total user_total
We then create an alert validating a custom condition against the calculated percentage...
search percent > 10
The resulting email provides the 3 fields in the last part of the search.
This could be modified to use some regex in place of string searches (to dedup users,etc..), but it works for my needs as is. If there is a better way to do this, I'm all ears.
Cheers.
... View more