Splunk Search

Anomaly detection using historial event count with help of hour-based time slices

furkan_caliskan
New Member

Hi,

I'm currently searching for a method that will help me alerting anomalies in historial event logs.

Let's say; i've a log source that generates events. In day 1, between 09:30 - 10:00 it generates total 105 events, in day 2, between 09:30 - 10:00 it generates 110 events so on and so forth...

I want to write a search which will work every 30 minutes and will find deviation (%50 for example) from average for last 2 days for the same 30 minutes period

Example:

At 10:00, search will look for past 2 days events (value 2 can be variable) and calculate average;

Yesterday, between 09:30 - 10:00; 105 events generated

The day before, between 09:30 - 10:00; 110 events generated

Average is: (105+110)/2=107,5

If (current event count for today's 09:30 - 10:00 period) > 160, it will generate an alert since %50 deviation threshold is exceeded.

(Using version 6.2.1)
Any idea?

Thanks,

0 Karma

jwalzerpitt
Influencer

This may help as I cobbled it together (using it against our email index) using bits and pieces from posts in Splunk's answer form (and I believe some of lguinn's magic may be in here):

(Show where there is an increase in the number of events, the increase is a min of 80+% for previous hour and events > 10)

index=email _index_earliest=-h@h _index_latest=@h Subject="xxx" SenderAddress="xxx@xxx.edu" 
| timechart span=1h partial=false count 
| delta count as difference 
| eval difference=coalesce(difference,0) 
| eval percentDifference =round(abs(difference/(count - difference))*100) 
| where (difference > 1 AND percentDifference > 100)
| where count > 10 
0 Karma

lguinn2
Legend

This should work

earliest=-49h latest=-30m yoursearchcriteria
| eval endday2=relative_time(now(),"-49h+30m")
| eval  startday1=relative_time(now(),"-25h")
| eval endday1=relative_time(now(),"-25h+30m")
| eval starttoday=relative_time(now(),"-1h")
| eval Day = case(_time <= endday2,"day2",
                _time <= endday1 AND _time >= startday1,"day1",
                _time >= starttoday,"today",
                1==1,"eliminate")
| where Day!="eliminate"
| stats count by Day
| transpose
| eval avg=(day1+day2)/2
| where today > avg * 1.5

This seems like an ugly way to do what you want, but it was the first thing that crossed my mind. It isn't very flexible either.

Note that I have included a 30-minute lag in the calculations - without a lag, your numbers may not be accurate, as it often takes a minute or so for events to be detected on the remote server, forwarded, parsed and indexed.

You might want to look at the timewrap app - it's free and designed to help with these sorts of comparisons.

0 Karma

furkan_caliskan
New Member

When I run it, it gives "Error in 'eval' command: The arguments to the 'if' function are invalid."

0 Karma

lguinn2
Legend

Should be case not if! Sorry, I have edited the answer...

0 Karma

jcvytla
New Member

@Iguinn

I'm trying to use machine learning toolkit. In which assistant , shall I use the code you have posted above?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...