Splunk Search

How can I detect and alert on significant changes in a field value over time?

jedatt01
Builder

I want to create an alert that will trigger when the count of a certian type of event changes significantly from what it has seen in the past. See my data table below. You can see right in the middle of dataset the count jumps suddenly higher and stays there for hours. I want to be alerted when this happens, in additional I need to be able to account for datasets where the normal value is not close to 0. It may be normal to see 4000 events per time bucket but I want to know when it jumps to 8000 for example.

_time                          MESSAGE_TEXT                            msg_severity count
2016-02-10T08:00:00.000-0500    Communication with domain controller failed ERROR   7
2016-02-10T08:05:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T08:10:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T08:15:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T08:20:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T08:25:00.000-0500    Communication with domain controller failed ERROR   4
2016-02-10T08:30:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T08:35:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T08:40:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T08:45:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T08:50:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T08:55:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T09:00:00.000-0500    Communication with domain controller failed ERROR   6
2016-02-10T09:05:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T09:10:00.000-0500    Communication with domain controller failed ERROR   5
2016-02-10T09:15:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T09:20:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T09:25:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T09:30:00.000-0500    Communication with domain controller failed ERROR   3
2016-02-10T09:35:00.000-0500    Communication with domain controller failed ERROR   3
2016-02-10T09:40:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T09:45:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T09:50:00.000-0500    Communication with domain controller failed ERROR   50     <------------- start
2016-02-10T09:55:00.000-0500    Communication with domain controller failed ERROR   122
2016-02-10T10:00:00.000-0500    Communication with domain controller failed ERROR   100
2016-02-10T10:05:00.000-0500    Communication with domain controller failed ERROR   74
2016-02-10T10:10:00.000-0500    Communication with domain controller failed ERROR   93
2016-02-10T10:15:00.000-0500    Communication with domain controller failed ERROR   86
2016-02-10T10:20:00.000-0500    Communication with domain controller failed ERROR   75
2016-02-10T10:25:00.000-0500    Communication with domain controller failed ERROR   69
2016-02-10T10:30:00.000-0500    Communication with domain controller failed ERROR   67
2016-02-10T10:35:00.000-0500    Communication with domain controller failed ERROR   83
2016-02-10T10:40:00.000-0500    Communication with domain controller failed ERROR   100
2016-02-10T10:45:00.000-0500    Communication with domain controller failed ERROR   78
2016-02-10T10:50:00.000-0500    Communication with domain controller failed ERROR   89
2016-02-10T10:55:00.000-0500    Communication with domain controller failed ERROR   96
2016-02-10T11:00:00.000-0500    Communication with domain controller failed ERROR   65
2016-02-10T11:05:00.000-0500    Communication with domain controller failed ERROR   77
2016-02-10T11:10:00.000-0500    Communication with domain controller failed ERROR   74
2016-02-10T11:15:00.000-0500    Communication with domain controller failed ERROR   89
2016-02-10T11:20:00.000-0500    Communication with domain controller failed ERROR   90
2016-02-10T11:25:00.000-0500    Communication with domain controller failed ERROR   84
2016-02-10T11:30:00.000-0500    Communication with domain controller failed ERROR   63
2016-02-10T11:35:00.000-0500    Communication with domain controller failed ERROR   78
2016-02-10T11:40:00.000-0500    Communication with domain controller failed ERROR   79
2016-02-10T11:45:00.000-0500    Communication with domain controller failed ERROR   78
2016-02-10T11:50:00.000-0500    Communication with domain controller failed ERROR   56
2016-02-10T11:55:00.000-0500    Communication with domain controller failed ERROR   103
2016-02-10T12:00:00.000-0500    Communication with domain controller failed ERROR   101
2016-02-10T12:05:00.000-0500    Communication with domain controller failed ERROR   87
2016-02-10T12:10:00.000-0500    Communication with domain controller failed ERROR   90
2016-02-10T12:15:00.000-0500    Communication with domain controller failed ERROR   74
2016-02-10T12:20:00.000-0500    Communication with domain controller failed ERROR   64
2016-02-10T12:25:00.000-0500    Communication with domain controller failed ERROR   74
2016-02-10T12:30:00.000-0500    Communication with domain controller failed ERROR   84
2016-02-10T12:35:00.000-0500    Communication with domain controller failed ERROR   91
2016-02-10T12:40:00.000-0500    Communication with domain controller failed ERROR   87
2016-02-10T12:45:00.000-0500    Communication with domain controller failed ERROR   78
2016-02-10T12:50:00.000-0500    Communication with domain controller failed ERROR   61
2016-02-10T12:55:00.000-0500    Communication with domain controller failed ERROR   117
2016-02-10T13:00:00.000-0500    Communication with domain controller failed ERROR   80
2016-02-10T13:05:00.000-0500    Communication with domain controller failed ERROR   62
2016-02-10T13:10:00.000-0500    Communication with domain controller failed ERROR   40
2016-02-10T13:15:00.000-0500    Communication with domain controller failed ERROR   66
2016-02-10T13:20:00.000-0500    Communication with domain controller failed ERROR   63
2016-02-10T13:25:00.000-0500    Communication with domain controller failed ERROR   59
2016-02-10T13:30:00.000-0500    Communication with domain controller failed ERROR   79
2016-02-10T13:35:00.000-0500    Communication with domain controller failed ERROR   33
2016-02-10T13:40:00.000-0500    Communication with domain controller failed ERROR   46
2016-02-10T13:45:00.000-0500    Communication with domain controller failed ERROR   56
2016-02-10T13:50:00.000-0500    Communication with domain controller failed ERROR   70
2016-02-10T13:55:00.000-0500    Communication with domain controller failed ERROR   44
2016-02-10T14:00:00.000-0500    Communication with domain controller failed ERROR   60
2016-02-10T14:05:00.000-0500    Communication with domain controller failed ERROR   47
2016-02-10T14:10:00.000-0500    Communication with domain controller failed ERROR   63
2016-02-10T14:15:00.000-0500    Communication with domain controller failed ERROR   54
2016-02-10T14:20:00.000-0500    Communication with domain controller failed ERROR   43
2016-02-10T14:25:00.000-0500    Communication with domain controller failed ERROR   87
2016-02-10T14:30:00.000-0500    Communication with domain controller failed ERROR   48
2016-02-10T14:35:00.000-0500    Communication with domain controller failed ERROR   38
2016-02-10T14:40:00.000-0500    Communication with domain controller failed ERROR   66
2016-02-10T14:45:00.000-0500    Communication with domain controller failed ERROR   29
2016-02-10T14:50:00.000-0500    Communication with domain controller failed ERROR   72
2016-02-10T14:55:00.000-0500    Communication with domain controller failed ERROR   87
2016-02-10T15:00:00.000-0500    Communication with domain controller failed ERROR   40
2016-02-10T15:05:00.000-0500    Communication with domain controller failed ERROR   45
2016-02-10T15:10:00.000-0500    Communication with domain controller failed ERROR   61
2016-02-10T15:15:00.000-0500    Communication with domain controller failed ERROR   43
2016-02-10T15:20:00.000-0500    Communication with domain controller failed ERROR   52
2016-02-10T15:25:00.000-0500    Communication with domain controller failed ERROR   44
2016-02-10T15:30:00.000-0500    Communication with domain controller failed ERROR   53
2016-02-10T15:35:00.000-0500    Communication with domain controller failed ERROR   61
2016-02-10T15:40:00.000-0500    Communication with domain controller failed ERROR   62 <-------- end
2016-02-10T15:45:00.000-0500    Communication with domain controller failed ERROR   3
2016-02-10T15:50:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T15:55:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T16:00:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T16:05:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T16:10:00.000-0500    Communication with domain controller failed ERROR   4
2016-02-10T16:15:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T16:20:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T16:25:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T16:30:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T16:35:00.000-0500    Communication with domain controller failed ERROR   9
2016-02-10T16:40:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T16:45:00.000-0500    Communication with domain controller failed ERROR   1
2016-02-10T16:50:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T16:55:00.000-0500    Communication with domain controller failed ERROR   10
2016-02-10T17:00:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T17:05:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T17:10:00.000-0500    Communication with domain controller failed ERROR   5
2016-02-10T17:15:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T17:20:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T17:25:00.000-0500    Communication with domain controller failed ERROR   9
2016-02-10T17:30:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T17:35:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T17:40:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T17:45:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T17:50:00.000-0500    Communication with domain controller failed ERROR   12
2016-02-10T17:55:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T18:00:00.000-0500    Communication with domain controller failed ERROR   1
2016-02-10T18:05:00.000-0500    Communication with domain controller failed ERROR   2
2016-02-10T18:10:00.000-0500    Communication with domain controller failed ERROR   10
2016-02-10T18:15:00.000-0500    Communication with domain controller failed ERROR   7
2016-02-10T18:20:00.000-0500    Communication with domain controller failed ERROR   0
2016-02-10T18:25:00.000-0500    Communication with domain controller failed ERROR   5
2016-02-10T18:30:00.000-0500    Communication with domain controller failed ERROR   0
0 Karma
1 Solution

sundareshr
Legend

See if this works for you (not sure if you need the MESSAGE_TEXT grouping)

.... | eventstats stdev(count) as stdev by MESSAGE_TEXT | eval high=if(count>stdev, "Yes", "No") | eval low=if(count<2, "Yes", "No") | table _time count high low

View solution in original post

0 Karma

sundareshr
Legend

See if this works for you (not sure if you need the MESSAGE_TEXT grouping)

.... | eventstats stdev(count) as stdev by MESSAGE_TEXT | eval high=if(count>stdev, "Yes", "No") | eval low=if(count<2, "Yes", "No") | table _time count high low
0 Karma

jedatt01
Builder

This works pretty well.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...