Splunk Search

Splunk Alert when search result drop below defined threshold possibile?

vessev
Path Finder

Hello fellow splunkers,
i want to create an alert for the following search.
The search creates a statistics matrix which list the number of events from a host for the timespan defined in the search.

index=wineventlog source="WinEventLog:Security" host=testsrv1 OR host=dc* | timechart span=6h count by host limit=100

I want to define a threshold value for events in that timespan. If one of the host would drop below this threshold in my 6h timespan an alert should be triggered. There i could define a Email/SMS Messaging etc.
I've attached a picture - my goal would be to detect an unnormal behaviour like a drop or a very high peak. I'm not sure if i can have a dynamic threshold or somehting like that - but a static threshold would be good for the moment.

BR vess

Statistics
graphic

1 Solution

skoelpin
SplunkTrust
SplunkTrust

Lots and lots of ways to go about this. It all depends on skill level and complexity for what solution to implement.

If you wanted to use a static threshold, then its as simple as adding this

| eval threshold_high=150000
| eval threshold_low=1000
| eval ALERT=if(FIELD>threshold_high OR FIELD<threshold_low, "ALERT" , "GOOD")

If you wanted a more complex moving average threshold, then you can use timewrap (assuming you remove the split by host). I see you're using timechart here, this is going to add a lot of columns and also increase resource usage to monitor all these entities rather than the aggregate.

https://docs.splunk.com/Documentation/SplunkCloud/latest/SearchReference/Trendline

Option 3 is to do something more sophisticated like this which allows you to monitor each entity and have a good dynamic threshold

https://answers.splunk.com/answers/590464/how-you-detect-an-anomaly-from-a-time-frame-the-pr.html

View solution in original post

to4kawa
Ultra Champion
| tstats count where index=wineventlog source="WinEventLog:Security" host=testsrv1 OR host=dc* by _time span=6h host
| @skeolpin method.

tstats is better for this.

vessev
Path Finder

It was the combination of @skoelpin and your answer which was a success.

Thanks!

skoelpin
SplunkTrust
SplunkTrust

Lots and lots of ways to go about this. It all depends on skill level and complexity for what solution to implement.

If you wanted to use a static threshold, then its as simple as adding this

| eval threshold_high=150000
| eval threshold_low=1000
| eval ALERT=if(FIELD>threshold_high OR FIELD<threshold_low, "ALERT" , "GOOD")

If you wanted a more complex moving average threshold, then you can use timewrap (assuming you remove the split by host). I see you're using timechart here, this is going to add a lot of columns and also increase resource usage to monitor all these entities rather than the aggregate.

https://docs.splunk.com/Documentation/SplunkCloud/latest/SearchReference/Trendline

Option 3 is to do something more sophisticated like this which allows you to monitor each entity and have a good dynamic threshold

https://answers.splunk.com/answers/590464/how-you-detect-an-anomaly-from-a-time-frame-the-pr.html

vessev
Path Finder

I've merged the two answers.
But i'm not quite sure how to configure an "alert" when a specific filed has the string "alert" in it.

| tstats count where index=wineventlog source="WinEventLog:Security" host=s*wdc* OR host=dc-* by _time span=60min host  
| eval threshold_high=400000
| eval threshold_low=10
| eval ALERT=if( count>threshold_high OR count<threshold_low, "ALERT" , "GOOD")
0 Karma

skoelpin
SplunkTrust
SplunkTrust

You can create a custom alert condition so if it sees the field ALERT have a value of ALERT then fire. You can make it numerical instead, you can even have the alert fire if the number of results is greater than 0 etc.. Tons and tons of ways to make this work

vessev
Path Finder

I used:

search ALERT="ALERT"

as "custom" trigger, that worked pretty well for me.
I added | sort + ALERT to my search string which allowed me to see the important parts in the mail i've triggered.

Thanks for your help!

BR vess

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...