Knowledge Management

Sending conditional alerts based on previous search result

technie101
Explorer

Following is the json log format being stored in Splunk.

{   data:[
     {
       "endpoint":"ep_1",
       "service":"service_1",
       "http_status_code":"500"
     },
     {
       "endpoint":"ep_2",
       "service":"service_1",
       "http_status_code":"500"
     },
     {
       "endpoint":"ep_3",
       "service":"service_2",
       "http_status_code":"503"
     }   ] }

Currently, an alert is setup to check once every 5 mins with the following search query and send an email if results>0.

host=something "data{}.http_status_code"= 5*| eval endpoint='data{}.endpoint' | Stats Count("data{}.http_status_code") as Count by endpoint | where Error_Count > 0

The generated result is :

alt text

Now, the ask is, once an alert is generated (which contains a particular endpoin, say ep_1), in the next run, if the search result contains the same endpoint, it should NOT send an email for another hour.

So in this case, when the search query runs after 5 mins and contains ep_1 in the result, it should not send an email. So effectively, I need a condition for this endpoint's occurrence in the previously executed search result.

Is there a way this can be accomplished? Please help.

0 Karma
1 Solution

s2_splunk
Splunk Employee
Splunk Employee

The only way to accomplish this that I can think of is to save the results of your alerting search in a lookup file (using outputlookup) or the KVStore and use the contents of that lookup file/KVStore collection to filter out the unwanted endpoints (ie. all the ones in the lookup file from the last run). You'll have to include last alert timestamp per endpoint as a column in your lookup and make timestamp comparison part of your filter criteria.

View solution in original post

s2_splunk
Splunk Employee
Splunk Employee

The only way to accomplish this that I can think of is to save the results of your alerting search in a lookup file (using outputlookup) or the KVStore and use the contents of that lookup file/KVStore collection to filter out the unwanted endpoints (ie. all the ones in the lookup file from the last run). You'll have to include last alert timestamp per endpoint as a column in your lookup and make timestamp comparison part of your filter criteria.

technie101
Explorer

Thanks. Trying this approach with a KVStore now.

For the above usecase, how do we compare the endpoint from the search results with the endpoint from the KVStore?

0 Karma

inventsekar
Ultra Champion
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...