Splunk Search

Throttle Alerts for a table of results until end of the current day

njohnson7
Path Finder

I am trying to setup an alert which will run every hour and considers the data from the start of current day(earliest=@d) and send the summary of table results via email. As long as the first set of results are produced I would like to throttle it until end of the current day.

When I select the trigger option ONCE , i have the values like hours, days etc for which if select 24 or 1 respectively , I am afraid my alert will trigger again only after 24 hours or a day from the time it was triggered last.

When I evaluate a Date field and try to throttle it using the PER RESULT option and using Date as a suppressing field, the alert email only has the first line & the Grand Total line from my table summary - since my table of results are lost , I cant use this method.

Could you please advise a way to keep my table result structure while letting me to throttle until end of the current day.

Desired Result:

      SYSTEM      Qname        FAIL         TRYAGAIN  Grand Total
      sys1         ABC*        300            25        325
      sys1         DEF*        210            55        265
      sys1         GHI*        470            25        495
      sys1         JK*         250            25        275
               Grand Total     1230           130       1360

Query :

index=xyz sourcetype="abc" SYSTEM="sys1" 
| dedup client NAME STATUS 
| search STATUS=FAIL OR STATUS=TRYAGAIN 
| eval "Qname"=case(NAME like "ABC%" , "ABC*", NAME like "DEF%" ,"DEF*", NAME like "GHI%" , "GHI*" , NAME like "JK%","JK*", 1=1,"Others") 
| search "Qname"!=Others 
| eventstats count as Grandtotal 
| where Grandtotal >50 
| eval Date=strftime(_time, "%d/%m/%Y") 
| stats count(eval(STATUS="FAIL")) as "FAIL" , count(eval(STATUS="TRYAGAIN")) as "TRYAGAIN" values(Date) as Date by SYSTEM, "Qname" 
| table Date SYSTEM "Qname" FAIL TRYAGAIN
| addtotals fieldname="Grand Total" col=t row=t labelfield="Qname" label="Grand Total"
0 Karma

jacobpevans
Motivator

When I evaluate a Date field and try to throttle it using the PER RESULT option and using Date as a suppressing field, the alert email only has the first line & the Grand Total line from my table summary - since my table of results are lost , I cant use this method.

I'm not fully following this. Could you try doing that again but with _time to now() and values(Date) as Date by SYSTEM, "Qname" to by Date SYSTEM "Qname" e.g.:

 index=xyz sourcetype="abc" SYSTEM="sys1" 
 | dedup client NAME STATUS 
 | search STATUS=FAIL OR STATUS=TRYAGAIN 
 | eval "Qname"=case(NAME like "ABC%" , "ABC*", NAME like "DEF%" ,"DEF*", NAME like "GHI%" , "GHI*" , NAME like "JK%","JK*", 1=1,"Others") 
 | search "Qname"!=Others 
 | eventstats count as Grandtotal 
 | where Grandtotal >50 
 | eval _Date=strftime(now(), "%d/%m/%Y") 
 | stats count(eval(STATUS="FAIL")) as "FAIL" , count(eval(STATUS="TRYAGAIN")) as "TRYAGAIN" by _Date SYSTEM "Qname" 
 | table _Date SYSTEM "Qname" FAIL TRYAGAIN
 | addtotals fieldname="Grand Total" col=t row=t labelfield="Qname" label="Grand Total"
Cheers,
Jacob

If you feel this response answered your question, please do not forget to mark it as such. If it did not, but you do have the answer, feel free to answer your own post and accept that as the answer.
0 Karma

njohnson7
Path Finder

@jacobevans thankyou but it is more or less similar to what I already tried. The results appears in the below format.

Date             SYSTEM      Qname        FAIL         TRYAGAIN  Grand Total
08/08/2019       sys1         ABC*        300            25        325
08/08/2019       sys1         DEF*        210            55        265
08/08/2019       sys1         GHI*        470            25        495
08/08/2019       sys1         JK*         250            25        275
            Grand Total     1230           130       1360

Now when I try to trigger the alerts using the "FOR EACH RESULT" option and giving Date as the field value for suppressing the alerts.

I only get the first line and the last line from my table of results since the date value is same:

 Date             SYSTEM      Qname        FAIL         TRYAGAIN  Grand Total
 08/08/2019       sys1         ABC*        300            25        325
                           Grand Total     1230           130       1360

I would like to send my entire table contents in the alerts in the format I have shown in the example , however I would like to suppress it only until end of the day and start over again as soon as the next day starts.

0 Karma
Get Updates on the Splunk Community!

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...