Splunk Search

How to search the names of triggered alerts, their trigger time, and the events that triggered them?

LuiesCui
Communicator

Hi guys,

I have a problem with the triggered alerts and I really need your help!
Now, I have some alerts working great in my Splunk. To count the number of the alerts triggered in a period of time, I made a report with the search string like:

index=_audit action="alert_fired" | timechart count by ss_name

From this search string, I get the time and name of the triggered alerts as below:
alt text

But I also want to know the original events which triggered these alerts. All I want is a table with 3 columns. Column 1 shows the trigger time of the alerts, Column 2 shows the names of the triggered alerts, and Column 3 shows the events which triggered the alerts. How can I do that?

Tags (4)
1 Solution

MuS
SplunkTrust
SplunkTrust

Hi LuiesCui,

I must admit, this makes a really good use case for join since the alert threshold is hold in the savedsearch.conf and is accessible over the REST endpoint /services/saved/searches.
You will first need to get the alerts_threshold from REST and join it with the triggered alerts by the title:

index=_audit action="alert_fired" 
| rename ss_name AS title 
| join title [ | rest /services/saved/searches | table title, alert_threshold ] 
| timechart values(alert_threshold) AS alert_threshold count by title

Hope this helps to get you started ...

cheers, MuS

View solution in original post

MuS
SplunkTrust
SplunkTrust

Hi LuiesCui,

I must admit, this makes a really good use case for join since the alert threshold is hold in the savedsearch.conf and is accessible over the REST endpoint /services/saved/searches.
You will first need to get the alerts_threshold from REST and join it with the triggered alerts by the title:

index=_audit action="alert_fired" 
| rename ss_name AS title 
| join title [ | rest /services/saved/searches | table title, alert_threshold ] 
| timechart values(alert_threshold) AS alert_threshold count by title

Hope this helps to get you started ...

cheers, MuS

iqbalintouch
Path Finder

Hi MuS,

I have same issue and looking for resolution. I followed the same query but it didn't work for me. Can you please help me out.
FYI: I am using enterprise splunk, version 6.3.2
I was able to get the list of all the enabled alert from here: | rest /servicesNS/-/-/saved/searches

But I need to get the details of how many times the alert was triggered in particular time duration, what was the alert and what time(when) ?

Thank you!

0 Karma

MuS
SplunkTrust
SplunkTrust

I will check that we I have access to an instance that has alerts enabled. Just one thing for now, rest calls are ignoring time but I'm sure there is some information available to see when an alert was triggered.

0 Karma

iqbalintouch
Path Finder

Hi MuS ,

Yes, I am quite sure there must be a place where we capture the details of when the alert crossed the threshold value and an email was sent out. Using that value we can pull the data when this alert was triggered, how many times (let's say how many times in last 15 days) and whom the email notification was sent.

I use below query for all the enabled alerts:
| rest /servicesNS/-/-/saved/searches splunk_server=local | search disabled=0 actions=* is_scheduled=1

so using the above query can you please help me out here?

0 Karma

MuS
SplunkTrust
SplunkTrust

Okay, can you please explain what is not working for you? You need to have the admin role assigned to use the search posted in the original answer.

0 Karma

iqbalintouch
Path Finder

Hi MuS,
the query itself not returning means i is not giving any result:

index=_audit action="alert_fired"
| rename ss_name AS title
| join title [ | rest /services/saved/searches | table title, alert_threshold ]
| timechart values(alert_threshold) AS alert_threshold count by title

though I can get the desired output by this:
| rest /servicesNS/-/-/saved/searches splunk_server=local | search disabled=0 actions=* is_scheduled=1

so my request is simple I need to pull a report OR data where how many times an alert was triggered during a given time period, when it was triggered and whom the email was sent.

I do have power user access on splunk but not an admin level.

0 Karma

MuS
SplunkTrust
SplunkTrust

Well your request is indeed very simple, though there is not much I or anyone can do to help. Since you have the power user role and not admin you cannot access the index _audit and therefore will not be able to get the search working.

Ask your friendly Splunk admin to create a report for you or to schedule a saved search which feeds into a summary index so you can run the reports on the summary index.

cheers, MuS

iqbalintouch
Path Finder

Hi MuS,

Got it, thank you 🙂

0 Karma

LuiesCui
Communicator

Hi, thank you for your reply! I kind of worked this out but got a new problem. Could you help me with this please? Thank you again. http://answers.splunk.com/answers/305369/why-the-results-from-triggered-alert-is-different.html

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...