Getting Data In

Monitor alerts (alarm if alerts do not work)

igschloessl
Explorer

Hello together,

i want to monitor existing alerts in splunk. For the case that an alarm doesn't work proper and doesn't find anything I want to get a notice or an alarm for that.

I do not know how to do this far.

Is there something in the internal index where splunk logs its alarms? Any suggestions?

Thanks in advance.

0 Karma

harsmarvania57
Ultra Champion

As a starting point have a look at index=_internal sourcetype=scheduler this will give you all scheduled search logs like, when it started, when its completed, eventcount, resultcount, timetaken etc.

igschloessl
Explorer

Thank you. I tried to do a summary with appendcols

index=_internal sourcetype=scheduler earliest=-7d@d latest=-0d@d
| eval period="-7d"
| stats count min(result_count) as min_result_count max(result_count) as max_result_count avg(result_count) as avg_result_count by savedsearch_name app period
| eval avg_result_count= round(avg_result_count,2)
| table savedsearch_name app period min_result_count max_result_count avg_result_count count

| appendcols [search index=_internal sourcetype=scheduler earliest=-14d@d latest=-7@d
| eval period= "-14d"
| stats count min(result_count) as min_result_count max(result_count) as max_result_count avg(result_count) as avg_result_count by savedsearch_name app period
| eval avg_result_count= round(avg_result_count,2)
| table savedsearch_name app period min_result_count max_result_count avg_result_count count
]
| appendcols [search index=_internal sourcetype=scheduler earliest=-21d@d latest=-14@d
| eval period= "-21d"
| stats count min(result_count) as min_result_count max(result_count) as max_result_count avg(result_count) as avg_result_count by savedsearch_name app period
| eval avg_result_count= round(avg_result_count,2)
| table savedsearch_name app period min_result_count max_result_count avg_result_count count
]
But this does not work. And I dont know how to alarm if the counts are highly different from the weeks before.

0 Karma
Get Updates on the Splunk Community!

Get ready to show some Splunk Certification swagger at .conf24!

Dive into the deep end of data by earning a Splunk Certification at .conf24. We're enticing you again this ...

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Now On-Demand Join us to learn more about how you can leverage Service Level Objectives (SLOs) and the new ...

Database Performance Sidebar Panel Now on APM Database Query Performance & Service ...

We’ve streamlined the troubleshooting experience for database-related service issues by adding a database ...