Alerting

5.0.3.1 -- Scheduled search creating lots of directories in dispatch despite setting expiration to 1s

the_wolverine
Champion

Conditions:
1) Scheduled search that runs every minute and writes to summary index.

2) Additionally, configured to alert "always" and email to someone@email.address.
3) Expiration set to custom time, 1 second.
4) savedsearches.conf config for this search clearly shows 1s expiration value:

alert.expires = 1s

Expectation -- only 1 (or a few) directories related to this search in dispatch. Its ok if the cleanup job can't run often enough to keep it exactly at 1 second, even a few minutes would be fine.

Observation -- over 1000 directories generated by this scheduled search.

0 Karma

the_wolverine
Champion

Suspicion - This scheduled search is affected because it is considered an alert and different ttl applies to the generated "alert" artifacts.

Of more specific suspicion, in etc/system/default/alert_actions.conf:

ttl         = 86400

(86400 seconds = 24 hours)

Or, there's a bug.

sowings
Splunk Employee
Splunk Employee

A job which triggers an alert action takes on the TTL of that action.

0 Karma
Get Updates on the Splunk Community!

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Wednesday, May 29, 2024  |  11AM PST / 2PM ESTRegister now and join us to learn more about how you can ...

Get Your Exclusive Splunk Certified Cybersecurity Defense Engineer Certification at ...

We’re excited to announce a new Splunk certification exam being released at .conf24! If you’re headed to Vegas ...