Alerting

5.0.3.1 -- Scheduled search creating lots of directories in dispatch despite setting expiration to 1s

the_wolverine
Champion

Conditions:
1) Scheduled search that runs every minute and writes to summary index.

2) Additionally, configured to alert "always" and email to someone@email.address.
3) Expiration set to custom time, 1 second.
4) savedsearches.conf config for this search clearly shows 1s expiration value:

alert.expires = 1s

Expectation -- only 1 (or a few) directories related to this search in dispatch. Its ok if the cleanup job can't run often enough to keep it exactly at 1 second, even a few minutes would be fine.

Observation -- over 1000 directories generated by this scheduled search.

0 Karma

the_wolverine
Champion

Suspicion - This scheduled search is affected because it is considered an alert and different ttl applies to the generated "alert" artifacts.

Of more specific suspicion, in etc/system/default/alert_actions.conf:

ttl         = 86400

(86400 seconds = 24 hours)

Or, there's a bug.

sowings
Splunk Employee
Splunk Employee

A job which triggers an alert action takes on the TTL of that action.

0 Karma
Get Updates on the Splunk Community!

More Ways To Control Your Costs With Archived Metrics | Register for Tech Talk

Tuesday, May 14, 2024  |  11AM PT / 2PM ET Register to Attend Join us for this Tech Talk and learn how to ...

.conf24 | Personalize your .conf experience with Learning Paths!

Personalize your .conf24 Experience Learning paths allow you to level up your skill sets and dive deeper ...

Threat Hunting Unlocked: How to Uplevel Your Threat Hunting With the PEAK Framework ...

WATCH NOWAs AI starts tackling low level alerts, it's more critical than ever to uplevel your threat hunting ...