Reporting

How do you manually add entries to savedsearches.conf for alerts?

reallyliri
Explorer

I want to create a lot of saved searches for alerts. Because I need to create about 20 different ones, I prefer to do it programatically.

I wrote a short program to generate the .conf file and replaced it with the existing one. However, after that, no alerts were triggered at all.

I checked again and again, and the the entries look like the splunk-generated ones.

From that, I'm assuming that the problem is that the file shouldn't be edited manually, or that there is some additional editing that needs to be done?

Any help is much appreciated.

Example for a splunk-generated entry:

[Errors alert]
action.email = 1
action.email.include.trigger_time = 1
action.email.inline = 1
action.email.mailserver = localhost
action.email.sendresults = 1
action.email.to = mymail@gmail.com
action.email.useNSSubject = 1
action.slack_webhook_alert = 1
action.slack_webhook_alert.param.message = Your *$name$* alert matched at least $job.resultCount$ events. Link: $results_link$. First result:    ```$result.exc_info$```
action.slack_webhook_alert.param.slack_webhook_name = Slack-Alerts
alert.suppress = 1
alert.suppress.period = 60s
alert.track = 1
counttype = number of events
cron_schedule = * * * * *
description = logs with level > INFO
dispatch.earliest_time = rt-1m
dispatch.latest_time = rt-0m
enableSched = 1
quantity = 0
relation = greater than
request.ui_dispatch_app = search
request.ui_dispatch_view = search
search = source="*-server" host=dev_*| spath levelno | search levelno>20

Example for a progrmatically-generated entry:

[Errors alert]
action.email = 1
action.email.include.trigger_time = 1
action.email.inline = 1
action.email.mailserver = localhost
action.email.priority = 2
action.email.sendresults = 1
action.email.to = mymail@gmail.com
action.email.useNSSubject = 1
action.slack_webhook_alert = 1
action.slack_webhook_alert.param.message = Your *$name$* alert matched at least $job.resultCount$ events. Link: $results_link$. First result:    ```$result._raw$```
action.slack_webhook_alert.param.slack_webhook_name = Slack-Alerts
alert.suppress = 1
alert.suppress.period = 60s
alert.track = 1
counttype = number of events
cron_schedule = * * * * *
description = logs with level > INFO
dispatch.earliest_time = rt-1m
dispatch.latest_time = rt-0m
enableSched = 1
quantity = 0
relation = greater than
request.ui_dispatch_app = search
request.ui_dispatch_view = search
search = source="*-server" host=dev_*| spath levelno | search levelno>20
0 Karma

ddrillic
Ultra Champion

-- From that, I'm assuming that the problem is that the file shouldn't be edited manually, or that there is some additional editing that needs to be done?

Any config file can be edited manually. Probably _internal would have some information ...

0 Karma

reallyliri
Explorer

Best I could do was add the saved-search with the cli, then manually edit the savedsearched.config file, but surely theres a simpler way. The documentation is horrible.

/opt/splunk/bin/splunk add saved-search -name 'Errors' -search 'source="*-server" host=dev_*| spath levelno | search levelno>20'
0 Karma
Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...