Why are my alerts not being triggered?

[APNelson]

I have an alert that I created. When I click "Open in Search and trigger the event, it shows up in the search window, but the event does not trigger the alert (send e-mail, execute the script, or show up in Triggered Alerts).

The alert is in the savedsearches.conf file in system/local and shows up with the Owner as "nobody", the App as "system" and Sharing as "Global".

What can I do to fix this problem? I have several alerts and it appears that none of them are working properly at this time.

UPDATE: It appears the number of searches may be partially responsible. When I have just one real-time alert in the savedsearches.conf file it appears to work correctly, but when I get up to 6, it stops working. The requirements being fulfilled by Splunk require as many as 14 real time searches to trigger alerts when necessary, so I definitely need some kind of solution to this problem.

[ben363]

Try searching Splunk for index=_internal source="/opt/splunk/var/log/splunk/scheduler.log" status!=success OR NOT INFO.

You can also search directly in scheduler.log.

[mbrownec]

Thank you! index=_internal source=*scheduler.log status!=success OR NOT INFO savedsearch_name="[name of saved search here]" | dedup reason | table reason In my case, reason == "maxRtsearches limit reached"

[martin_mueller · Accepted solution]

Manually running the search isn't supposed to trigger the alert action, you need to wait for a scheduled run... not sure if that's what's missing here though, do elaborate if not.

[martin_mueller]

Great. For future growth, there should be logs in _internal stating that this limit has been reached... I think. If you found those you could consider setting up a (non-realtime) alert for them to add more cores / add more search heads / increase the limit.

[APNelson]

That did the trick. I changed base_max_searches and max_rt_search_multiplier and now they're all showing up in Jobs and my test one is responding properly.

[martin_mueller]

There are two limits to be concerned about here, one is the number of real-time searches your user can run (see Settings -> Authentication -> Roles), the other is the number of real-time searches your Search Head can run (see limits.conf, depends on the number of cores your SH has).

Not sure how the limit for the nobody user is calculated though.

[APNelson]

It appears the number of searches may be partially responsible. When I have just one real-time alert in the savedsearches.conf file it appears to work correctly, but when I get up to 6, it stops working. The requirements being fulfilled by Splunk require as many as 14 real time searches to trigger alerts when necessary, so I definitely need some kind of solution to this problem.

[martin_mueller]

Yeah, running in system/local isn't such a great idea... however, if they're still not running if moved to an app context then there's gotta be an error message for that that's different from the system/local one.

[APNelson]

I see 8 searches in the Jobs view, but not one for each of my alerts. Four have their status marked as "Done", while 4 others (which are some of my alerts, but not the one I'm using for testing) have the status "Running (100%).

I see an entry in the scheduler log indicating that it cannot execute scheduled searches that live at the system level for some reason, but I'm getting the same behavior regardless of whether my savedsearches.conf file is in apps/search/local or system/local (with a restart after moving the file so the searches are moved into an app context).

[martin_mueller]

I see. There's a huge list of things that could be going wrong. Is the search running in the job inspector and showing results? What's the trigger condition and similar configs for the alert? Anything suspicious / erroring in _internal?

[APNelson]

The search for the alert is real-time, not scheduled. I'm just using the fact that the entry is showing up in search using the same criteria to prove to myself that the event was received.