Alerting

New Real Time Alerts not working

yrajah
Explorer

Hello,

We have a number of real time alerts that are working fine (that are being generated by certain Active Directory events via the Universal Forwarder installed on the DC), but when I try to create any new real time alerts they do not seem to work; I am not receiving the email, and the Alert counter on the Searches and Reports page remains on 0. When I run the search manually for the last 15 minutes, I get results that I would expect, so the search parameters seem to be ok.

I even cloned a working rule, and created an event. The original alert triggered, but the new cloned one did not 😞

Tags (1)

jkeellogic
Explorer

Hey dudes.
I am fighting the same problems, but I do have some clues. Mine I beleive are related to LDAP so I don't know your environment but configured to LDAP can be and issue.
My real time alerts changed everytime I changed added more complex strings to LDAP.

I have other ideas about working around this but it takes time.

jim

0 Karma

yrajah
Explorer

I did have further problems with this, and I now believe I found the cause.

In my case I think it was simply because I had too many real time searches running, and was hitting my my limit. I believe that you can change the limit in limits.conf as long as your hardware is up to the job. I just cleaned up some stuff, and changed some real time searches/alerts to a daily report and have not had any issues since.

http://docs.splunk.com/Documentation/Splunk/5.0.3/Admin/Limitsconf

http://docs.splunk.com/Documentation/Splunk/5.0.1/Search/Realtimeperformanceandlimitations

salem34
Path Finder

Thanks for pointing that out, idk if i ever thoght about limits here.

0 Karma

jtrucks
Splunk Employee
Splunk Employee

Are the newly created/cloned alerts owned by a different user? If so, does that user have a valid email address set?

--
Jesse Trucks
Minister of Magic
0 Karma

jtrucks
Splunk Employee
Splunk Employee

You should post an answer that a reboot fixed it and then accept the answer. 🙂

--
Jesse Trucks
Minister of Magic
0 Karma

brettcave
Builder

nope, owned by my user. i have a valid email address. the alerts started working after a splunk server restart.

yrajah
Explorer

Hello,
I still get issues with real time alerting every now and then. The last one I had (maybe not exactly the same as this) was resolved by a restart of the splunk services. I would be interested to know if this fixes your problem?

0 Karma

brettcave
Builder

+1 - me too. alerts were working, and then modified the search. now they're not. I even deleted the search, recreated it, and still not getting results.

0 Karma

sd100
Explorer

Hello, Same issue here, i'm interested in the answer.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...