Why am I still receiving alerts from Splunk even after disabling them?

[AzmathShaik]

Hi,

i am using Splunk 6.4.3. i have configured real-time alerts to verify. once it is done, i have disabled and deleted savedsearches.conf from the search head but still am getting alerts from Splunk. can anyone help on how to get rid of alerts?

[dilsheer]

is there any way to disable the real time alert option permanently??

[woodcock]

How are you determining that alerts are still firing? Let's say that you have the alert send you an email. Email is VERY laggy and it is possible that there are thousands of oustanding emails that, had the volume been slower/smaller, should have already been delivered, but because you were flooding emails, systems inbetween in the MTA chain are throttling delivery. You may see email for weeks to come that were sent BEFORE you turned off the alert. You can go to your Search Head and do this:

find $SPLUNK_HOME -name "savedsearches.conf"

Then look inside every file that you find and make SURE that anything like your search has disabled=1.

[somesoni2]

The real-time saved searches never stop running, so did you kill those searches after you deleted the savedsearches.conf entry? From the search head servers, you can do (for linux) psef splunk and see any process running for those real-time searches. If found, kill them.

[AzmathShaik]

i did ps -ef | grep splunk.

i could not find any process running with my savedsearches name.

[somesoni2]

It'll not be with your saved search name, there may be process which contains rt_ keyword and may have very old start time.

[AzmathShaik]

yeah i did check that but i could not find any process running.

[richgalloway]

Are you saying you modified the savedsearches.conf file from the command line? If so, did you also refresh or start Splunk? Most command line changes don't take effect until refresh/restart.

[AzmathShaik]

yeah i did restart splunk. but still am getting lot of alerts

[richgalloway]

Did you modify the right file? Do the searches still appear in the GUI?

[AzmathShaik]

No There are no alerts or savaedsearches i can see in GUI