How to remove custom correlation searches in Splunk Enterprise Security?

[kausar]

I've been trying to remove some custom correlation searches, but they are still generating notables. So far I've tried:

• disabling them in savedsearches.conf, but didn't work
• removing them from the savedsearches.conf and correlationsearches.conf files, still didn't help

What am I missing? I see the names of these correlation rules in ./SA-ThreatIntelligence/lookups/correlationsearchmeta.csv and ./SA-ThreatIntelligence/local/eventtypes.conf files, any idea what they are for and should I delete the entries from here as well? Thanks

[hazekamp]

Disabling the saved search configurations for a correlation search will successfully prevent notable events from being generated (since this search will not be dispatched by the scheduler).

Removing the search from savedsearches.conf and correlationsearches.conf will also successfully prevent notable events from being generated (since the search no longer exists).

Perhaps this was done via the configuration file system without refreshing splunkd (this could attribute to the search still being recognized and scheduled by splunkd). Perhaps if the search was a RT search, the currently running search may need to be finalized (this could attribute to the continued generation of notable events).

With respect to correlationsearchmeta.csv, this is a legacy cache that in no way affects the generation of notable events.
With respect to eventtypes.conf, references to the correlation search represent notable event suppressions (filters) of notable events pertaining to the correlation search and in no way affects the generation of notable events.

Hope this helps.

David

[kausar]

Hi David,

Thanks your reply. This is not something I tried today or yesterday, it has been at least a week since I suppressed all the notables, cleaned/removed the rules from the savedsearches.conf and correlationsearches.conf files and restarted splunk many time after that.

It is very weird that these notables are still being generated (I search for past 24 hours everyday).
I noticed that this is the issue with custom correlation rules of which some were RT but mostly scheduled.

[hazekamp]

kauser,

Thanks for the clarifications. Notable events are simply the results of saved searches which persist into the "notable" index. If notable events are still being created then it's highly likely that the search is still running or being executed by the scheduler. I would consult the search audit logs for the saved search in question to determine whether it is in fact still running or being executed. I would also run:

| rest splunk_server=local count=0 /servicesNS/-/-/saved/searches | search title=<search name>

Or consult the saved search manager to ensure that the search is in fact "gone". You are also encouraged to file a support ticket so that we can investigate this more in-depth if needbe.

Thanks,
David

[kausar]

I don't see deleted saved searches when run the query,

| rest /servicesNS/-/-/saved/searches | search title="*correlation rule*" | table title, search 

OR used btool,

./splunk cmd btool --debug savedsearches list | grep -i "correlation rule"

But I do see them in index=_audit.

Filed a ticket as suggested. Thanks!

[hgrow]

Maybe silly but have you restarted your splunk after you cleaned the savedsearch.conf or did a debug-refresh?

Like @hazekamp mentioned "Perhaps this was done via the configuration file system without refreshing splunkd..."