How to delete searches in a Search Head Cluster environment?

[matts1234]

I run a distributed environment within which the Search tier is comprised of a Search Head Cluster. All my Splunk apps are deployed to this Search Head Cluster via a separate VM designated as the Deployer.

In this deployment scheme, the Deployer takes all configurations within an apps ./local/ directory and cascades them into its ./default/ directory (resulting in an initially empty ./local/ directory on the individual Search Heads themselves). Because of this, all of my searches are being considered default app searches on the Search Heads and it removes the ability to delete them from the Search Manager.

Is there any way around this?
Is the only way for me to delete Searches to manually prune them from the configuration files before re-uploading it to the Deployer for distribution?

[cramasta]

Is the only way for me to delete Searches to manually prune them from the configuration files before re-uploading it to the Deployer for distribution? Yes unfortunately this is the only supported way (for now?)

Manually placing conf files on the search head cluster members local apps directory like other have suggested is not a supported way. Im not saying that people haven't done it and it seems to work, but no one from Splunk will tell you that its ok to do it that way

[RMartinezDTV]

I went through this myself and moving to a SHC is more of a mind-twist than anything.

Since I assume no one other than admins can access (let alone know how to use) your deployer, you'll quickly end up with 3 cases for searches/alerts/etc

1. Search pushed 100% from deployer (often migrated from pre-cluster)
2. Search created 100% locally via Splunk UI on cluster member
3. "Limbo" searches which were pushed from deployer but have local changes saved

It's pretty annoying, but my suggestion is to move all saved searches into the "local" config (not pushed from the deployer) and remove any references to the searches in the deployer's file structure. Do this by copy-pasting in the Splunk UI or via the REST API. DO NOT do this by putting .conf files in "local" directories. That will put you back into "old" Splunk where the cluster is no longer an issue.

Either way, you will need to use the REST API to delete searches that did not originate from the Splunk UI.

EDIT: Corrected some language to clarify what is meant by "local"

[teunlaan]

Simple anwser: Yes you should remove them from the config before re-uploading.

It would be nice if there was a way to force a sync from a SHC member to the rest of the members. If you could, you could move your ./local to a SHC member and push it to the others.

If you put you local's now on the SHC, your SHC will work with them BUT they won't get synced unless you edit it