Changes to search configuration (field extractions etc) don't take effect right away in my distributed search environment. Why so? Can it be changed?

[jrodman]

I'm adding and modifying settings to my Splunk search-time behavior -- improving extractions, creating lookups, and so on. This works. However, there seems to be a delay before these changes take effect. Sometimes the delay is fairly short -- a few seconds, while other times it can take over a minute.

Is this intended? Can I alter this behavior?

[mattness · Accepted solution]

You can do this fairly simply by making a change in limits.conf--you just set sync_bundle_replication to 1.

With this setting when you try to fire up a search and the indexers don't have the current configuration, Splunk will push it to them, and then run the search. The tradeoff is that the search won't start quite as fast—you'll hit Search and there will be a pause of a second or two while the config gets updated on the indexers before the search actually starts running. It's up to you to determine whether this lag is worth the satisfaction of seeing immediate application of your config changes.

One caveat: if you have a lot of searches running at once (you have a lot of users, or a lot of scheduled searches running in the background) this could cause some major inefficiencies. Usually bundles are replicated every minute--in this case you're replicating bundles with every search. So this solution scales poorly as the number of searches being run on your system increases because you'll be doing more bundle replication than searching.