Splunk Search

In a distributed search environment, where do my configurations go? The search head? The Indexers?

jrodman
Splunk Employee
Splunk Employee

If I have, say five, indexers, and a search head that points at them, where do my field extractions, tagging, lookups, and so on get stored? Do I have to manually distribute them to my indexing nodes? If the configuration is distributed automatically, when does it happen, and are there any exceptions? What about a conflict between settings on the different nodes?

1 Solution

gkanapathy
Splunk Employee
Splunk Employee

Search time configurations, including lookup tables, lookup scripts, and custom search commands, as well as field extractions, tags, event types, aliases, etc. go on the search head and the search head only. The Distributed Search mechanism will make sure the configuration items are sent to the indexers when a search is issued.

However, do note that lookup tables and lookup/search scripts must be in an app or system lookup or bin directories. Also any resources that scripts themselves may reference will only be copied to the indexers if they are files located inside of the bin or lookup other app folders, and that such references must be relative to the app or script base (and not absolute). (Other resources will only be available if you use some other method to get them the indexers and reference them accordingly in your scripts.) If you have scripts, you may rely on this mechanism to distribute the scripts, or you can look at the localop command and local option on the lookup search command.

View solution in original post

gkanapathy
Splunk Employee
Splunk Employee

Search time configurations, including lookup tables, lookup scripts, and custom search commands, as well as field extractions, tags, event types, aliases, etc. go on the search head and the search head only. The Distributed Search mechanism will make sure the configuration items are sent to the indexers when a search is issued.

However, do note that lookup tables and lookup/search scripts must be in an app or system lookup or bin directories. Also any resources that scripts themselves may reference will only be copied to the indexers if they are files located inside of the bin or lookup other app folders, and that such references must be relative to the app or script base (and not absolute). (Other resources will only be available if you use some other method to get them the indexers and reference them accordingly in your scripts.) If you have scripts, you may rely on this mechanism to distribute the scripts, or you can look at the localop command and local option on the lookup search command.

gfriedmann
Communicator

I ran into inconsistent behavior with 4.2.2 when the indexers had an old copy of a lookup table defined and referenced. The search head had the updated copy. It was a bit of a pain to troubleshoot. The difference ended up becoming apparent if the lookup was applied after the main search results instead of fueling the main search itself.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

I have tested this, and the one from the search head is used. The search head sends over a bundle containing every single app, system and user config, and I believe that for purposes of executing a search from the search head, the splunk-search process loads that entire configuration.

0 Karma

jrodman
Splunk Employee
Splunk Employee

Incidentally, a variety of things were not replicated to the search nodes correctly in versions of 4.0.x, for example lookup scripts didn't make it across until 4.0.7 or so.

Still wish i knew what happens in case of conflict. Search head says the transform uses REGEX1, the indexer says it uses REGEX2....

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...