Splunk Search

Where does a lookup table need to be in a distributed search environment?

bruceclarke
Contributor

All,

I'm having an issue where one of my indexers is complaining about a lookup table that I have setup on my search head. I get the error

[IndexerMachine] Streamed search execute failed because: Error in 'lookup' command The lookup table 'groupIdToName' does not exist.

From what I can tell from other Splunk Answers, the lookup table should be replicated to the search peers as part of the bundle replication (http://answers.splunk.com/answers/28541/lookup-table-does-not-exist.html). However, when I look at $SPLUNK_HOME/var/run/searchpeers/{most recent bundle} on the search peer, I don't see the lookup that should have been copied. In fact, I don't even see the system folder in that bundle.

So, I have two questions:

  1. How should a lookup be set up for a distributed search environment (i.e. should the lookup live on the search head, indexers, or both)?
  2. Assuming my set up is correct and the lookup should only live on the search head, how do I make sure that the lookup gets copied as part of the bundle replication?

Thanks!

0 Karma
1 Solution

bruceclarke
Contributor

This was related to an app hitting an error when trying to perform bundle replication. The app was creating a file name that was way too long. Adding the app to the blacklist for bundle replication fixed the issue.

View solution in original post

bruceclarke
Contributor

This was related to an app hitting an error when trying to perform bundle replication. The app was creating a file name that was way too long. Adding the app to the blacklist for bundle replication fixed the issue.

nlembrechts
Explorer

Where did you find what app was causing the problem?

0 Karma

bruceclarke
Contributor

I don't remember, but I likely found it in the splunkd.log file. I'm sure there was an error there about bundle replication.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Does the lookup have appropriate permissions in Search Head?

0 Karma

bruceclarke
Contributor

Yes. It is a globally permissioned lookup stored in the $SPLUNK_HOME$/etc/system/lookup folder. Everyone has read access to the lookup.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Try moving it feom etc//system to etc//apps into any application.

0 Karma

bruceclarke
Contributor

@somesoni2 - I moved the lookup to an application. I see it in the $SPLUNK_HOME$\var\run\searchpeers\$BUNDLE_FOLDER$\apps\myApp\lookups folder, but Splunk still shows the same error saying it wasn't found on the indexer.

Right now, my workaround is to use local=true for the lookup, but that's obviously not ideal. I'm not sure how to debug further.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

I was referring to move it to $SPLUNK_HOME\etc\apps\lookups folder. This way it will be part of replication bundle. You can use search app for testing, if you don't want to create a new one.

0 Karma

bruceclarke
Contributor

Right, I moved it there on the search head. My point is that it appears to be replicated (it shows up in the replication folder on the indexer). But I'm still getting the error.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...