When I arrived in the office today I discovered our Splunk 6 logs were filling with errors like these:
10-10-2013 08:50:13.075 -0400 ERROR LookupOperator - Error 'Could not find all of the specified lookup fields in the lookup table.' for conf '(?::){0}PerfmonMk*:*' and lookup table 'IPLookup'.
10-10-2013 08:50:13.075 -0400 ERROR LookupOperator - Error 'Could not find all of the specified lookup fields in the lookup table.' for conf '(?::){0}WMI:*' and lookup table 'IPLookup'.
10-10-2013 08:50:13.076 -0400 ERROR LookupOperator - Error 'Could not find all of the specified lookup fields in the lookup table.' for conf '(?i)source::....zip(.\d+)?' and lookup table 'IPLookup'.
10-10-2013 08:50:13.076 -0400 ERROR LookupOperator - Error 'Could not find all of the specified lookup fields in the lookup table.' for conf 'ASR' and lookup table 'IPLookup'.
10-10-2013 08:50:13.076 -0400 ERROR LookupOperator - Error 'Could not find all of the specified lookup fields in the lookup table.' for conf 'ActiveDirectory' and lookup table 'IPLookup'.
These appear to be for all apps and continued for hours until I restarted Splunk. They resumed an hour later. I have no idea why. The errors vary, but always complain about lookup table 'IPLookup'. Where is that lookup file? It seems like something was corrupted, but I have no idea where to look? Any help? None of my apps are working with Splunk in this state.
Can you provide more information as to what apps are installed in your system.
How did you upgrade or install Splunk 6?
In your install, some app should be using a lookup IPLookup. The configuration of this
lookup should be in transforms.conf (it will tell you the name of the csv file or
external python script to be used). You will have to search for transforms.conf under
your install to locate which app has configured that lookup.
There seems to be a field-name mismatch between the csv file of the lookup and the way
it is invoked. The invocation of the lookup happens in props.conf (automatic lookups)
or directly by using search command lookup. The invocation is defined there. The
field names of the invocation should match the field names of the csv, otherwise
this error is thrown.
You can also go to your splunk web and look for Settings->Lookups->Lookup Definitions
so is your issue resolved now? Did you check to see whether the header line in
the csv was compatible with your props.conf (I am guessing that was the issue).
I am surprised that this error would not show up in splunk 5 system, if the
props.conf stanza was getting used.
Can you provide more information as to what apps are installed in your system.
How did you upgrade or install Splunk 6?
In your install, some app should be using a lookup IPLookup. The configuration of this
lookup should be in transforms.conf (it will tell you the name of the csv file or
external python script to be used). You will have to search for transforms.conf under
your install to locate which app has configured that lookup.
There seems to be a field-name mismatch between the csv file of the lookup and the way
it is invoked. The invocation of the lookup happens in props.conf (automatic lookups)
or directly by using search command lookup. The invocation is defined there. The
field names of the invocation should match the field names of the csv, otherwise
this error is thrown.
You can also go to your splunk web and look for Settings->Lookups->Lookup Definitions
Splunk 6 was installed fresh. I then copied our app from our Splunk 5 system and installed the following apps: Splunk Add-on for *Nix, Asset Discovery,
Splunk DB Connect, Splunk + OData, Sideview Utils,
S.o.S, Splunk App for Unix, Deployment Monitor, Tenable Security Center.
I swear I grepped for 'IPLookup' before I posted this question and didn't find it. This time I did find it.
The lookup was an old experiment that didn't go anywhere. I've commented it out of props.conf.
We do not see this error on our Splunk 5 system.