Splunk Search

External lookups: lookup not found error

twinspop
Influencer

I'm following the instructions here and can't get it to even recognize the lookup. Did I miss something?

My transforms.conf:

[SUBJDECODE]
external_cmd = utfconv.py Subject
fields_list = Subject

My props.conf:

[source::/syslog/mail/*]
LOOKUP_table = SUBJDECODE Subject

Any search gives me the error: "The lookup table 'SUBJDECODE' does not exist. It is referenced by configuration 'source::/syslog/mail/*'."

I've even verified the lookup exists through the GUI -> Manager -> Lookups -> Lookup Defs

SUBJDECODE   external   No owner   system   Global | Permissions   Enabled ....

It appears to recognize the props file, but is not fully integrating the transforms stanza. It shows in the GUI manager but can't be used. Both conf files are in $splunk/etc/system/local, but I've also tried them in the $splunk/etc/apps/search/local dir with equivalent results.

Tags (1)
1 Solution

twinspop
Influencer

The stanza for the external lookup was not correct. The docs are ambiguous in a few places, and the absolutely terrible error message sent me on a wild goose chase, but I think I finally got there.

In transforms.conf you need to list the name of the field that will be handed to the lookup AS WELL AS the field name you want the script to output post-lookup. So:

[SUBJDECODE]
external_cmd = utfconv.py Subject decoded_subject
fields_list = Subject, decoded_subject

Even though decoded_subject doesn't exist, it needs to be there. I guess. Maybe. Anyway, it's working for me now. In my original stanza I was attempting to replace the original Subject field with the new value-- apparently a NOOP that blows up the logic and returns a completely unrelated error message.

To call the lookup, you need to leave off the output field (apparently):

source=*mail* | lookup SUBJDECODE Subject

Tada. It worked.

View solution in original post

twinspop
Influencer

The stanza for the external lookup was not correct. The docs are ambiguous in a few places, and the absolutely terrible error message sent me on a wild goose chase, but I think I finally got there.

In transforms.conf you need to list the name of the field that will be handed to the lookup AS WELL AS the field name you want the script to output post-lookup. So:

[SUBJDECODE]
external_cmd = utfconv.py Subject decoded_subject
fields_list = Subject, decoded_subject

Even though decoded_subject doesn't exist, it needs to be there. I guess. Maybe. Anyway, it's working for me now. In my original stanza I was attempting to replace the original Subject field with the new value-- apparently a NOOP that blows up the logic and returns a completely unrelated error message.

To call the lookup, you need to leave off the output field (apparently):

source=*mail* | lookup SUBJDECODE Subject

Tada. It worked.

twinspop
Influencer

OK, I copied the dnslookup stanza from etc/system/default/transforms.conf and put it into local/transforms.conf. I named it dnslookup2. That works. So external lookups do work, but my custom command isn't working. That leads me to believe the error is with my script. If so, the error message provided is terribly misleading.

As for the script, running on the command line works fine. Piping CSV data into STDIN with the required args results in CSV being spit back out.

0 Karma

hexx
Splunk Employee
Splunk Employee

A few other things you may want to check here :

1) Where is the "utconfv.py" script located? As transforms.conf.spec states :

external_cmd = <string>
* Command and arguments to invoke to perform lookups.
* This string is parsed like a shell command.
* The first argument is expected to be a python script located in $SPLUNK_HOME/etc/<app_name>/bin (or ../etc/searchscripts) <=========
* Presence of this field indicates that lookup is external command based.

2) Are there no permission/ownership issues with utconf.py?

3) Check in $SPLUNK_HOME/var/log/splunk/python.log for errors referencing your lookup script.

twinspop
Influencer

The script is in $SPLUNK/etc/searchscripts and is set to 755. The python.log file is empty.

0 Karma

sophy
Splunk Employee
Splunk Employee

it might be an issue with your permissions? you can run:

splunk cmd btool transforms list --user=<user-running-search> --app=search --debug

and if it doesn't list the SUBJDECODE stanza, then it's a permissions issue w/ that particular user...

twinspop
Influencer

Done... yes the lookup stanza is there.

0 Karma

twinspop
Influencer

Now with shiny, new, strong, faster, better 4.1.5. Problem persists. 😞

0 Karma

twinspop
Influencer

Despite not being in the docs, I've added the metadata stanza (export=system). The stanza was already in the search app metadata. However, it was not in the system metadata file. I've added there also. Still no go. Anyone? Buehler?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...