I recently upgrade a test system to Splunk 4.1, and my lookups are all giving me the following error:
The lookup table 'MyFieldLookup' does not exist. It is referenced by configuration 'my_source_type'.
I get this error appears is a flashy read box on all searches in any app except for the app where MyFieldLookup
is defined. (MyFieldLookup
is exported globally.) This occurs even with searches that are not using any fields related to the defined lookup, even if when searching on sourcetypes that are not associated with the lookup.
Are there a known issues with Lookups beyond what's listed Known issues page? (I tried changing the EOLs from CR/LF to LF, but that had no effect, based on the listed SPL-29434)
I have a single lookup defined which was working fine on Splunk 4.0.10.
I suspect that there could be some kind of permissions issue here, since the lookup works when I'm within the context of that app, but as soon as I search from a different app (which is the normal case based on my setup) then I get the error shown above. I've double checked all my permissions, and even added an explicit entry for [lookups/my_lookup_file.csv]
, but nothing seems to make a difference.
If I run: btool transforms list --app=<app-1> MyFieldLookup
, (where app-1
is where the config is defined) I see the configuration entry, but if I run the command with --app=<app-2>
(the app that most frequently uses the lookup), then there is no output, so it would appear that splunk cannot see that entry (or any other entry in the transforms.conf file)
My default.meta
includes:
[transforms] access = read : [ * ], write : [ admin ] owner = nobody export = system
So the transforms
entries should be visible from other apps.
Update: I found out that
btool
seems to behave differently if you specify a user. For example, if I runbtool transforms list --app=<app-1> --user=someuser MyFieldLookup
, now I get the results I was expecting. So my problem wasn't with mytransforms.conf
entry or it's permissions.
For the simplicity, let me consolidate the answers and comments here:
NOTICE: Splunk 4.1.1 and later add a migration note about these permissions changes when during the upgrade. So hopefully fewer users will run into this issue. The migration notes point your to the following docs page.
In Splunk 4.1. it is now possible to set permissions on individual lookup tables (aka .csv
files) in the lookups
folder of an app. Because of this new control in Splunk 4.1, you have the option to grant or restrict access to these files. There are actually 3 different sets of permissions to control access to the lookups now.
You basically have two options: (1) Disable your lookups to make the error message go away, and use the lookup
search operator instead. Or (2), make your transforms and lookups global (or otherwise tweak your permissions till you get it all working again...)
Assuming your have sensitive info in your lookup file, you can avoid this error message by disabling (comment-out) or changing the permissions of your LOOKUP-<class>
entry in your props.conf
file. You can do this from the UI via Manager » Lookups » Automatic lookups, then find your stanza name (which will be based on your source, sourcetype, or host and the name of your LOOKUP
line) and change the permission using the Permissions link.
If you prefer editing config files directly, you can also set your permissions in metadata/local.meta
with an entry like this:
[props/my_sourcetype/LOOKUP-MyFieldLookup] access = read : [ * ], write : [ admin ] export = none
Keep in mind, that if you choose to completely remove (or comment out) your LOOKUP
entries, rather than change the permissions, then you will need to use the lookup
search command now, since the lookup fields will no longer be "automatic".
You can go back to the globally-available behavior of Splunk 4.0 by setting your lookups permissions using the UI (Manger >> Lookups >> Lookup Table files, then click on the Permissions link).
This can also be done by updating the metadata/local.meta
file manually. To allow all access to all csv files from any application, add an entry like so:
[lookups] export = system
Or, for a single lookup file:
[lookups/my_lookup_file.csv] export = system
NOTE: You can also specify permission to the Lookup definition (based on the transforms.conf
name), in same way as in Splunk 4.0. So if you want your lookup to be global, you would use something like:
[tranforms/MyFieldLookup] export=system
Again, this can also be done from the UI. In Splunk 4.1, navigate to Manger >> Lookups >> Lookup definitions, then click the Permissions link.
I just wanted to add one more piece of detail to Lowell's option 2. If you simply do:
[lookups/my_lookup_file.csv]
export = system
within an app's local.meta file for automatic lookup, this will let you configure props.conf lookup within the app without issues. This assumes that you will only use this app and not use any other app.
As soon as you move to another app (e.g., start using the search app and then use the Windows app), you will get the error below within the other app's dashboards.
The lookup table 'MyFieldLookup' does not exist. It is referenced by configuration 'my_source_type'.
If you want to avoid that, you have to enable your corresponding transforms to be exported to global. In the context of this example, the original app's local.meta file should be:
[lookups/my_lookup_file.csv]
export = system
[tranforms/MyFieldLookup]
export = system
After doing this, you can use any other app within Splunk and not receive the error.
I don't think the export is working properly globally. To use your example, if I do
[lookups/my_lookup_file.csv]
export = system
in my local.meta file for the app, the error message disappears as noted. However, as soon I move to another app that doesn't even use the my_lookup_file.csv and its associated props.conf and transforms.conf files, the error:
The lookup table 'MyFieldLookup' does not exist. It is referenced by configuration 'my_source_type'.
returns when loading a dashboard from within another app. The workaround is to comment out the lookup entry in props.conf to disable automatic lookups for this lookup. You could still call the lookup explicitly within a search using the lookup command. I have not made my transforms global within local.meta, which may solve this, because I do not want to make these entries global.
I updated my consolidated answer to include your input. Thanks.
Yes, thats the same issue I was seeing (I updated the question to make the point about any search no matter how unrelated results in this error message.) As you point out, you certainly do have two choices. Either make it all global, or hide the transformer section and remove the lookups altogether and do it via the search lookup
command.
For the simplicity, let me consolidate the answers and comments here:
NOTICE: Splunk 4.1.1 and later add a migration note about these permissions changes when during the upgrade. So hopefully fewer users will run into this issue. The migration notes point your to the following docs page.
In Splunk 4.1. it is now possible to set permissions on individual lookup tables (aka .csv
files) in the lookups
folder of an app. Because of this new control in Splunk 4.1, you have the option to grant or restrict access to these files. There are actually 3 different sets of permissions to control access to the lookups now.
You basically have two options: (1) Disable your lookups to make the error message go away, and use the lookup
search operator instead. Or (2), make your transforms and lookups global (or otherwise tweak your permissions till you get it all working again...)
Assuming your have sensitive info in your lookup file, you can avoid this error message by disabling (comment-out) or changing the permissions of your LOOKUP-<class>
entry in your props.conf
file. You can do this from the UI via Manager » Lookups » Automatic lookups, then find your stanza name (which will be based on your source, sourcetype, or host and the name of your LOOKUP
line) and change the permission using the Permissions link.
If you prefer editing config files directly, you can also set your permissions in metadata/local.meta
with an entry like this:
[props/my_sourcetype/LOOKUP-MyFieldLookup] access = read : [ * ], write : [ admin ] export = none
Keep in mind, that if you choose to completely remove (or comment out) your LOOKUP
entries, rather than change the permissions, then you will need to use the lookup
search command now, since the lookup fields will no longer be "automatic".
You can go back to the globally-available behavior of Splunk 4.0 by setting your lookups permissions using the UI (Manger >> Lookups >> Lookup Table files, then click on the Permissions link).
This can also be done by updating the metadata/local.meta
file manually. To allow all access to all csv files from any application, add an entry like so:
[lookups] export = system
Or, for a single lookup file:
[lookups/my_lookup_file.csv] export = system
NOTE: You can also specify permission to the Lookup definition (based on the transforms.conf
name), in same way as in Splunk 4.0. So if you want your lookup to be global, you would use something like:
[tranforms/MyFieldLookup] export=system
Again, this can also be done from the UI. In Splunk 4.1, navigate to Manger >> Lookups >> Lookup definitions, then click the Permissions link.
I met this problem too.
The lookup works in the flashtimeline view of my app but does NOT work in any other of my savedsearch/dashboard/view of that app. The error message is the same: The lookup table xxxxx does not exist.
By referring the question and answer, I've solved my problem by using "export = system" of [lookups/...] and [transforms/...]. Their original values are "export = none".
I am still confusing why it should export the objects (lookup file and transform) to "system" for a lookup which is only supposed to be used in its app?
Since I posted my last comment. I too am running to an issue where I can't access an object shared at the application level. The only way to access it seems to be to set the access-level to global... very weird. (I'm seeing the same behavior in Splunk 4.0, BTW)
You are correct, you shouldn't have to export your objects to use them within that app. In my original scenario, my lookups worked within the app they were defined in, but not from another app. You may want to check your settings again, it's really easy to overlook some of the permissions settings.
Lookups and searchscripts are now permissioned in the metadata/*.meta
file(s). It should work with a stanza like:
[lookups]
export=system
or
[lookups/MyFieldLookup]
export=system
You should use the transforms.conf
-defined name of the lookup to qualify it, not the file name.
Incidentally, I would use only local.meta for my own apps, not default.meta, or you'll wind up having to manage two files if you edit permissions through the GUI.
With 6.1.1, for a dynamic lookup you have to export both [transforms]
and [searchscripts]
to system in order to make your dynamic lookup usable across apps
I think you are mixing two things here. I think when you said "[lookups/MyFieldLookup]
", you mean "[transforms/MyFieldLookup]
". The [lookups/...]
entries seem to be for specific files in the lookups
folder, not the name of the lookup (which is in the transforms file). I think my actual problem was I was some how missing the export=system
in my [lookups]
entry.