Lookup File Editor in Search Head Cluster - "The requested lookup file does not exist"

[mnm1987]

Hello Splunkers,
I Tried installing the latest version of the lookup_Editor app on our Search Head Cluster.
Accessing the lookup files in the editor gives me the following message "The requested lookup file does not exist".

But the same version works on my Standalone Dev Splunk Instance.

Are there any known issues w.r.t to the lookup_Editor app working on a Search Head Cluster ?

Regards,
Mukund M

[LukeMurphey]

Try restarting the cluster after you confirm that the app appears on the Search Heads.

Details
I met with a customer (via @kcepull_splunk) who was experiencing the exact same problem today and we found that the app started working once the Search Heads were restarted.

This was on a Windows environment which may be more restart prone that Linux (which is perhaps why I couldn't get a repro on the Linux environment I created).

More details are available here: https://lukemurphey.net/issues/2098.

I'm looking at making this sort of troubleshooting easier in the future by providing a system that lets you know that controller or REST endpoints are not available or having the app request a restart.

[mikelanghorst]

Maybe I'm still new to SHC's, but I was expecting that since adding a new app typically required restarts, that it was doing this in the background when applying the shcluster-bundle.

Performing a rolling-restart did fix the problem for me, I'm on RHEL 7.

Thanks for your assistance Luke.

[mnm1987]

Thanks a lot @LukeMurphey, the rolling restart helped.
Really Appreciate your help. 🙂

[LukeMurphey]

You not alone, this threw me off too (and several others)!

I ran through a live debugging session with several people and none of us thought to try this until the administrator decided just to give it a shot. Sure enough, it worked.

[kcepull_splunk]

I'm getting the same errors. SHC 6.5.1, LFE 2.71. Fresh install of the app into an existing SHC.

[LukeMurphey]

@kcepull: could you try version 2.7.0?

I have deployed this on an SHC environment and cannot get a repro on this. I'm wondering if this is limited to 2.7.1.

[kcepull_splunk]

We tried 2.7.0, and got the same results (error).

[LukeMurphey]

Do you see any errors when you run the following search? Make sure to run if far enough back that it includes the time that the Splunk was started.

index=_internal lookup_edit sourcetype=splunk_web_service ERROR

[kcepull_splunk]

The only errors I see from that search are ones like these:

2017-10-19 16:53:46,563 INFO [59e910dz8c7f8ca46e80] error: 138 - Masking the original 404 message: 'The path '/en-US/custom/lookup_editor/lookup_edit/get_lookup_backups_list' was not found.' with 'Page not found!' for security reasons

I don't see any errors from startup. (NOTE: we deployed the app by dropping it on the Deployer, then pushing it to the SHC. I'm not sure if the SH gets restarted automatically by this process or not.)

[mwdbhyat]

Have you checked the permissions on the lookup files in question?

[mikelanghorst]

Running Splunk as root, and logged in as Admin. No lookup files can be viewed/opened in the app.

[LukeMurphey]

I tried to reproduce this with 2.7.1 but I was able to create and save a lookup.

BTW: I wouldn't recommend trying the one from GIT yet. That version is still in development and isn't release quality yet.

[mikelanghorst]

Also of note, in your troubleshooting suggestion:
index=_internal sourcetype="lookup_editor_controller"

Returns no results.

[LukeMurphey]

Thats a good find. It sounds like the controller couldn't start.

Could you run a search for the following and let me know if it shows anything of value (especially if any of the messages indicate an error)?

index=_internal lookup_edit sourcetype=splunk_web_service

[mikelanghorst]

Ah, that gives something:

2017-10-16 11:22:00,555 INFO [59e4f8c88b7f1ae85b9d90] error:133 - Masking the original 404 message: 'The path '/en-US/custom/lookup_editor/lookup_edit/get_lookup_contents' was not found.' with 'Page not found!' for security reasons

[LukeMurphey]

Do you also see an error that indicates why the controller couldn't load? It should somewhere around the time that Splunk started.

It may appear with this search:

index=_internal lookup_edit sourcetype=splunk_web_service ERROR

[mikelanghorst]

just more page not found messages:

2017-10-12 11:59:01,664 INFO    [59dfbb75a77fadc4208750] error:133 - Masking the original 404 message: 'The path '/en-US/custom/lookup_editor/lookup_edit/save' was not found.' with 'Page not found!' for security reasons
2017-10-12 10:34:02,935 INFO    [59dfa78aec7f695c31cbd0] error:133 - Masking the original 404 message: 'The path '/en-US/custom/lookup_editor/lookup_edit/get_lookup_backups_list' was not found.' with 'Page not found!' for security reasons
2017-10-12 10:34:02,717 INFO    [59dfa78ab57f695c31ce90] error:133 - Masking the original 404 message: 'The path '/en-US/custom/lookup_editor/lookup_edit/get_lookup_contents' was not found.' with 'Page not found!' for security reasons
2017-10-12 10:28:37,443 INFO    [59dfa6456e7f695c57aa10] error:133 - Masking the original 404 message: 'The path '/en-US/custom/lookup_editor/lookup_edit/save' was not found.' with 'Page not found!' for security reasons
2017-10-12 10:28:33,226 INFO    [59dfa641357f695c57aa50] error:133 - Masking the original 404 message: 'The path '/en-US/custom/lookup_editor/lookup_edit/save' was not found.' with 'Page not found!' for security reasons
2017-10-12 10:27:03,380 INFO    [59dfa5e75f7f695c637710] error:133 - Masking the original 404 message: 'The path '/en-US/custom/lookup_editor/lookup_edit/get_lookup_backups_list' was not found.' with 'Page not found!' for security reasons
2017-10-12 10:27:03,315 INFO    [59dfa5e74e7f698052f550] error:133 - Masking the original 404 message: 'The path '/en-US/custom/lookup_editor/lookup_edit/get_lookup_contents' was not found.' with 'Page not found!' for security reasons

[LukeMurphey]

Could you verify that the following two files exist on one of the affected search heads?

1. $SPLUNK_HOME/etc/apps/lookup_editor/appserver/controllers/lookup_edit.py
2. $SPLUNK_HOME/etc/apps/lookup_editor/default/web.conf

If those exist, then make sure that web.conf includes the following:

[endpoint:lookup_edit]

[mikelanghorst]

Both exist, and web.conf has that content on all 3 of my SHC nodes.

[LukeMurphey]

BTW: sorry for the extensive debugging questions. I tried several times to reproduce this in various environments (various Splunk versions, SHC and standalone, etc) but couldn't get it to reproduce for me.