slave-apps doesn't like my app

[vanvan]

Hi,

I built a simple app with the add-on builder (using the python script inputs) and then copied it in the "master-apps" of my cluster master.

After successful bundle replication I see the app is on the indexers as well, but I noticed some strange things immediately:

1. The icons used in the app were missing from the GUI and were replaced by the default "app" icon.
2. When I tried to access the app from the menu on the indexers, I got only a blank screen with only "Splunk>" logo in it...
3. The _internal logs were showing that my custom app is missing some python scripts...

I already checked app sizes on Master and Peers and it is the same everywhere. No files are missing from the app...

Any ideas why this might happen? Some strange limitations on "slave-apps" content maybe?

[vanvan]

Ok, I have a workaround that seems to solve the issue...but it's dirty.

The root cause is that splunkd does not search for any scripts,icons,css,whatever resources when apps are under "slave-apps". This is how it works currently, I don't know why. If you create a symbolic link with the same name under "apps" directory the problem will disappear.

E.g. you have an app, called "TA-myapp" and you deploy it through the Cluster Master to "slave-apps" on the Cluster Peers. The app path would be something like "/opt/splunk/etc/slave-apps/TA-myapp". In order to solve the issue with python scripts not found, simply run the command:
ln -s /opt/splunk/etc/slave-apps/TA-myapp /opt/splunk/etc/apps/TA-myapp

This will create a symbolic link under "/opt/splunk/etc/apps" that points to the app under "slave-apps" and then Splunkd would be able to find the missing resource. This will fix also issues with missing icons (e.g. like in the case of some other add-ons like TA for PaloAlto).

[atownson]

I can confirm the same behavior with such apps as config_explorer. Thanks for the workaround.

[woodcock]

Are you using Deployment Server and is it running WIndows OS?

[vanvan]

No,

I am using the Cluster Master (with its master-apps directory) and it is running on SuSE Linux.

[richgalloway]

Don't deploy scripted inputs to indexer clusters. Each indexer will run the script(s) and you'll end up with duplicated data.
Instead, deploy the app to a heavy forwarder.

[richgalloway]

Yes, it is related to the issue you are having. If you weren't trying to do something that shouldn't be done, you wouldn't have the issue.

[vanvan]

Ok 🙂

I am trying to run scripts on all the indexers in order to do some housekeeping locally on the machine. The resulting output is collected as events, which are relatable only to the local machine, e.g. they are not "duplicate" in any sense whatsoever. Kinda like the stuff in "_internal", you know?

If the peers weren't in a cluster I would've used the Deployment Server and probably it would've worked, but since the peers are part of an indexer cluster...guess I'm stuck with the master-apps method.
Which gives me the issue...

What you are talking about is perfectly fine, but doesn't really explain why it doesn't/shouldn't work in my case.

[vanvan]

Ok, thanks for the feedback, although it isn't related to the issue, I'm afraid.

I think I found something else though... The permissions look very different when doing "ls -al" on master-apps compared to the slave-apps... Maybe that's the reason.

[vanvan]

...but it isn't...Still getting the same results, even when the permissions are 100% same...