Persistent "Splunk must be restarted for changes to take effect" UI banner when accessing the Splunk on Splunk app's home view

[hexx]

I have upgraded the Splunk on Splunk app to version 2.0. Whenever I go to the home view of the app, a message appears in the UI stating that "Splunk must be restarted for changes to take effect". In a distributed search environment, I actually get one message for each search peer that my search-head can reach. The only way to get rid of it is to restart the Splunk instance(s) reported, but the messages come back every time I go back to S.o.S' home view.

[splunksurekha]

From where to get the modified home.xml file ???

[hexx · Accepted solution]

The main search in the home.xml view (the one powering the "A glimpse of your Splunk instance" panel) of the SoS app retrieves the values of SPLUNK_HOME and SPLUNK_DB from the REST API endpoint @ https://[splunkd_host]:[splunkd_management_port]/services/server/settings.

It appears that in some cases, when this endpoint is hit, it improperly triggers the Splunk restart UI message. This is a core Splunk bug which has been filed under reference SPL-46736.

Until this bug is fixed in core Splunk, the SoS development team will provide a work-around. To set it up in your environment, please follow these steps on the instance where you installed the SoS app and in accordance with the installed version:

Steps for SoS 2.0:

To work around this issue on SoS 2.0, we will use a modified home.xml file which prevents which disables the offending portion of the search.

• Get a copy of the modified home.xml file. You'll have to use your splunk.com credentials to download this file.
• Make a backup of your original home.xml:
  cp $SPLUNK_HOME/etc/apps/sos/default/data/ui/views/home.xml $SPLUNK_HOME/etc/apps/sos/default/data/ui/views/home.xml.old
• Copy the modified home.xml file in place:
  cp home_SUP-368.xml $SPLUNK_HOME/etc/apps/sos/default/data/ui/views/home.xml
  • Reload the view on your search-head by pointing your browser to http[s]://[splunkweb_host]:[splunkweb_port]/debug/refresh?entity=admin/views
  • Clear the restart messages by restarting splunkd on the affected instances. There doesn't seem to be any other way to achieve this, unfortunately.
  • Hit the SoS app home view again @ http[s]://:/app/sos/home

NOTE: Until the root cause is fixed in a new core Splunk release and your instance is upgraded to that version, this operation will need to be performed each time SoS is upgraded to a newer version. Alternatively, you can upgrade to SoS 2.1 and use the work-around provided just below which will persist through further SoS upgrades.

Steps for SoS 2.1:

To work around this issue on SoS 2.1, we will modify the default/macros.conf file to modify the search that triggers this issue.

• Copy $SPLUNK_HOME/etc/apps/sos/default/macros.conf to $SPLUNK_HOME/etc/apps/sos/local/macros.conf
• Edit $SPLUNK_HOME/etc/apps/sos/local/macros.conf
• As instructed on line 23 of that file, comment out the first definition of the macro get_splunk_instances_info on line 21 and uncomment the alternative definition located on line 25.
• Restart splunkd
  or
• Dynamically reload search macros by hitting the following URL: http[s]://:/debug/refresh?entity=admin/macros
  • Hit the SoS app home view again @ http[s]://:/app/sos/home

You should no longer see any UI messages indicating the need to restart Splunk coming from your search peers at that point.