"Splunk must be restarted" message will not go away.

[I_am_Jeff]

Running version 4.2.3 on a dedicated indexer.

A few days ago we got the dreaded, "Splunk must be restarted for changes to take effect. Click here to restart from the Manager," message.

• We restart Splunk, both via the command line and through the GUI. The message reappears.
• We stop Splunk via the command line. Wait several minutes and restart. The message reappears.
• We click "Clear Restart Message" and the message reappears immediately.

Any ideas? Even a hint on what logs to look at?

[hexx]

Note that the SoS app in its version 2.x should not be installed on your indexers, only on your search-head. For best practices on deploying SoS in a distributed environment, please read this Splunk Answer.

[tmeader]

I actually opened a bug on this with Splunk (don't know if they've identified the actual issue yet) and there is a workaround available in S.o.S 2.1:

http://splunk-base.splunk.com/answers/37102/persistent-splunk-must-be-restarted-for-changes-to-take-...

Hope that helps you too.

[hexx]

Indeed, the issue that @tmeader is referencing is a core Splunk bug which has been filed under reference SPL-46736. For more details, please read the Splunk Answer referenced above.

[hexx · Accepted solution]

Please check the following Splunk Answer to see if it matches the issue you are encountering and if the proposed work-around will work for you.

[I_am_Jeff]

SoS is version 2.1.0 on both indexers.

[I_am_Jeff]

Update

I disabled S.o.S. Message appeared saying something like SoS was disabled and an index size change was made, please restart. Restarted and message has been gone for the last 30 minutes. (I previously tried changing the size of an index via the GUI to see if it would get the message to go away after a restart.) I have another indexer with SoS enabled, but no messages about restarting on that one.

I enabled SoS and message reappears. "User 'iamjeff' triggered the 'disable' action on app 'sos', and the following objects required a restart: indexes."

SoS disabled. No message again.