Why does the Splunk Universal Forwarder shut down after assigning app to client for upgrade?

[mygoalfinder]

Hello, I am upgrading 800 splunk universal forwarders with Red Hat Linux OS. Using this custom app

When I assign this custom app to  the universal forwarders, it shuts down and does not start again. With version  7.2.6, it works fine. With no issue. Anything after that version, the script does not work.

03-21-2023 12:26:05.128 -0500 INFO PipelineComponent - Performing early shutdown tasks
03-21-2023 12:26:05.128 -0500 INFO loader - Shutdown HTTPDispatchThread
03-21-2023 12:26:05.128 -0500 INFO ShutdownHandler - Shutting down splunkd
03-21-2023 12:26:05.128 -0500 INFO ShutdownHandler - shutting down level "ShutdownLevel_Begin"
03-21-2023 12:26:05.128 -0500 INFO ShutdownHandler - shutting down level "ShutdownLevel_FileIntegrityChecker"
03-21-2023 12:26:05.128 -0500 INFO ShutdownHandler - shutting down level "ShutdownLevel_JustBeforeKVStore

[PickleRick]

Ugh. You should _not_ do that. Trying to manipulate stuff outside of directories managed by Splunk (and trying to overwrite whole $SPLUNK_HOME is just that generally speaking) is asking for trouble.

Same goes for using a script spawned from splunkd to shut down splunkd - there's no guarantee that splunkd will not kill hard all its child processes (in fact I'd count on it).

Not to mention troubleshooting this whole process and such (which you've just shown).

So no, ugly hacks using scripting is not a replacement for a configuration management software. BTW, you're using RH - why install from tarball when you have a well working RPM archive?I'd rather go for creating a custom repo containing the forwarder RPMs and autoyum if I didn't have any external tools.