If I restart the splunkd manually on the client side, I can start it up so I don't think it's typo or any configuration error.
This is the client that just got new app deployed. it received signal from deployment server and shutdown itself successfully.
it just doesn't come back up again.
VVV
tail -f /opt/splunkforwarder/var/log/splunk/splunkd.log
07-25-2018 21:03:57.382 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_LoadLDAPUsers"
07-25-2018 21:03:57.382 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_MetricsManager"
07-25-2018 21:03:57.382 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_Pipeline"
07-25-2018 21:03:57.382 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_Queue"
07-25-2018 21:03:57.382 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_CallbackRunner"
07-25-2018 21:03:57.382 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_HttpClient"
07-25-2018 21:03:57.383 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_DmcProxyHttpClient"
07-25-2018 21:03:57.383 +0000 INFO ShutdownHandler - shutting down level
"ShutdownLevel_Duo2FAHttpClient"
07-25-2018 21:03:57.383 +0000 INFO ShutdownHandler - Shutdown complete in
1561.8 milliseconds
07-25-2018 21:03:58.322 +0000 INFO loader - All pipelines finished.
... View more