Child processes by lea_loggrabber do not terminate making later-triggered-parent-process into hung state. This happens only for one of the entity (lets call it entity1) while for other (entity2) it is fine.
Due to this at the time of next run (after 60 secs), splunk finds the loggrabber for entity1 already running and does not trigger another instance and runs the loggrabber for entity2.
Hence, we are able to pull logs from entity2 however not from entity1.
Any suggestion about troubleshooting this would be much appreciated.
Splunk 6.0.4 (build 207768)
Splunk_TA_opseclea_linux22 - Version: 1.11.1
Posting below are the running processes at this moment (including child and hung parent processes) -
[XXXXX@XXXXXXXXX ~]$ ps -ef | grep splunk
splunk 3869 1 0 May21 ? 00:00:03 ./lea_loggrabber --configentity LEAXX.XX.XX.XX --appname Splunk_TA_opseclea_linux22
splunk 6503 1 0 May23 ? 00:03:04 ./lea_loggrabber --configentity LEAXX.XX.XX.XX --appname Splunk_TA_opseclea_linux22
root 9109 8644 0 04:55 ? 00:00:00 sshd: splunk [priv]
splunk 9149 9109 0 04:55 ? 00:00:00 sshd: splunk@pts/0
splunk 9150 9149 0 04:55 pts/0 00:00:00 -bash
splunk 10069 9150 0 04:59 pts/0 00:00:00 ps -ef
splunk 10070 9150 0 04:59 pts/0 00:00:00 grep splunk
splunk 12320 1 0 May21 ? 00:01:38 ./lea_loggrabber --configentity LEAXX.XX.XX.XX --appname Splunk_TA_opseclea_linux22
splunk 13010 1 0 May21 ? 00:01:24 ./lea_loggrabber --configentity LEAXX.XX.XX.XX --appname Splunk_TA_opseclea_linux22
splunk 13065 1 0 May21 ? 00:01:36 ./lea_loggrabber --configentity LEAXX.XX.XX.XX --appname Splunk_TA_opseclea_linux22
splunk 14604 30083 0 May26 ? 00:00:00 /bin/bash /opt/splunk/etc/apps/Splunk_TA_opseclea_linux22/bin/lea-loggrabber.sh --configentity LEAXX.XX.XX.XX
splunk 14611 14604 1 May26 ? 00:03:55 ./lea_loggrabber --configentity LEAXX.XX.XX.XX --appname Splunk_TA_opseclea_linux22
splunk 24306 1 0 May25 ? 00:06:12 ./lea_loggrabber --configentity LEAXX.XX.XX.XX --appname Splunk_TA_opseclea_linux22
splunk 29825 1 0 May21 ? 00:01:04 ./lea_loggrabber --configentity LEAXX.XX.XX.XX --appname Splunk_TA_opseclea_linux22
splunk 30082 1 12 May26 ? 02:22:02 splunkd -p 8089 restart
splunk 30083 30082 0 May26 ? 00:01:08 [splunkd pid=30082] splunkd -p 8089 restart [process-runner]
splunk 30248 1 0 May26 ? 00:01:58 python -O /opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/root.py restart
splunk 30874 1 0 May21 ? 00:00:45 ./lea_loggrabber --configentity LEAXX.XX.XX.XX --appname Splunk_TA_opseclea_linux22
splunk 30904 1 0 May21 ? 00:00:51 ./lea_loggrabber --configentity LEAXX.XX.XX.XX --appname Splunk_TA_opseclea_linux22
