I have downloaded the eventgen master from github, extracted to SA-Eventgen folder & also downloaded the McAfee add-on (Splunk-TA-Mcafee) which contains sample data & an eventgen.conf on my heavy forwarder.
This all works nicely, I am getting data into my main index, generated on the HF & sprayed across two indexers, but it appears that when I stop splunk on the HF - the python eventgen process is not killed / stopped.
You can see below that the child eventgen.py (PID 13292) is being called by /bin.sh (PID 13291)
When I stop Splunk - the child eventgen.py process (PID 13292) is still running.
Stopping / Starting splunk causes multiple of these processes to run, duplicating the generated data.
Has anyone seen this ? / know what could be causing it ?
root@DEV-vFWD01:~# ps -ef | grep python
root 13259 13207 0 05:00 ? 00:00:08 /opt/splunk/bin/python -O /opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/root.py --proxied=127.0.0.1,8065,8000
root 13291 13207 0 05:00 ? 00:00:00 /bin/sh -c python /opt/splunk/etc/apps/SA-Eventgen/bin/eventgen.py
root 13292 13291 2 05:00 ? 00:01:01 python /opt/splunk/etc/apps/SA-Eventgen/bin/eventgen.py
root 13408 8844 0 05:45 pts/0 00:00:00 grep --color=auto python
root@DEV-vFWD01:~# /opt/splunk/bin/splunk stop
Stopping splunkd...
Shutting down. Please wait, as this may take a few minutes.
..
Stopping splunk helpers...
Done.
root@DEV-vFWD01:~# ps -ef | grep python
root 13292 1 2 05:00 ? 00:01:02 python /opt/splunk/etc/apps/SA-Eventgen/bin/eventgen.py
root 13439 8844 0 05:46 pts/0 00:00:00 grep --color=auto python
root@DEV-vFWD01:~#
No, your problem is the use of dash as the default shell in Ubuntu. Run the following:
debconf-set-selections <<< "dash dash/sh string false"
dpkg-reconfigure -f noninteractive dash
This will set your default shell to bash and the problem will likely go away.
I've filed a bug against dash in Ubuntu for this misbehaviour. Searching the internet shows its cropped up in various guises over the years depending on which application spawned dash as /bin/sh (e.g. PHP), and in every case the accepted solution was to stop using dash. As someone who isn't fond of bash (zsh FTW! 🙂 I don't think that solution is long-term acceptable.
No, your problem is the use of dash as the default shell in Ubuntu. Run the following:
debconf-set-selections <<< "dash dash/sh string false"
dpkg-reconfigure -f noninteractive dash
This will set your default shell to bash and the problem will likely go away.
I am marking this as correct as it resolved the problem - the question now remains as to "why" this is happening.
You can see below that indeed dash/sh was set to true, then changing it to false, I no longer get the issue.
root@splunkbox:~# debconf-show dash
* dash/sh: true
root@splunkbox:~# debconf-set-selections <<< "dash dash/sh string false"
root@splunkbox:~# debconf-show dash
* dash/sh: false
root@splunkbox:~# dpkg-reconfigure -f noninteractive dash
Removing 'diversion of /bin/sh to /bin/sh.distrib by dash'
Adding 'diversion of /bin/sh to /bin/sh.distrib by bash'
Removing 'diversion of /usr/share/man/man1/sh.1.gz to /usr/share/man/man1/sh.distrib.1.gz by dash'
Adding 'diversion of /usr/share/man/man1/sh.1.gz to /usr/share/man/man1/sh.distrib.1.gz by bash'
root@splunkbox:~# debconf-show dash
* dash/sh: false
root@splunkbox:~#
Python is no longer called with /bin/sh -c
root@splunkbox:~# ps -ef | grep python
root 5571 5514 0 20:25 ? 00:00:00 python /opt/splunk/etc/apps/file_meta_data/bin/file_meta_data.py
root 5572 5514 1 20:25 ? 00:00:01 /opt/splunk/bin/python -O /opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/root.py --proxied=127.0.0.1,8065,8000
root 5640 5514 0 20:25 ? 00:00:00 python /opt/splunk/etc/apps/syndication/bin/syndication.py
root 5645 5514 0 20:25 ? 00:00:00 python /opt/splunk/etc/apps/website_input/bin/web_input.py
root 6479 1918 0 20:28 pts/0 00:00:00 grep --color=auto python
root@splunkbox:~# /opt/splunk/bin/splunk stop
Stopping splunkd...
Shutting down. Please wait, as this may take a few minutes.
..
Stopping splunk helpers...
Done.
root@splunkbox:~# ps -ef | grep python
root 6843 1918 0 20:29 pts/0 00:00:00 grep --color=auto python
root@splunkbox:~#
Support ticket has been closed & documentation updated: http://docs.splunk.com/Documentation/Splunk/6.2.4/Installation/InstallonLinux#Default_shell
Bottom line - dont use dash 😄
i have noticed the same problem in Ubuntu 14.04. Up voting for asking it 🙂
I have also logged a Splunk support ticket 252386, will post back once they work it out.
Just to provide some more feedback, I built a new 14.04 machine, and I see the same problem with the syndication RSS app too https://splunkbase.splunk.com/app/2646/
I dont think it is App specific, but as you say, specific to Ubuntu 14.04 - the investigation continues.......
What operating system?
Ubuntu 14.04
Although I havent tried to reproduce it on another machine, I think the eventgen was installed through the GUI then the folder was renamed from eventgen-master to SA-Eventgen afterwards - maybe this could contribute to the issue.
I've seen this before as well. Modular Inputs in Splunk 5.X used to do it. It has something to do with the way they are spawned as inputs, and they don't check the parent process after spawning. Windows does not have this problem (oddly enough).
The only "fix" that I know of would be to edit the python file "eventgen.py" to add conditions that check the parent pid that spawned the process. If it is 1 (meaning it went to root of the system, the Splunk daemon disappeared), then exit the script.
I'll ping the author on it, maybe he can shed some additional light.
We already do that :).