Security

Why am I getting "Error in HTTP server: shutting down" when starting splunkweb?

letsrumbl
Engager

Hello all !

Environment:
- OS - Ubuntu 12.04.6
- Splunk - splunk-6.1.3-220630-linux-2.6-amd64.deb
- Installed via dpkg
- Independent installation

I'm having an issue with starting splunkweb on the first go.

Here is the error from web_services.log

2014-10-03 19:17:09,447 INFO    [542ef6340610d2450] root:138 - ENGINE: Bus STARTING
2014-10-03 19:17:09,475 INFO    [542ef6340610d2450] root:138 - ENGINE: Started monitor thread '_TimeoutMonitor'.
2014-10-03 19:17:09,492 ERROR   [542ef6340610d2450] root:138 - ENGINE: Error in HTTP server: shutting down
Traceback (most recent call last):
  File "/opt/splunk/lib/python2.7/site-packages/cherrypy/process/servers.py", line 73, in _start_http_thread
    self.httpserver.start()
  File "/opt/splunk/lib/python2.7/site-packages/cherrypy/wsgiserver/__init__.py", line 1725, in start
    raise socket.error, msg
error: [Errno 13] Permission denied

2014-10-03 19:17:09,492 INFO    [542ef6340610d2450] root:138 - ENGINE: Bus STOPPING
2014-10-03 19:17:09,492 ERROR   [542ef6340610d2450] root:138 - ENGINE: Error in 'stop' listener <bound method Server.stop of <cherrypy._cpserver.Server object at 0x10d2350>>
Traceback (most recent call last):
  File "/opt/splunk/lib/python2.7/site-packages/cherrypy/process/wspbus.py", line 147, in publish
    output.append(listener(*args, **kwargs))
  File "/opt/splunk/lib/python2.7/site-packages/cherrypy/process/servers.py", line 107, in stop
    self.httpserver.stop()
  File "/opt/splunk/lib/python2.7/site-packages/cherrypy/wsgiserver/__init__.py", line 1909, in stop
    self.requests.stop(self.shutdown_timeout)
  File "/opt/splunk/lib/python2.7/site-packages/cherrypy/wsgiserver/__init__.py", line 1458, in stop
    self._monitor_thread.join()
AttributeError: 'NoneType' object has no attribute 'join'

2014-10-03 19:17:09,494 INFO    [542ef6340610d2450] root:138 - ENGINE: Stopped thread '_TimeoutMonitor'.

To replicate:

Any time I restart splunk (/etc/init.d/splunk restart OR $SPLUNK_HOME/bin/splunk restart) the splunkd restarts fine but web_services.log throws the error above.

If I follow the restart with a splunk start splunkweb the web server comes up with no errors. Main problem with this is that on system start, web server does not fire off and I have to manually log in to start it.

Tried:
- Reinstalling package
- Changing permissions
- Changed to root (to test)
- Also verified port is not in use before starting (shouldn't matter on restart)

Anyone encounter this issue ?

Tags (2)

grijhwani
Motivator

Looks to me like you previously started Splunk as root (started on boot?), and now you are trying to start it as some unpriveleged user (perhaps the splunk user itself) as a manual start.

letsrumbl
Engager

When I added it to the startup scripts I tried both combinations (as root user and as splunk).

Regardless of what I did both scenarios yielded the same outcome.

I believe that if it was a permissions issue, and I were to run splunk start splunkweb as root, it would start as expected. No matter which user I use, the first time I run splunk start splunkweb it renders in the error above. As I explained above, if this is followed by the same command immediately after, the service starts correctly.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...