Splunkweb crashes with "Bad file descriptor"

[MattK]

Splunk 4.1.4 on Win2008 R2, splunkweb is repeatedly terminating with "Bad file descriptor" error shown in web_serivce.log, requiring a restart of the service. A uninstall/reinstall has not resolved this issue.

Any known issues, solutions?

2010-11-09 16:25:35,484 ERROR   [4cd975ff585418320] module:62 - [Errno 9] Bad file descriptor
Traceback (most recent call last):
  File "F:\Splunk\Python-2.6\Lib\site-packages\splunk\appserver\mrsparkle\controllers\module.py", line 60, in renderModule
    pageContent = instance.generateResults(**args)
  File "F:\Splunk\share\splunk\search_mrsparkle\modules\results\SimpleResultsTable.py", line 137, in generateResults
    adjustedMin, adjustedMax = util.getPercentiles(ordered, .05, .95)
  File "F:\Splunk\Python-2.6\Lib\site-packages\splunk\appserver\mrsparkle\lib\util.py", line 454, in getPercentiles
    return (v(lowerPercentile, orderedList), v(upperPercentile, orderedList))
  File "F:\Splunk\Python-2.6\Lib\site-packages\splunk\appserver\mrsparkle\lib\util.py", line 446, in v
    n, k, d = f(percentile, len(oList))
  File "F:\Splunk\Python-2.6\Lib\site-packages\splunk\appserver\mrsparkle\lib\util.py", line 442, in f
    print n,k,d
IOError: [Errno 9] Bad file descriptor
2010-11-09 16:36:32,776 ERROR   [4cd96d628f2342080] root:120 - ENGINE: Error in HTTP server: shutting down
Traceback (most recent call last):
  File "F:\Splunk\Python-2.6\Lib\site-packages\cherrypy\process\servers.py", line 73, in _start_http_thread
    self.httpserver.start()
  File "F:\Splunk\Python-2.6\Lib\site-packages\cherrypy\wsgiserver\__init__.py", line 1665, in start
    self.tick()
  File "F:\Splunk\Python-2.6\Lib\site-packages\cherrypy\wsgiserver\__init__.py", line 1730, in tick
    prevent_socket_inheritance(s)
  File "F:\Splunk\Python-2.6\Lib\site-packages\cherrypy\wsgiserver\__init__.py", line 1474, in prevent_socket_inheritance
    raise WinError()
WindowsError: [Error 6] The handle is invalid.

2010-11-09 16:36:32,792 INFO    [4cd96d628f2342080] root:120 - ENGINE: Bus STOPPING
2010-11-09 16:36:51,809 INFO    [4cd96d628f2342080] root:120 - ENGINE: HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('0.0.0.0', 8000)) shut down
2010-11-09 16:36:51,825 INFO    [4cd96d628f2342080] root:120 - ENGINE: Stopped thread 'Monitor'.
2010-11-09 16:36:51,855 INFO    [4cd96d628f2342080] root:120 - ENGINE: Stopped thread 'Monitor'.
2010-11-09 16:36:51,917 INFO    [4cd96d628f2342080] root:120 - ENGINE: Stopped thread 'Monitor'.
2010-11-09 16:36:51,980 INFO    [4cd96d628f2342080] root:120 - ENGINE: Stopped thread '_TimeoutMonitor'.
2010-11-09 16:36:51,980 ERROR   [4cd96d628f2342080] root:120 - ENGINE: Removed handler for console events.
2010-11-09 16:36:51,980 WARNING [4cd96d628f2342080] custompidfile:36 - Removing pidfile at F:\Splunk\var\run\splunk\splunkweb.pid  (PID: 8080)"
2010-11-09 16:36:51,980 INFO    [4cd96d628f2342080] root:120 - ENGINE: Bus STOPPED
2010-11-09 16:36:51,980 INFO    [4cd96d628f2342080] root:120 - ENGINE: Bus EXITING
2010-11-09 16:36:51,980 INFO    [4cd96d628f2342080] root:120 - ENGINE: Bus EXITED

[tskinnerivsec]

Are there any updates for this issue? I'm currently seeing this on a very busy Enterprise Security Search head. It seems to happen about 20 times in a month period. Would this get resolved by increasing the ulimit settings on the linux server, or is there some limits.conf setting that should be looked at on the search head? This is Splunk Enterprise 6.2, so it is still happening on newer versions. One other caveat, it resides in amazon cloud.,

[claudio_manig]

Same here - any news?

[grahampoulter]

I have exactly the same thing. Running Splunk 4.1.6 on Windows Server 2008 R2 64-bit, sometimes when I change my search rapidly (edit query it in the middle of a search and re-start the search, repeat a couple of times), Splunk Web crashes. Very annoying. This is the error in web_service.log:

2010-12-03 12:37:27,052 ERROR   [4cf8bcbcda1ee6208] root:141 - ENGINE: Error in HTTP server: shutting down
Traceback (most recent call last):
  File "C:\Program Files\Splunk\Python-2.6\Lib\site-packages\cherrypy\process\servers.py", line 73, in _start_http_thread
    self.httpserver.start()
  File "C:\Program Files\Splunk\Python-2.6\Lib\site-packages\cherrypy\wsgiserver\__init__.py", line 1665, in start
    self.tick()
  File "C:\Program Files\Splunk\Python-2.6\Lib\site-packages\cherrypy\wsgiserver\__init__.py", line 1730, in tick
    prevent_socket_inheritance(s)
  File "C:\Program Files\Splunk\Python-2.6\Lib\site-packages\cherrypy\wsgiserver\__init__.py", line 1474, in prevent_socket_inheritance
    raise WinError()
WindowsError: [Error 6] The handle is invalid.

I have applied the patch and will note in a couple of days whether the crash recurs.

[sideview]

in case it helps -- i've seen that 'bad file descriptor' stack trace too once in a while on 4.1.5, but I've never seen it trigger a shutdown.

[araitz · Accepted solution]

See:

http://www.cherrypy.org/ticket/1016

Please reply back here if applying the patch to the specified file proves successful.

[Flynt]

This particular link is now inactive. Any information on a current link?

[araitz]

Thanks for reporting back. As noted in the bug, what a weird fix!

[MattK]

Based on previous history of multiple crashes per day, and a full day since patching without issue, it looks like this was the solution.

[MattK]

Patch applied. If I don't get another crash in a day, I will assume issue is fixed. Will report my findings.