I'm running Splunk Enterprise v 6.6.1 on Windows 2008 R2 (not by choice). Without making any configuration changes (that I'm aware of) one user has started receiving "500 internal server errors" when trying to access the Search & Reporting app. Other apps are not presenting this issue. All other users are fine. The errors are only present when UserA opens the Search & Reporting app.
The error message links to a search for index=_internal source=web_service.log requestid=[\xx] . When looking at the log file web_service.log in notepad++, there is no matching request id.
splunkd_acces.log is not showing any errors. All the entries for 127.0.0.1 with UserA have http status 200
There are entries in splunkd_ui_access.log and web_access.log with the HTTP 500 error and matching username and timestamp, but they useful for finding the problem. They only show the GET request, user-Agent, HTTP status, and request ID (web_acces) or session ID (splunkd_ui_access).
127.0.0.1 - [username] [date&time] "GET /en-US/app/search/search HTTP/1.1" 500 3037 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36" - [ID] 707ms
Restarting splunkd and the user's hosts have not had any impact. Threat/Socket limit is well above what we actually use, and if they were exhausted I would expect to see errors in splunkd.log and for all users to be seeing http 500 erros.
Has anyone else experienced an issue like this? Are there any log files other than those in [$SPLUNK_HOME$]\var\log\splunk that could help?
... View more