Deployment Architecture

Why am I unable to start 2 of 4 search heads with "WARNING: web interface does not seem to be available!"?

jamesoconnell
Path Finder

I have four search head instances running in our NONPROD environment, but am getting the following error below when starting two of the four instances.

We are running Splunk Enterprise Splunk 6.2.0 (build 237341)

Regards,
-James O'Connell

...

 All preliminary checks passed.
 Starting splunk server daemon (splunkd)...
 Done
                                                       [  OK  ]
 Waiting for web server at http://127.0.0.1:38000 to be available
 WARNING: web interface does not seem to be available!
0 Karma

sloshburch
Splunk Employee
Splunk Employee

When I've had this issue, I chalked it up to entropy on the system over time. Assuming the config and the ports are all good, then I start with some simple and basic things:

  1. Run splunk stop and then do a kill -9 on the results of: ps -ef | grep -i splunk | grep -v grep | awk '{print $2;}' to stop anything that didn't actually shut down
  2. Restart the actual server thereby purging any zombie processes that might not be needed but are still running from usage over time
  3. When all else fails, reinstall that instance. This can be made easy through proper use of the configuration distribution features

renjith_nair
SplunkTrust
SplunkTrust

Are you running all instances on same machine? If that's the case make sure your ports are not conflicting.

Check splunkd.log for errors. One of your configuration might be incorrect or there are some conflicts.

Also check if mongod instance (kvstore) is shutdown properly and no process is running. We had an issue where stale process of mongo was running which prevents re-starting.

As always, log files are the first place to look for any issues.

Happy Splunking!
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...