Splunk Search

Cannot connect remotely to Splunk Web interface

reswob4
Builder

I have a problem where I cannot remotely access the web interface (not via HTTPS or HTTP on either 8000 or 8089) of one of my Windows HF.

However, when I RDP to the server, I can launch a browser and connect to either the loopback IP (https://127.0.0.1:8000) or the public IP (https://x.x.x.x:8000)

If I check wireshark, I see the syn packets come in, but no response at all being sent out.

This, of course, points to the firewall being a problem, but I checked the firewall and it is configured to accept all connections from any IP on any port for the Splunk application (and the splunk web application, but if I remember correctly, that's not used anymore). I tried disabling the firewall by going to Windows Firewall with Advanced Security on Local Computer Properties and turning the firewall state to OFF, but I still couldn't connect.

This started sometime in the past month. Since I don't routinely connect to this interface, I don't know when it stopped working. And it means it used to work without outgoing firewall rules configured.

There are NO references to failed connections or anything similar in splunkd.log. I enabled the firewall text log, but that genereated no entrieds either (or I'm doing that wrong).

Here's the output from the startup:

Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
Checking critical directories... Done

All preliminary checks passed.

Starting splunk server daemon (splunkd)...

Splunkd: Starting (pid 4312)
Done

Waiting for web server at https://127.0.0.1:8000 to be available... Done

If you get stuck, we're here to help.
Look for answers here: http://docs.splunk.com

The Splunk web interface is at https://splunk-04:8000

C:>"Program Files\Splunk\bin"\splunk status
Splunkd: Running (pid 4312)

Here's the output for the ports from netstat

C:>netstat -anob | findstr 8000
TCP 0.0.0.0:8000 0.0.0.0:0 LISTENING 6852

C:>netstat -anob | findstr 8089
TCP 0.0.0.0:8089 0.0.0.0:0 LISTENING 6852

Here are some links to other things I've tried:

https://answers.splunk.com/answers/26598/splunk-receiver-listening-on-9997-but-only-on-localhost.htm...

https://answers.splunk.com/answers/179823/splunk-always-starts-at-1270018000.html

https://docs.splunk.com/Documentation/Splunk/6.6.2/Admin/BindSplunktoanIP

neither setting SPLUNK_BINDIP=0.0.0.0 or SPLUNK_BINDIP= works

The only things I know have changed (besides patches being applied) are java 1.8 and Logstash 5.4.1 were installed. Logstash is NOT running.

While I go create some outbound firewall rules to see if that works, I figured I'd post here to see if anyone else has any ideas. I also still have to look in the Windows Event log to see if there are entries there...

Thanks for any suggestions.

Tags (2)
0 Karma

woodcock
Esteemed Legend

Check that server for a rogue web.conf with startwebserver = 0; run this on the CLI:

$SPLUNK_HOME/bin/splunk btool list web --debug

BTW, best practice is to leave the splunkweb off on everything except for your Search Heads and to admin via DS or some other configuration manager.

0 Karma

reswob4
Builder

FYI, you have list and web reversed. But regardless, my startwebserver = 1 so that's not it.

Since I have other pans burning in the fire, and the HF is working doing what a HF is supposed to do, I think I'll let this go for now and manage via the cli.

Thanks

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...