My CentOS 7 server is not listening to port 8000

[neroi]

Hello!
I have a problem with splunkweb daemon:

root@srv # ./splunk status
splunkd is running (PID: 32010)
splunk helpers are running (PIDs: 32016 32073)

root@srv # ./splunk start splunkweb
splunkd is running (PID: 32010)
splunk helpers are running (PIDs: 32016 32073)

root@srv # ./splunk set web-port 8081
HTTP/1.1 404 Not Found

this happened after I completed the process of restarting the splunkweb daemon.
(After installing the addon for Linux, splunkweb did not restart)

[neroi]

Thank you all for the answers!
Stopping the light Forwarder app solved the issue.

[jkat54 · Accepted solution]

grep -i error /opt/splunk/var/log/splunk/splunkd.log

[neroi]

Hello!
there is my problem:

[root@srvsplunk bin]# ./splunk restart
Stopping splunkd...
Shutting down. Please wait, as this may take a few minutes.
........ [ OK ]
Stopping splunk helpers...
[ OK ]
Done.

Splunk> Be an IT superhero. Go home early.

Checking prerequisites...
Checking mgmt port [8089]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
Validated: _thefishbucket firedalerts os test
Done
Checking filesystem compatibility... Done
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunk/splunk-6.6.1-aeae3fe0c5af-linux-2.6-x86_64-manifest'
All installed files intact.
Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...

Done
[ OK ]
[root@srvsplunk bin]# ./splunk show web-port
Your session is invalid. Please login.
Splunk username: admin
Password:
Web port: 8000
[root@srvsplunk bin]# ./splunk restart splunkweb
splunkweb is not running. [СБОЙ ]
[root@srvsplunk bin]# ./splunk start splunkweb
[root@srvsplunk bin]# ./splunk status
splunkd is running (PID: 19517).
splunk helpers are running (PIDs: 19520 19560 19593 19667 19929).
splunkweb is not running.
[root@srvsplunk bin]#

[jkat54]

grep -i error /opt/splunk/var/log/splunk/splunkd.log | grep -v nix

What’s the output of this?

[jkat54]

So what was the error?

[neroi]

Thank you for the answer!

There are to many of errors:

11-02-2017 15:05:09.806 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/service.sh" To see services enabled on particular target use
11-02-2017 15:05:09.806 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/service.sh" 'systemctl list-dependencies [target]'.
11-02-2017 15:05:09.878 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/time.sh" Error resolving 0.centos.pool.ntp.org: Name or service not known (-2)
11-02-2017 15:05:09.878 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/time.sh" 2 Nov 15:05:09 ntpdate[7774]: Can't find host 0.centos.pool.ntp.org: Name or service not known (-2)
11-02-2017 15:05:09.878 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/time.sh" 2 Nov 15:05:09 ntpdate[7774]: no servers can be used, exiting
11-02-2017 15:06:37.872 +0300 ERROR TcpOutputProc - Processing server from outputs.conf: can't resolve a valid IP address for host="splunkaday-linux-light"
11-02-2017 15:06:38.189 +0300 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/django_error.log'.
11-02-2017 15:06:38.293 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/rlog.sh" Redirecting to /bin/systemctl status auditd.service
11-02-2017 15:06:38.354 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/time.sh" Error resolving 0.centos.pool.ntp.org: Name or service not known (-2)
11-02-2017 15:06:38.354 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/time.sh" 2 Nov 15:06:38 ntpdate[8441]: Can't find host 0.centos.pool.ntp.org: Name or service not known (-2)
11-02-2017 15:06:38.354 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/time.sh" 2 Nov 15:06:38 ntpdate[8441]: no servers can be used, exiting
11-02-2017 15:06:38.354 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/service.sh" Note: This output shows SysV services only and does not include native
11-02-2017 15:06:38.354 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/service.sh" systemd services. SysV configuration data might be overridden by native
11-02-2017 15:06:38.354 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/service.sh" systemd configuration.
11-02-2017 15:06:38.354 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/service.sh" If you want to list systemd services use 'systemctl list-unit-files'.
11-02-2017 15:06:38.354 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/service.sh" To see services enabled on particular target use
11-02-2017 15:06:38.354 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/service.sh" 'systemctl list-dependencies [target]'.
11-02-2017 15:12:07.920 +0300 ERROR TcpOutputProc - Processing server from outputs.conf: can't resolve a valid IP address for host="splunkaday-linux-light"
11-02-2017 15:13:32.747 +0300 ERROR TcpOutputProc - Processing server from outputs.conf: can't resolve a valid IP address for host="splunkaday-linux-light"
11-02-2017 15:13:33.135 +0300 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/django_error.log'.
11-02-2017 15:13:33.273 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/rlog.sh" Redirecting to /bin/systemctl status auditd.service
11-02-2017 15:13:33.316 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/time.sh" Error resolving 0.centos.pool.ntp.org: Name or service not known (-2)
11-02-2017 15:13:33.317 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/time.sh" 2 Nov 15:13:33 ntpdate[9319]: Can't find host 0.centos.pool.ntp.org: Name or service not known (-2)
11-02-2017 15:13:33.317 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/time.sh" 2 Nov 15:13:33 ntpdate[9319]: no servers can be used, exiting
11-02-2017 15:13:33.320 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/service.sh" Note: This output shows SysV services only and does not include native
11-02-2017 15:13:33.320 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/service.sh" systemd services. SysV configuration data might be overridden by native
11-02-2017 15:13:33.320 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/service.sh" systemd configuration.
11-02-2017 15:13:33.320 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/service.sh" If you want to list systemd services use 'systemctl list-unit-files'.
11-02-2017 15:13:33.320 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/service.sh" To see services enabled on particular target use
11-02-2017 15:13:33.320 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/service.sh" 'systemctl list-dependencies [target]'.
11-02-2017 15:16:05.826 +0300 ERROR TcpOutputProc - Processing server from outputs.conf: can't resolve a valid IP address for host="splunkaday-linux-light"
11-02-2017 15:16:06.163 +0300 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/django_error.log'.
11-02-2017 15:16:08.435 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/service.sh" Note: This output shows SysV services only and does not include native
11-02-2017 15:16:08.435 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/service.sh" systemd services. SysV configuration data might be overridden by native
11-02-2017 15:16:08.435 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/service.sh" systemd configuration.
11-02-2017 15:16:08.435 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/service.sh" If you want to list systemd services use 'systemctl list-unit-files'.
11-02-2017 15:16:08.435 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/service.sh" To see services enabled on particular target use
11-02-2017 15:16:08.435 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/service.sh" 'systemctl list-dependencies [target]'.
11-02-2017 15:16:08.437 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/rlog.sh" Redirecting to /bin/systemctl status auditd.service
11-02-2017 15:16:08.437 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/time.sh" Error resolving 0.centos.pool.ntp.org: Name or service not known (-2)
11-02-2017 15:16:08.437 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/time.sh" 2 Nov 15:16:06 ntpdate[10471]: Can't find host 0.centos.pool.ntp.org: Name or service not known (-2)
11-02-2017 15:16:08.437 +0300 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/time.sh" 2 Nov 15:16:06 ntpdate[10471]: no servers can be used, exiting
11-02-2017 15:21:43.052 +0300 ERROR TcpOutputProc - Processing server from outputs.conf: can't resolve a valid IP address for host="splunkaday-linux-light"

[neroi]

Hello!

[root@srv-asutp bin]# ./splunk restart
Stopping splunkd...
Shutting down. Please wait, as this may take a few minutes.
[ OK ]
Stopping splunk helpers...
[ OK ]
Done.

Splunk> Map. Reduce. Recycle.

Checking prerequisites...
Checking mgmt port [8089]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
Validated: _thefishbucket sample unix_summary
Done
Checking filesystem compatibility... Done
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunk/splunk-7.0.0-c8a78efdd40f-linux-2.6-x86_64-manifest'
All installed files intact.
Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...

Done
[ OK ]
[root@srv-asutp bin]# ./splunk status
splunkd is running (PID: 18364).
splunk helpers are running (PIDs: 18366 18427).

./splunk restart
does not help

[neroi]

This happens after upload the linux addon for splunk

[mattymo]

what happens?? I see a successful restart there...now what port are we serving on??

[neroi]

Hello!

[root@srv-asutp bin]# ./splunk restart
Stopping splunkd...
Shutting down. Please wait, as this may take a few minutes.
[ OK ]
Stopping splunk helpers...
[ OK ]
Done.

Splunk> Map. Reduce. Recycle.

Checking prerequisites...
Checking mgmt port [8089]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
Validated: _thefishbucket sample unix_summary
Done
Checking filesystem compatibility... Done
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunk/splunk-7.0.0-c8a78efdd40f-linux-2.6-x86_64-manifest'
All installed files intact.
Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...

Done
[ OK ]
[root@srv-asutp bin]# ./splunk status
splunkd is running (PID: 18364).
splunk helpers are running (PIDs: 18366 18427).

./splunk restart
does not help

[mattymo]

Hey neroi,

splunkweb is part of splunkd, and has been for a while now..what version of splunk are you on??

also what does the ouptut of show web-port say?

[splunker@n00bserver bin]$ ./splunk show web-port
Web port: 8000

check who is running splunk as well:

ps -ef | grep splunkd

Also don't forget to add rules to firewalld!

[neroi]

Thank you for comment!

[root@srv-asutp bin]# ./splunk show web-port
Web port: 8081

i think, (./splunk set web-port 8081) was executed correctly.

[root@srv-asutp bin]# ./splunk set web-port 8000
The server's web port has been changed.
You need to restart the Splunk Web Server (splunkweb) for your changes to take effect.
[root@srv-asutp bin]# ./splunk restart splunkweb
HTTP/1.1 404 Not Found
[root@srv-asutp bin]# ./splunk show web-port
Web port: 8000

[root@srv-asutp bin]# ps -ef | grep splunkd
root 2742 31025 0 18:51 pts/0 00:00:00 grep --color=auto splunkd
root 32010 1 0 15:50 ? 00:00:11 splunkd -p 8089 restart
root 32016 32010 0 15:50 ? 00:00:00 [splunkd pid=32010] splunkd -p 8089 restart [process-runner]
root 32073 32016 0 15:50 ? 00:00:12 /opt/splunk/bin/splunkd instrument-resource-usage -p 8089

I have empty iptable and firewall use default profile with "any to any accept"

[mattymo]

honestly, despite it working on my server, i never use ./splunk restart splunkweb...a while back splunkweb was moved into splunkd itself....so i wonder if when you try to run the restart of the web after the change its tripping splunkd up....have you tried just using ./splunk restart ?

[neroi]

yes, i tried

[jkat54]

Yes exactly, you should be running ./splunk restart

Not

./splunk restart splunkweb