Monitoring Splunk

splunkd service will not remain running

ellisj1
New Member

I am a new user to Linux and Splunk. I have a CentOS 6.2 x64 VM running on a Windows 2008 R2 SP1 environment. I had installed Splunk 5.0.2 to the server and have ports 8000 & 8089 opened at the firewall on the VM. I had accessed the Webpage and was in the process of downloading additional apps for splunk when the system crashed. Now the splunkd service will not remain running.

I have stopped and started the splunk services utilizing the ./splunk stop & ./splunk start commands. I have the services set to auto start upon reboots. I have also reinstalled the Splunk server but still am experiencing the issue.

[root@splunk_gsa_slc bin]# ./splunk start
Splunk> Finding your faults, just like mom.
Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Checking configuration... Done.
Checking indexes...
Validated databases: _thefishbucket cntfconf cntflogs cntfscrl cntfscrm cntfscrs jmx os sample sos sos_summary_daily splunk_monitoring summary_forwarders summary_hosts summary_indexers summary_pools summary_sources summary_sourcetypes websphere wi_summary_daily wi_summary_fivemin wi_summary_hourly
Done
Checking filesystem compatibility... Done
WARN IniFile - /opt/splunk/etc/apps/splunk_monitoring/local/tags.conf, line 2: Cannot parse into key-value pair: www
Possible typo in stanza [perfmon://CPUTime] in /opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf, line 72: counters = % Processor Time;% User Time
Possible typo in stanza [perfmon://CPUTime] in /opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf, line 74: instances = _Total
Possible typo in stanza [perfmon://CPUTime] in /opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf, line 75: interval = 10
Possible typo in stanza [perfmon://CPUTime] in /opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf, line 76: object = Processor
Possible typo in stanza [perfmon://FreeDiskSpace] in /opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf, line 80: counters = Free Megabytes;% Free Space
Possible typo in stanza [perfmon://FreeDiskSpace] in /opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf, line 82: instances = *
Possible typo in stanza [perfmon://FreeDiskSpace] in /opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf, line 83: interval = 10
Possible typo in stanza [perfmon://FreeDiskSpace] in /opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf, line 84: object = LogicalDisk
Possible typo in stanza [perfmon://Memory] in /opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf, line 88: counters = % Committed Bytes In Use;Available MBytes;Committed Bytes
Possible typo in stanza [perfmon://Memory] in /opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf, line 90: interval = 10
Possible typo in stanza [perfmon://Memory] in /opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf, line 91: object = Memory
Possible typo in stanza [perfmon://LocalNetwork] in /opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf, line 95: counters = Bytes Received/sec;Bytes Sent/sec;Bytes Total/sec;Current Bandwidth
Possible typo in stanza [perfmon://LocalNetwork] in /opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf, line 97: instances = *
Possible typo in stanza [perfmon://LocalNetwork] in /opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf, line 98: interval = 10
Possible typo in stanza [perfmon://LocalNetwork] in /opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf, line 99: object = Network Interface
Possible typo in stanza [connection_failed] in /opt/splunk/etc/apps/SplunkforF5Security/default/eventtypes.conf, line 3: viewstate.resultView = normalView
Possible typo in stanza [connection_success] in /opt/splunk/etc/apps/SplunkforF5Security/default/eventtypes.conf, line 7: viewstate.resultView = normalView
WARN IniFile - /opt/splunk/etc/apps/webintelligence/default/macros.conf, line 21: Cannot parse into key-value pair: ~
Possible typo in stanza [webping] in /opt/splunk/etc/apps/webping/default/props.conf, line 4: INE_BREAKER = (\nWebPingProcessor|WebPingProcessor)
Possible typo in stanza [webping] in /opt/splunk/etc/apps/webping/default/props.conf, line 6: XMUST_BREAK_AFTER = WebPingProcessor
Possible typo in stanza [webping] in /opt/splunk/etc/apps/webping/default/props.conf, line 7: XBREAK_ONLY_BEFORE = neverbreakshere
There might be typos in your conf files. For more information, run 'splunk btool check --debug'
Checking conf files for typos... Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)... Done
[ OK ]
[ OK ]
Starting splunkweb... Done

If you get stuck, we're here to help.

Look for answers here: http://docs.splunk.com

The Splunk web interface is at http://splunk_gsa_slc:8000

After the above indication that the services were started I immediately checked the status of splunk and the results are shown below.

[root@splunk_gsa_slc bin]# ./splunk status
splunkd 2700 was not running.
Removing stale pid file... done.
splunkweb is running (PID: 2737).

root@splunk_gsa_slc bin]# ./splunk display boot-start
Init script installed at /etc/init.d/splunk.
Init script is configured to run at boot.

[root@splunk_gsa_slc bin]# ps -ef | grep splunk
avahi 1422 1 0 13:24 ? 00:00:00 avahi-daemon: registering [splunkgsaslc.local]
root 2651 2261 0 13:49 pts/0 00:00:00 grep splunk

Tags (1)
0 Karma

ellisj1
New Member

This issue has now been resolved.

I completely uninstalled Splunk from the server utilizing the instructions found at http://docs.splunk.com/Documentation/Splunk/latest/installation/UninstallSplunk. After I uninstalled Splunk I checked the status of all services running on the server to see if any Splunk service was still being referenced on the server. This showed me the following error at /etc/init.d/splunk: line 34: /opt/splunk/bin/splunk: No such file or directory.

I deleted the referenced file and reinstalled Splunk per the instructions found at http://docs.splunk.com/Documentation/Splunk/latest/Installation/InstallonLinux. After I installed Splunk again I verified that I could access the Splunk Websites on port 8000 and 8089, the default ports for Splunk.

I ran in an additional issue after I restarted the Splunk services where splunkweb did not start. To rectify this issue I had to copy the web.conf from the default directory to the /opt/splunk/etc/system/local/web.conf. Once I did that I was able to successfully restart Splunk services and have splunkweb restart as well.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...