I am a new user to Linux and Splunk. I have a CentOS 6.2 x64 VM running on a Windows 2008 R2 SP1 environment. I had installed Splunk 5.0.2 to the server and have ports 8000 & 8089 opened at the firewall on the VM. I had accessed the Webpage and was in the process of downloading additional apps for splunk when the system crashed. Now the splunkd service will not remain running.
I have stopped and started the splunk services utilizing the ./splunk stop & ./splunk start commands. I have the services set to auto start upon reboots. I have also reinstalled the Splunk server but still am experiencing the issue.
[root@splunk_gsa_slc bin]# ./splunk start
Splunk> Finding your faults, just like mom.
Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Checking configuration... Done.
Checking indexes...
Validated databases: _thefishbucket cntfconf cntflogs cntfscrl cntfscrm cntfscrs jmx os sample sos sos_summary_daily splunk_monitoring summary_forwarders summary_hosts summary_indexers summary_pools summary_sources summary_sourcetypes websphere wi_summary_daily wi_summary_fivemin wi_summary_hourly
Done
Checking filesystem compatibility... Done
WARN IniFile - /opt/splunk/etc/apps/splunk_monitoring/local/tags.conf, line 2: Cannot parse into key-value pair: www
Possible typo in stanza [perfmon://CPUTime] in /opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf, line 72: counters = % Processor Time;% User Time
Possible typo in stanza [perfmon://CPUTime] in /opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf, line 74: instances = _Total
Possible typo in stanza [perfmon://CPUTime] in /opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf, line 75: interval = 10
Possible typo in stanza [perfmon://CPUTime] in /opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf, line 76: object = Processor
Possible typo in stanza [perfmon://FreeDiskSpace] in /opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf, line 80: counters = Free Megabytes;% Free Space
Possible typo in stanza [perfmon://FreeDiskSpace] in /opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf, line 82: instances = *
Possible typo in stanza [perfmon://FreeDiskSpace] in /opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf, line 83: interval = 10
Possible typo in stanza [perfmon://FreeDiskSpace] in /opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf, line 84: object = LogicalDisk
Possible typo in stanza [perfmon://Memory] in /opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf, line 88: counters = % Committed Bytes In Use;Available MBytes;Committed Bytes
Possible typo in stanza [perfmon://Memory] in /opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf, line 90: interval = 10
Possible typo in stanza [perfmon://Memory] in /opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf, line 91: object = Memory
Possible typo in stanza [perfmon://LocalNetwork] in /opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf, line 95: counters = Bytes Received/sec;Bytes Sent/sec;Bytes Total/sec;Current Bandwidth
Possible typo in stanza [perfmon://LocalNetwork] in /opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf, line 97: instances = *
Possible typo in stanza [perfmon://LocalNetwork] in /opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf, line 98: interval = 10
Possible typo in stanza [perfmon://LocalNetwork] in /opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf, line 99: object = Network Interface
Possible typo in stanza [connection_failed] in /opt/splunk/etc/apps/SplunkforF5Security/default/eventtypes.conf, line 3: viewstate.resultView = normalView
Possible typo in stanza [connection_success] in /opt/splunk/etc/apps/SplunkforF5Security/default/eventtypes.conf, line 7: viewstate.resultView = normalView
WARN IniFile - /opt/splunk/etc/apps/webintelligence/default/macros.conf, line 21: Cannot parse into key-value pair: ~
Possible typo in stanza [webping] in /opt/splunk/etc/apps/webping/default/props.conf, line 4: INE_BREAKER = (\nWebPingProcessor|WebPingProcessor)
Possible typo in stanza [webping] in /opt/splunk/etc/apps/webping/default/props.conf, line 6: XMUST_BREAK_AFTER = WebPingProcessor
Possible typo in stanza [webping] in /opt/splunk/etc/apps/webping/default/props.conf, line 7: XBREAK_ONLY_BEFORE = neverbreakshere
There might be typos in your conf files. For more information, run 'splunk btool check --debug'
Checking conf files for typos... Done
All preliminary checks passed.
Starting splunk server daemon (splunkd)... Done
[ OK ]
[ OK ]
Starting splunkweb... Done
If you get stuck, we're here to help.
Look for answers here: http://docs.splunk.com
The Splunk web interface is at http://splunk_gsa_slc:8000
After the above indication that the services were started I immediately checked the status of splunk and the results are shown below.
[root@splunk_gsa_slc bin]# ./splunk status
splunkd 2700 was not running.
Removing stale pid file... done.
splunkweb is running (PID: 2737).
root@splunk_gsa_slc bin]# ./splunk display boot-start
Init script installed at /etc/init.d/splunk.
Init script is configured to run at boot.
[root@splunk_gsa_slc bin]# ps -ef | grep splunk
avahi 1422 1 0 13:24 ? 00:00:00 avahi-daemon: registering [splunkgsaslc.local]
root 2651 2261 0 13:49 pts/0 00:00:00 grep splunk
This issue has now been resolved.
I completely uninstalled Splunk from the server utilizing the instructions found at http://docs.splunk.com/Documentation/Splunk/latest/installation/UninstallSplunk. After I uninstalled Splunk I checked the status of all services running on the server to see if any Splunk service was still being referenced on the server. This showed me the following error at /etc/init.d/splunk: line 34: /opt/splunk/bin/splunk: No such file or directory.
I deleted the referenced file and reinstalled Splunk per the instructions found at http://docs.splunk.com/Documentation/Splunk/latest/Installation/InstallonLinux. After I installed Splunk again I verified that I could access the Splunk Websites on port 8000 and 8089, the default ports for Splunk.
I ran in an additional issue after I restarted the Splunk services where splunkweb did not start. To rectify this issue I had to copy the web.conf from the default directory to the /opt/splunk/etc/system/local/web.conf. Once I did that I was able to successfully restart Splunk services and have splunkweb restart as well.