Getting Data In

Splunkd is not running after executing add monitor command

sravan2j
Explorer

Hi,

I installed splunk on Ubuntu 12.04 64-bit in GoGrid.

I have 8 clusters (1 master, 1 search node, 3 indexers, 3 forwarders). Installation completed successfully. But when I executed the command "ADD monitor \var\log" command in any of the indexer or in forwarder, to monitor \var\log directory, then splunkd stops running in that particular indexer or forwarder. Again when I remove monitor, everything works fine. Could you please let me know what may be the issue. I tried deleting all the clusters and created them once again and installed splunk. But I am facing the same problem again. Please let me know what I need to do know. Thanks for your help.

0 Karma

sritej
Engager

I have added monitors on 2 indexers and works fine. When i try to add monitor on indexer to 3 through web interface im facing an error that splunkd stopped working in that particular indexer.

in the master web interface i see only 2 indexers active and 1 indexer as down. and there is a warning symbol beside replication factor not met.

Any idea how to resolve this.

Thanks,
Sri Tej N.

0 Karma

sravan2j
Explorer

Hi,

Below link contains the crash log when "splunkd stops running"

http://pastebin.com/yHKxVDLU

0 Karma

yannK
Splunk Employee
Splunk Employee

looks like a permission issue :
- make sure that you call the command under the same user than the user running splunk (to have permissions to write the files)
- if you have any type of search-head pooling (with shared storage for your configuration) double check the permissions on the files and shared storage.

0 Karma

sravan2j
Explorer

Hi,

I installed splunk as root and I am executing the "add monitor" command as root.

I have only one search-head in my cluster definition, there is no search-head pooling.

Below link contains the crash log when "splunkd stops running"

http://pastebin.com/yHKxVDLU

sravan2j
Explorer

Hi,
I am using latest version i.e., 6.0.
I am not able to find any errors in splunkd.log file. But I am able to see some errors in splunkd_stderr.log file. Below is the error, i found in that file,

Conf mutator lockfile has disappeared; error condition possible.

When splunkd is not running, it created some log files, but those file names are different and I am not sure where they are saved. In that log file, I have seen that it is not able to find manifest file.

Please let me know do you want to see log file. I will try to recreate the scenario. Thanks for your help

0 Karma

lukejadamec
Super Champion

Which version of Splunk are you using, and what errors are you seeing the splunkd.log?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...