Splunkd is not running after executing add monitor command

[sravan2j]

Hi,

I installed splunk on Ubuntu 12.04 64-bit in GoGrid.

I have 8 clusters (1 master, 1 search node, 3 indexers, 3 forwarders). Installation completed successfully. But when I executed the command "ADD monitor \var\log" command in any of the indexer or in forwarder, to monitor \var\log directory, then splunkd stops running in that particular indexer or forwarder. Again when I remove monitor, everything works fine. Could you please let me know what may be the issue. I tried deleting all the clusters and created them once again and installed splunk. But I am facing the same problem again. Please let me know what I need to do know. Thanks for your help.

[sritej]

I have added monitors on 2 indexers and works fine. When i try to add monitor on indexer to 3 through web interface im facing an error that splunkd stopped working in that particular indexer.

in the master web interface i see only 2 indexers active and 1 indexer as down. and there is a warning symbol beside replication factor not met.

Any idea how to resolve this.

Thanks,
Sri Tej N.

[sravan2j]

Hi,

Below link contains the crash log when "splunkd stops running"

http://pastebin.com/yHKxVDLU

[yannK]

looks like a permission issue :
- make sure that you call the command under the same user than the user running splunk (to have permissions to write the files)
- if you have any type of search-head pooling (with shared storage for your configuration) double check the permissions on the files and shared storage.

[sravan2j]

Hi,

I installed splunk as root and I am executing the "add monitor" command as root.

I have only one search-head in my cluster definition, there is no search-head pooling.

Below link contains the crash log when "splunkd stops running"

http://pastebin.com/yHKxVDLU

[sravan2j]

Hi,
I am using latest version i.e., 6.0.
I am not able to find any errors in splunkd.log file. But I am able to see some errors in splunkd_stderr.log file. Below is the error, i found in that file,

Conf mutator lockfile has disappeared; error condition possible.

When splunkd is not running, it created some log files, but those file names are different and I am not sure where they are saved. In that log file, I have seen that it is not able to find manifest file.

Please let me know do you want to see log file. I will try to recreate the scenario. Thanks for your help

[lukejadamec]

Which version of Splunk are you using, and what errors are you seeing the splunkd.log?