After I stopped both indexers and cleaned the eventdata from both main indexes it will it will not re replicate, any ideas how to fix this?
I had a similar issue, where after cleaning an index (and the associated fishbucket on the forwarder), the cluster-master cluster management page reported the index as non-searchable (and no replicated or searchable copies), just like your screen capture. This was interesting, because I was able to successfully run searches through the search head, and saw activity to the index in the splunkd.log.
I was able to clear it up by restarting splunk on the cluster master. It seems to be a UI bug in the cluster management screen.
I am going to assume that you are monitoring files here, I'm also assuming you have "cleaned" the indexes from the command line using, "./splunk clean
"...
To cut a long story short, Splunk uses CRC and other mechanisms to record whether a file has been monitored or not, and (if tailing) how far through that file it has read. This is to avoid duplication in normal circumstances.
You will need to clean make sure _thefishbucket
index is cleaned as well.. However, if you have other indexes that are still working from before the clean that you performed, it may (and probably will) cause duplication in the indexing of events to those indexes. If you only use the main
index then you should be fine. Please read the following.
http://blogs.splunk.com/2008/08/14/what-is-this-fishbucket-thing/
To clean a specific index (again,I'm assuming you have already done this):
./splunk clean eventdata -index <indexName>