Cloning and shutting down an indexer

[skippylou]

So originally I thought the way cloning would work between two indexers was that data from a forwarder setup to clone would send identical events to each indexer - this part does work fine. However, when I bring down one indexer for a few minutes on a restart or whatnot, I thought the forwarders would send to the indexer that was up, while keeping track of what needed to be sent to the second indexer when it came back online.

It appears the indexer that was down never gets the data that it missed while it was down once it comes back online. Does cloning work in that as long as one of the indexers is online that is good enough for those events? Doesn't seem right as I would think cloned should be identical, but it appears that's what I'm seeing.

Is the only way around this to clone to two sets of autolb'ed indexers (ie. using four total indexers)? Or what I'm seeing I shouldn't be seeing?

Thanks,

Scott

[gkanapathy]

Regardless of what the documentation currently, using forwarder "cloning" to make a copy of your data is a bad idea and doesn't do what you would expect, precisely because of what you see. I would go so far as to say the documented recommendations are wrong.

What you should instead do is forward to your primary indexer(s), then each primary indexer would in turn forward to a matching secondary indexer, i.e., each primary would have a "clone" backup. This setup gives you what you're probably actually looking for.

If you are using a cluster of primary indexers and load balancing over them, then if one goes down, the rest of the cluster will be fine. In turn, the secondary that is connected to the primary will stop receiving data, and therefore will remain in sync.

[skippylou]

Thanks for the clarification. I meant I was going to go the cloned to two distinct lb groups, not feed from the primary downstream - but I understand what you describe as an option like that if I wanted to go that route.

[gkanapathy]

Well, just to be clear, the secondary set is not really LB. Each secondary server is "clone" of one of the primaries (the primaries are LB by the forwarders). I think that is what I said, but I want to be sure you don't forward from the primaries to all of the secondaries, but only one each.

[skippylou]

Thanks, I think my best bet at this point is the latter of what you meantion, cluster to two lb sets.

[jbsplunk · Accepted solution]

That is exactly the intended behavior. You configure your forwarders to send duplicate data to 2 indexers, and if one is down, the other can be used. In case of data lost, one can be used to refill the other, but a merge can be troublesome because you are going to have overlapping buckets. The forwarders have no way of knowing where they need to pick up in order to make the data identical. Auto-LB would ensure that in the group, all data is recieved. If you had a search head that was set up to use both indexers as search peers, this would allow you to be sure that between the two indexers, you'd have the entire set of data.

http://www.splunk.com/base/Documentation/4.2/Deploy/Clonedatad

[skippylou]

Thanks for the answer. I guess in my mind I think of cloned as two things having exact where autolb would be the spread amongst the members with at least one member getting a copy. All in all, I'll have to move to the combo of the two. Thanks again.