Full HA with only 2 servers?

[Kindred]

Hi,

Is it possible to have a truly resilient/replicated Splunk setup with only 2 servers?

Right now we have 1 server that is handling indexing and searching, but we'd like to duplicate this server and have it also active as an indexer/searcher - so both servers have the same indexed data and either can be used to search this data.

Clustering seems to split the indexers but needs more than 2 instances and a separate search head. I've seen references to Index and Forwarding, but I don't know if you can do this bi-directionally (each server is configured to index and forward to the other, but would that cause a loop?).

What can we do with just 2 servers?

[amit_saxena]

Hi,

Clustering seems to work exactly in your case but you don't want to have more than 2 VMs / hosts in your infrastructure. Either increase the number of nodes and opt for Splunk supported clustering or use external HA solution like VMotion etc.

Regards,
Amit Saxena

[bmacias84]

Check out these two posts:
http://splunk-base.splunk.com/answers/60572/splunk-disaster-recovery

http://splunk-base.splunk.com/answers/62791/data-replication-across-two-indexers

[sowings]

Syncing or cloning data is only part of it.

You'd also have to think about search artifacts, user-generated content, etc. This would typically be achieved with something Splunk calls "Search Head Pooling" that uses an NFS share to provide a common location for the multiple search heads to agree upon who's doing what. NFS share means NFS server, which probably therefore dictates a third machine. Cross-linking the Splunk servers by turning them both into NFS servers and mounting partitions from one another and then symlinking the locations to make it appear as though they're "one same" partition to the search head sounds... painful.

Sync or double-index the data. Periodically sync up the user-generated content ($SPLUNK_HOME/etc/users/), and expect a small amount of exposure to loss of some of that specific stuff if the "search head I'm currently connected to" goes down.

The data will still be there, they can simply run another search if their search head dies midstream.

[jgedeon120]

Yes it is possible but you will need to have the available license for it as you are indexing the data twice.

Set up you would set your forwarders to send data to both indexers in a non load balancing manner.

[tcpout]
defaultGroup = indexera,indexerb

[tcpout:indexera]
server = indexera:9997

[tcpout:indexerb]
server = indexerb:9997

[jgedeon120]

Well then you will need to add another servers to the mix and one would be a Splunk master node. You will then be able to create index replication. You will not be able to use the indexers as a search head.

Looking at what you are trying to accomplish you should be looking at a Splunk 5 cluster.

[Kindred]

It's more about keeping data in sync. If IndexerA went down for 24 hours, when it came back up it would be missing 24 hours worth of data. We need both servers to be identical to each other.

Edit: as an example, think MySQL master-master setup

[jgedeon120]

You're using each indexer as a stand alone indexer search head. What do you want to sync? If it is the apps and users then you can mount those on an NFS mount.

[Runals]

Data would simply sent to both locations vs sync. If you wanted to duplicate objects like saved searches you'd need to look at doing rsync or something.

[Kindred]

How would the indexers sync with each other if this method is used? Is this using the Index and Forward method pointing each indexer to the other indexer?