Search Affinity and Searchable replicas: What is the search head behavior if we lose a clustered indexer with searchable data on it?

[gavsdavs_GR]

We would like to make a change to the number of searchable replicas we keep in our environment - this is (possibly) a quick way to win back a fair amount of disk space, but I would like clarification on a question before I do this.

We currently have the following indexer cluster replication configuration:

site_replication_factor = origin:1, site1:2, site2:1, site3:2, total:5
site_search_factor      = origin:1, site1:2, site2:1, site3:2, total:5

So we have 2 searchable copies of our data in sites 1 and 3, where we have our search heads (No search heads in Site 2)

I am considering changing that to:

site_replication_factor = origin:1, site1:2, site2:1, site3:2, total:5
site_search_factor      = origin:1, site1:1, site2:1, site3:1, total:3

This means we'll have ONE searchable replica in each site, but an additional unsearchable copy in two sites.

Can you explain the search head behavior if we lose an indexer with searchable data on it?
1. I assume the indexer with the unsearchable copies begins to enrich them - this takes time.
2. A replica gets sent to the remaining 3 indexer in site to ensure SRF is met (sites 1 and 3)
3. What happens in search - does a site 1 Search head get results from the searchable copies in sites 2 or 3, or does a search head get incomplete results until the non-searchable copies in site 1 are fully enriched and searchable ?

I am unsure of the actual behavior here:
- If a search head strictly honors 'site affinity', it won't get results as it isn't allowed to get them from the non-local indexers
- If search head affinity is simply 'a preference', then I anticipate the SH will get results from the searchable copies in the remote sites.

Can someone explain the actual behavior as we can discard a searchable copy in 2 sites and win back a load of disk space.

[lguinn2 · Accepted solution]

Search head affinity is 'a preference'

When a search head contacts the cluster master, the cluster master will direct it to search peers on the same site as the search head. But if the data is not available at that site, the cluster master will direct the search head to other sites to retrieve the data.

[stanwin]

Looks like the behaviour is different for 7.1.0?

http://docs.splunk.com/Documentation/Splunk/7.1.0/Indexer/Multisitesearchaffinity

When search affinity is functioning,
each search head sends its searches to
all peers, across all sites, but only
the local peers search their data and
return results to the search head.

If a local peer holding some of the
searchable data goes down and the site
temporarily loses its valid state, the
search head will, if necessary, get
data from peers on remote sites while
the local site is undergoing bucket
fixing. During this time, the search
head will still get as much of the
data as possible from the local site.

[gjanders]

This section:

If a local peer holding some of the
searchable data goes down and the site
temporarily loses its valid state, the
search head will, if necessary, get
data from peers on remote sites while
the local site is undergoing bucket
fixing.

Confirms Iguinn's post. If the local peer (i.e. the peer on the site you have requested) goes down then the search head can receive data from remote sites (i.e. other sites in Iguinn's post).

[waechtler]

Not 100%ly the same:

docu says: "if peer .. goes down", then remote sites will be searched
Iguinn says: "if data is not available at that site " ...

Example
A Multisite customer wants to store some data on local indexers only, should be searchable only by searchheads on same site
Idea would be to exclude this index=X from replication

According to documentation, Search affinity should then restrict access to users on the local searchhead, because site ID is respected
According to Iguinn, it would not help, because searches for index=X will be carried out on remote sites as well, if index=X is not locally available

[gavsdavs_GR]

If you're in a multi-site cluster, I don't think you can prevent an index from being replicated across sites.

You can stop an index from being indexed entirely so you'll only have it available on the indexer it arrived on. Depends how you're managing which indexers are getting which data.