Deployment Architecture

Rysnc Search Heads

brdr
Contributor

Hi Splunk,

We will be using rsync to keep our 2 enterprise search heads in sync. I found a Splunk wiki page on this topic but it is for an older version (3.3) of Splunk. We are using Splunk 6.4. Is there a newer wiki or another documented resource on what to keep in sync between the 2 search heads?

Thank you

ddrillic
Ultra Champion

alacercogitatus
SplunkTrust
SplunkTrust

This is not feasible, as there are only 2 servers. SHC requires 3.

0 Karma

alacercogitatus
SplunkTrust
SplunkTrust

Indeed, SHP (search head pooling) is no longer a viable option (as of 6.2). SHC (search head clustering) requires minimum 3 search heads. A better way of keeping things in sync between the search heads is to use a Deployment Server.

Deployment server can keep your Splunk Apps in Sync between the search heads, however, local configurations will not be there. Same thing for users. That being said, what one search head does won't be transferred to another search head, and neither is aware of the other. Can you sync the dispatch folder? Technically yes, however, the sids won't exist on the other search head so it might get confused. So the best you can do is have 2 different search heads with the same users and apps. Any data or local changes would need to be rolled back into the DS to be pushed back down or rsynced. But don't rsync the search artifacts. It just won't work like you think it will.

To continue the discussion, join us on IRC (#splunk on EFNet), or Slack (www.splunk402.com/chat for access). Happy Splunking!

== EDIT ==
Changed the version on SHP from 6.3 to 6.2.

brdr
Contributor

Thank you. When you say search artifacts do you mean the users history?

Example:

user/user1/search/history/searchhead.csv

I would join the IRC but work prohibits chat sessions.

0 Karma

dshpritz
SplunkTrust
SplunkTrust

Search artifacts are not just the user's history, but also remnants/results from previous searches. For example, if you have a scheduled search that runs and sends out a link to the results, that link is to search artifacts rather than rerunning the search.

inventsekar
Ultra Champion

Starting with 4.2, you can use search head pooling to share configurations and user data between search heads. So, the Search Head Pooling with LDAP authentication, you do not need to rsync the apps, users, and dispatch jobs among search heads, anymore.

Note: Search head pooling has been deprecated in Splunk Enterprise version 6.2.
Search Head Cluster takes care of this one.

0 Karma

dshpritz
SplunkTrust
SplunkTrust

For the love of Benji, don't do SHP.

0 Karma

woodcock
Esteemed Legend

I must admit, I have not done so myself, but surely it is better than rolling one's own rsync, yes?

0 Karma

dshpritz
SplunkTrust
SplunkTrust

Barely. SHP has a lot of performance problems in most cases, as it is CIFS/NFS based. Rsync provides different levels of pain. From what I have seen in the past, attempting to sync search heads via rsync produces weirdness (at best). Best option here, unfortunately, is to get another search head. Splunk recommends that servers in SHC be similar, which makes sense, as SHC doesn't really have a concept of "weighting". However, perhaps a small VM that sees no end-user searches.

0 Karma

woodcock
Esteemed Legend

OK, so just add the Deployment Server as the 3rd Search Head for Clustering but do not allow anybody to login to it. That should solve everything, right?

0 Karma

dshpritz
SplunkTrust
SplunkTrust

No, it still wouldn't be perfect. You still need a deployer, which would most likely be the DS. So another system would still be needed.

0 Karma

brdr
Contributor

hello there,

we can't head SH clustering as we only have 2 search heads, and I read somewhere search head pooling was deprecated... I will check out your links.. thank you!

0 Karma

woodcock
Esteemed Legend

As Splunk says:

This feature has been deprecated as of Splunk Enterprise version 6.2. This means that although it continues to function, it might be removed in a future version.

So deprecated does not mean unuseable.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...