Deployment Architecture

error message "replication was unsuccessful"

dmlee
Communicator

During one of my searches, I got this following error message "unable to distribute to peer named splunk_index01 at url https://10.1.6.1:8089 because replication was unsuccessful replicationStatus in progress". Any ideas on why it would occur and what it means? we have one search head and two index servers.

Tags (1)
1 Solution

the_wolverine
Champion

We've been dealing with these bundle replication errors (and resulting slow search kicking off) very frequently until recently when a workaround was configured for us. The workaround involves NFS export of the bundles to the searchpeers and a few other configuration changes. Splunk support should be able to assist if you have tried the other suggestions and are still encountering this issue.

View solution in original post

0 Karma

the_wolverine
Champion

We've been dealing with these bundle replication errors (and resulting slow search kicking off) very frequently until recently when a workaround was configured for us. The workaround involves NFS export of the bundles to the searchpeers and a few other configuration changes. Splunk support should be able to assist if you have tried the other suggestions and are still encountering this issue.

0 Karma

the_wolverine
Champion

How does useClientSSLCompression setting affect this behavior? I set 'useSplunkdClientSSLCompression' to false and was still getting the error (although it seems, less often.)

I've just set 'useClientSSLCompression' to false as well.

0 Karma

dmlee
Communicator

according to Stephen's replied, we should set "enableSplunkdSSL=false" on indexer but not search head. FYR.

the_wolverine
Champion

I have assumed that all the suggested changes need to go on the search head... but I don't know now since it doesn't appear to be working reliably for me. Still getting the error occasionally.

0 Karma

dmlee
Communicator

which splunk instance should I modify ? search head or index server ? I modified search head's server.conf and distsearch.conf , I set useClientSSLCompression=false, useSplunkdClientSSLCompression=false, enableSplunkdSSL=false
, it still doesn't work .
I had filed a support case for this.

0 Karma

Stephen_Sorkin
Splunk Employee
Splunk Employee

Replication is the process of moving the search-time configuration from the search head to the indexers. This is usually very fast, but can take some time if the indexers are across a WAN or if the apps on the search head are particularly large. We've made some improvements in 4.1.5 to make replication faster, but it can still be slow for large apps.

If you are on an intranet, you can make it faster by adding to server.conf:

[sslConfig]
useSplunkdClientSSLCompression = false

Or you could even disable SSL entirely:

[sslConfig]
enableSplunkdSSL = false

Finally, you can find large assets in your apps, and disable replication in distsearch.conf:

[replicationWhitelist]
largeFiles = .../largefile.*

dmlee
Communicator

I follow Stephen's reply , modified distsearch.conf ( replicationBlacklist = .../*.csv ) on search head, it works ! but it only works on 4.1.5 not on 4.1.4.

0 Karma

Stephen_Sorkin
Splunk Employee
Splunk Employee

You need to have .../ at the beginning of your expression since it's an anchored match.

0 Karma

dmlee
Communicator

Hi Stephen Sorkin,
I modified distsearch.conf on search head , I added :
[replicationBlacklist]
donotreplicate = *.csv
and restart search head's splunk service.
I can still see there are many *.csv files under index server's searchpeers directory, it seems like above setting doesn't work correctly

0 Karma

Stephen_Sorkin
Splunk Employee
Splunk Employee

You should look in the $SPLUNK_HOME/var/run/searchpeers directory on the indexer(s) to find the large replicated files.

0 Karma

dmlee
Communicator

Hi Stephon, I upgrade search head and index server to 4.1.5 , but I still see the "WARN" message : "10-19-2010 12:14:53.634 WARN DistributedBundleReplicationManager - bundle replication to 1 peer(s) took too long (35005ms), bundle file size=129810KB".

I use "[replicationBlacklist]" to disable replicate *.csv, and also set enableSplunkdSSL = false . but I still see above "WARN" message, can I know which file is it (bundle file size=129810KB) ?

0 Karma

the_wolverine
Champion

Would it be possible to provide information on how to identify these large assets in apps? Also, it may be a conincidence, but might this be an new issue introduced in version 4.1.5? I don't believe we encountered this issue until we upgraded our search head to 4.1.5 recently. (Indexers are still on 4.1.2).

Thanks, Stephen.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...