Deployment Architecture

ERROR DistributedBundleReplicationManager - got non-200 response from peer.

jerinabeham
Explorer

Hi,

Currently, i have upgraded splunk from 6.0.4 to 6.1.1 in our test box.
Till then, i am able too the follwoig error log in splunkd.log

ERROR DistributedBundleReplicationManager - got non-200 response from peer.uri=****,
reply="HTTP/1.1 400 Bad Request" response_code=400

Could someone help to clarify and resolve the above?

Thanks
Jerina

Tags (1)

yannK
Splunk Employee
Splunk Employee

This happens when the search-head is pushing a search bundle that is too large to the indexers.

The default bundle max size (maxBundleSize) is 1GB
and the default http packet size (max_content_length) accepted by splunkd is 800MB 😞

Therefore :

  • when 1024MB> bundle >800MB see an http error from the indexers. "failed_because_BUNDLE_DATA_TRANSMIT_FAILURE" or "ERROR DistributedBundleReplicationManager - got non-200 response from peer"
  • when the bundle is >1024MB we see a different error, from the search-head.

Workarounds :

  • RECOMMENDED :reduce the bundle size (trim your lookups, use blacklists in distsearch.conf)
  • LESS RECOMMENDED : allow larger bundles

example : to bump the bundle size to 2GB max
on Indexers , edit server.conf (push from cluster master etc/master-apps in a cluster)

[httpServer]
max_content_length = 2147483648 
# in bytes => 2GBdistsearch.conf 

on Search-head

[replicationSettings] 
maxBundleSize= 2097152 
# in MB => 2GB

bkahlerventer
Explorer

I got these on old hardware when I upgraded to 6.1.3. It appears to be a timing issue and storage speed appears to play a role. Take a look at this thread.

http://answers.splunk.com/answers/12666/42-search-head-asynchronous-bundle-replication-error

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...