Deployment Architecture

Too many streaming errors to target on cluster

responsys_cm
Builder

I've got a cluster with three identical indexers. One indexer consistently generates the "Too many streaming errors to target". I've checked the network adapter on the server and there are zero errors over 4 TB of traffic. I've tried increasing the error count from 3 to 6, but it still happens.

How do I figure out the root cause of this issue?

Thanks.

Craig

Tags (1)

svasan_splunk
Splunk Employee
Splunk Employee

Were there any replication/tcp related errors in the target node's splunkd.log?

0 Karma

svasan_splunk
Splunk Employee
Splunk Employee

That could be the cause of your problem. Though the BucketReplicator errors on the originating node not on the target of the replication. When it says "Too many streaming errors" that's anything that causes the replication to fail. Do you see why the writes fail? Did they just timeout?

Also when you say you checked the network adapter, what did you check?

responsys_cm
Builder

WARN BucketReplicator - Failed to replicate warm bucket _internal~323~3B990028-D1F8-40B1-B39C-DD13D01D8FAF to E490E3C8-F237-490F-A245-082AE868B4F8. Replication to 10.230.226.21:8090:_internal~323~3B990028-D1F8-40B1-B39C-DD13D01D8FAF failed. Write failed.

ERROR BucketReplicator - Replication to 10.230.226.21:8090:_internal~323~3B990028-D1F8-40B1-B39C-DD13D01D8FAF failed. Write failed.

0 Karma

responsys_cm
Builder

These are the three error messages I see a ton of:

CMSlave - bid=_internal~323~3B990028-D1F8-40B1-B39C-DD13D01D8FAF src=3B990028-D1F8-40B1-B39C-DD13D01D8FAF tgt=E490E3C8-F237-490F-A245-082AE868B4F8 failing=E490E3C8-F237-490F-A245-082AE868B4F8 queued replication error job

0 Karma
Get Updates on the Splunk Community!

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Wednesday, May 29, 2024  |  11AM PST / 2PM ESTRegister now and join us to learn more about how you can ...

Get Your Exclusive Splunk Certified Cybersecurity Defense Engineer Certification at ...

We’re excited to announce a new Splunk certification exam being released at .conf24! If you’re headed to Vegas ...

Share Your Ideas & Meet the Lantern team at .Conf! Plus All of This Month’s New ...

Splunk Lantern is Splunk’s customer success center that provides advice from Splunk experts on valuable data ...