Deployment Architecture

Splunk 5.0.1 Clustered Indexes and Duplicate Data

dturner83
Path Finder

I have the following Splunk build below.

I have a replication factor of 3 and search factor of 2.
Just using 1 search head at the moment, splunksearch1, which is the master node. It distributes appropriately to splunkindex1, 2, and 3 but I get duplicate data back.

So I have a forwarder there at the bottom, it forwards data to splunkforward1 and splunkforward2, which in turn send to splunkindex1-3. When searching I get the results from all 3 with the same timestamp and exact same data so I'm assuming it's returning all the data. According to the documentation Clustering is supposed to only return the primary data, but I'm unsure how to check/troubleshoot farther than that.

Anyone got any ideas?

Splunk Environment

Update: Instead of having both forwarders forward to all 3 indexers I made them point at just 1. This has fixed the issue of seeing the data duplicated through the searches. But this seems less than ideal. If the indexer which is receiving the data goes down a change needs to be made to change the destination indexer.

Tags (2)
0 Karma

dturner83
Path Finder

I modified both heavy forwarders configs to this:
[tcpout:autolbgroup1]
server = 192.168.101.22:9997,192.168.101.23:9997,192.168.101.33:9997
autoLB = true
useACK = true

[tcpout]
defaultGroup = autolbgroup1
disabled = 0

the key appears to be autoLB = true. I previously understood that it was always true but didn't appear so. Anyway setting this to true fixes the entire problem. I'm assuming it was sending all indexers all copies of the data and they all thought they were new primary copies and then returning those results. Now it is all working properly.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...