Getting Data In

After forwarder network goes down and is restored, why does only one indexer receive lost data?

echonest_krystl
New Member

Hi,

I have data cloning to 2 splunk indexers (instances):

                   forwarder1
                  /          \
             Splunk01     Splunk02

when the network goes out on the forwarder1, splunk01 and splunk02 don't receive data. Which is expected. The problem is when, once the network is restored, splunk01 gets the lost data, but splunk02 does not get the data that was lost.

my forwarder outputs.conf is (server names have been changed to make this easier to understand):
[tcpout]
defaultGroup = firstsplunkserver,secondsplunkserver

[tcpout:firstsplunkserver]
server = splunk01:9997

[tcpout:secondsplunkserver]
server = splunk02:9997

Why isn't splunk02 getting the lost data? How do you clone this data from splunk01?

Thanks!

0 Karma

hortonew
Builder

I can't find anything that goes along with this issue. Have you run wireshark/tcpdump on splunk02 to view packets coming in, or on your forwarder to view packets going out, and verify that nothing is getting destined to splunk02? Or, are you just searching the data and not seeing it? Reason I ask is to determine which side of the connection is becoming a problem.

I would also search the forwarder and splunk02's splunkd.log to see if anything comes up during that time period indicating one side or the other.

I'll keep looking, but the way you're listing these servers in the defaultGroup should always clone the data to anything there.

0 Karma

hortonew
Builder

Not that I know of. Did you try reversing the order so 02 is first in the list? See if the data goes to it and not 01, or if 01 is the only one capable of receiving this data? That would be the last test I would try to pin point the actual issue.

0 Karma

echonest_krystl
New Member

I'm searching the data and not seeing it. On the splunk forwarder it just says it disconnects and reconnects to that server.

is there anything i need to enable on the splunk02 instance?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...