Getting Data In

Observing consistent uneven load balancing from universal forwarders to indexers

Runals
Motivator

I've come up with a query to see which indexers our forwarders are sending data to and the results are somewhat eye opening. I'd be interested in other sets of eyes on the logic of my query and to see if others are experiencing the same. The idea is to use the internal logs from the forwarders keying in on the phrase "connected to idx". The query is

index=_internal "connected to idx" | rex field=idx "(?<indexer>[^:]+)" | stats count by indexer | sort indexer

I'm sorting the indexers in numeric order (IP) because that is the way they are listed in the outputs.conf. What is interesting is the visual profile of the distribution is basically the same regardless of the time of day or time span of the query. In other words my first indexer receives about 2x the connections as the others whether that is in the last 5 minutes or 7 days. The same visual profile regardless of time/time range is generally true for the others. I've come up with a query to match the results of this data to the GB indexed per indexer in the same time period but it needs to be run in the deployment monitor app - wasn't going to rewrite all the macros. I also had to do some heavy massaging in order to match IP to indexer name and then to sort correctly. You might not have that issue /shrug. Run it in its base form and see (I've left out post append stuffs)

index=_internal "connected to idx" | rex field=idx "(?<indexer>[^:]+)" | eval "Splunk Server" = indexer | stats count by "Splunk Server" | append [search `all_indexers_summary`]  | sort "Splunk Server" 

Environment

20 indexers

1300 forwarders

Using auto load balancing built into outputs.conf

Forwarder versions range from 4.3.4 to 6.0.1

Tags (1)
0 Karma

MuS
SplunkTrust
SplunkTrust

Hi Runals,

see this answer from @lguinn, maybe this helps ....

cheers, MuS

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...