Getting Data In

Events in a single file are not getting evenly distributed to multiple indexers

ishugupta
Path Finder

I have a light weight forwarder pointing two indexers . I get a batch data everyday in a single file . The file size is 28 GB on an average .
All the events in that file gets ingested in one single indexer . I noticed splunk is not able to distribute the events within a single file . It distributes events to indexers based on file level .So in case I get 10- 11 file on some day , one single file gets ingested to one single indexer and other file on other indexer. Is there any solution that can help me distribute the data on event level.
I am using Splunk 6.1.7

0 Karma

Runals
Motivator

I suspect what you are running into is a forwarder will send data to an indexer for 30s OR until it hits an end of file and then switch. If these are single files as part of a batch job I'm not surprised you are seeing the behavior you are describing. I'm not sure there is a good way to adjust that. You could check out the forceTimebasedAutoLB setting but I'm not sure this is the best use case for that setting.

ishugupta
Path Finder

thanks Runals , Seems like there is not concrete solution to this . I will have to break my files to multiple smaller files .
I do not want to change the forceTimebasedAutoLB setting , as it might truncate an event itself.

0 Karma

satishsdange
Builder

LWF is deprecated. Please replace it by UF.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...