Getting Data In

Forwarding and Indexing Large Files

rbarajas
Explorer

Anyone have any experience with fowarding and indexing files larger than 200mb every minute?
I'm curious if there are any forwarding processes or indexing processes that need some tuning to keep track of new files being created by our app each minute.

0 Karma

djl
Explorer

Blessed by a good network and robust indexers, the universal forwarder on the endpoint is usually my bottleneck. By closely watching the local splunkd.log file you can see where the forwarder is struggling.

This post ended up being a major help in a recent major log source onboarding experience I had - https://answers.splunk.com/answers/38218/universal-forwarder-parsingqueue-kb-size.html

I also learned the lesson during that onboarding process to uncompress large log files before handing off to the UF for parsing. The UF can handle some on the fly decompression/parsing, but at a certain point it can't keep things straight... We were dealing with compressed text files that were 1-3 million lines long, not a normal type of log situation though.

Best,

David

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi rbarajas,
I experienced this situation, the only way to avoid problems is to have a really good infrastructure: at first a really quick storage (1200 iops), many CPUs on Indexers and a good network, large files indexing could have long indexing queues that cause slow performaces in all the system.
About network, Splunk optimize bandwidth occupation, if you haven't a great constrain you could change the network parameter and send larger packets but I usually don't do this!
As second consequence you could not have a really near time monitoring because there could be a delay in indexing: this is important for real time monitoring and acceletarions.
Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...