Getting Data In

When to use batch input?

dilipvpatel
Explorer

In the documentation of inputs.conf, Splunk does not recommend to use BATCH input method for small archive sizes and recommends instead to use MONITOR input method.

From Splunk's input.conf documentation:
"NOTE: Batch should only be used for large archives of historic data. If you want to continuously monitor a directory or index small archives, use monitor (see above). Batch reads in the file and indexes it, and then deletes the file from the Splunk instance."

What sized file, Splunk considers it to be large enough so that one can use the BATCH INPUT method? I can understand that one cannot have any hard boundary but I would like to know whether files running in 10s of MBs can be considered large enough or not.

What is the technical reason behind Splunk not allowing to use BATCH method for small archives? How small sized files are better for Monitor input. I know that BATCH method purges the input file once it has indexed/uploaded the events but which performance related reasons are involved for discouraing the use of BATCH input method for small archives.

Tags (1)
0 Karma
1 Solution

dart
Splunk Employee
Splunk Employee

You should use it only when you want your source file to be deleted after indexing.

Otherwise it's the same as monitor

View solution in original post

dart
Splunk Employee
Splunk Employee

You should use it only when you want your source file to be deleted after indexing.

Otherwise it's the same as monitor

dilipvpatel
Explorer

I want to use Batch input because Splunk will delete source files for me and I would not need to handle their purging. If my system is generating files as small as 1K files, should I use BATCH INPUT. I do not know what this boundary is for small v/s large archives from Splunk's point of view.

dilipvpatel
Explorer

From Splunk's input.conf documentation: "NOTE: Batch should only be used for large archives of historic data. If you want to continuously monitor a directory or index small archives, use monitor (see above).".....

For small archives, Splunk does not recommend to use batch input. What sized files are considered small archives by Splunk? Small/Large are very subjective terms. What is the Splunk's definition for Small v/s Large archives...

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...