Splunk Search

Indexing queues greater than 100 lines

jrodriguezap
Contributor

Hi all, I would like to set the transforms.conf started indexing log files when they exceed the 100 lines per second as a condition in the Regex, I believe that it is possible to Splunk, any ideas, suggestions or reference to a link to achieve?

Thanks in advance for your attention.
regards

0 Karma

jrodriguezap
Contributor

Oh, Sorry.
I could not find something about it to bring to my own question.
I'm sorry.

0 Karma

yannK
Splunk Employee
Splunk Employee

There is no such method to wait for number of lines before indexing ,Splunk will start indexing the file as soon as they are available.

What is the problem with your files, are the events cut in multiple events :

A - are your events longer than 250 lines per events.
Splunk has a default limit for the multiline events, you can configure your sourcetype in props,conf (at the indexer/heavy forwarder level) with MAX_EVENTS=

B - Is your application using a write buffer and writing slowly in chunk (cutting events in the middle of the line ? )
you can use the setting time_before_close in inputs.conf on the forwarders for this monitor, to force Splunk to wait longer before detecting the EOF.

see http://docs.splunk.com/Documentation/Splunk/5.0.4/admin/Inputsconf
time_before_close =
* Modtime delta required before Splunk can close a file on EOF.
* Tells the system not to close files that have been updated in past seconds.
* Defaults to 3.

0 Karma

rsennett_splunk
Splunk Employee
Splunk Employee

Señor Rodríguez:

I have deleted our struggles from last night and I am sending you an email. I will connect you with my colleague who covers Latin America.

I believe do understand what you need now, and I have a suggestion - but still am unclear about some of your environment and feel it is the translator that is not helping us.

When you have the solution, please post it here as the answer and I will up-vote so that others may benefit.

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!
0 Karma
Get Updates on the Splunk Community!

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Wednesday, May 29, 2024  |  11AM PST / 2PM ESTRegister now and join us to learn more about how you can ...

Get Your Exclusive Splunk Certified Cybersecurity Defense Engineer at Splunk .conf24 ...

We’re excited to announce a new Splunk certification exam being released at .conf24! If you’re headed to Vegas ...

Share Your Ideas & Meet the Lantern team at .Conf! Plus All of This Month’s New ...

Splunk Lantern is Splunk’s customer success center that provides advice from Splunk experts on valuable data ...