Splunk Search

Most efficent way to ignore last line when monitoring to avoid issues with external log buffering?

JSkier
Communicator

I'm having an issue with a custom application log file (text, xml, single line) where the log buffering done by the custom app is causing an occasional flurry of random splits when ingested by splunk. Basically, the last line is buffered (not completely written to immediately) by the application, for performance, and absolutely may not be changed. So I'd like the forwarder to ignore the very last line, until it is no longer the last line. Basically, tail the log file down to the second to last line in the log file monitored.

I am running splunk enterprise 6.1 (splunk on Linux), and a Windows application server with splunk forwarder.

0 Karma

splunkIT
Splunk Employee
Splunk Employee
0 Karma

JSkier
Communicator

I did notice that, and have been testing it out. There is some improvement but it doesn't fix the problem yet. I'm at time_before_close = 300 presently. I'll try testing it more with higher values and post any findings here.

The buffering is for performance reasons, and not something I could have changed unfortunately.

Was really hoping for an easy solution, there does not appear to be one. Support recommended a daily batch route, which would mean being a day behind with these logs.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...