Splunk Search

Most efficent way to ignore last line when monitoring to avoid issues with external log buffering?

JSkier
Communicator

I'm having an issue with a custom application log file (text, xml, single line) where the log buffering done by the custom app is causing an occasional flurry of random splits when ingested by splunk. Basically, the last line is buffered (not completely written to immediately) by the application, for performance, and absolutely may not be changed. So I'd like the forwarder to ignore the very last line, until it is no longer the last line. Basically, tail the log file down to the second to last line in the log file monitored.

I am running splunk enterprise 6.1 (splunk on Linux), and a Windows application server with splunk forwarder.

0 Karma

splunkIT
Splunk Employee
Splunk Employee
0 Karma

JSkier
Communicator

I did notice that, and have been testing it out. There is some improvement but it doesn't fix the problem yet. I'm at time_before_close = 300 presently. I'll try testing it more with higher values and post any findings here.

The buffering is for performance reasons, and not something I could have changed unfortunately.

Was really hoping for an easy solution, there does not appear to be one. Support recommended a daily batch route, which would mean being a day behind with these logs.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...