Events from my universal forwarder are getting random linefeeds inserted into the events. Why is this happening and how do I fix it?

[DerekB]

I have a universal forwarder monitoring a log file. The UF is passing this data to my indexer but when the data shows up in my search, it appears to have strange line breaks. I don't have any line breaks configuring in my props.conf for this sourcetype on the indexer. The source log file doesn't show the data this way. I even used notepad and had it show me the hidden characters in the file and there isn't anything showing up in the spots where the event is breaking. Each line of my data starts with a timestamp and that should be where the natural line break happens.

This is happening in a Windows environment.

This is what the event looks like in the source file (It's a single line in the file.)

12:34:54.000 GUID> RED5678< [SERVERNAME] GENERATE_SOMETHING {FJDKF3423423JHSD} _SRC=KLAATO-1234-5678:_CMD=GENERATE_SOMETHING;CALL_ID=INSIDETHEHOUSE;SOMETHINGString=88;

But when it shows up in my search it is broken into 2 different events.

16:11:36.263 RFD> 38FSDKF< [SERVERNAME] GENERATE_SOMETHING {FJDKF3

423423JHSD} _SRC=KLAATO-1234-5678:_CMD=GENERATE_SOMETHING;CALL_ID=INSIDETHEHOUSE;SOMETHINGString=88;

Why is this happening and how do I fix it?

[hexx · Accepted solution]

In all likelihood, this is happening because the log writer is buffering content before appending it to the file you are monitoring and occasionally pausing mid-event for longer than the default 'time_before_close' duration (3s), as defined in inputs.conf.spec:

time_before_close = <integer>
* Modtime delta required before Splunk can close a file on EOF.
* Tells the system not to close files that have been updated in past <integer> seconds.
* Defaults to 3.

The solution here is to adjust 'time_before_close' (on a per-stanza basis in inputs.conf) so that Splunk's tailing processor will only close a given file after the longest pause that the log writer can take.

For example, if you know that the log writer can pause for up to 5 seconds mid-event, adjust 'time_before_close' to a value that can comfortably accomodate that, say 10.

[aholzer]

You could add this to your props.conf to ensure that it only breaks at the start of lines that contain 2 digits followed by a colon:

BREAK_ONLY_BEFORE=^\d{2}:

Why it's happening is a tricky question. I'm assuming that it's happening because it believes that half way through your event, where there are a lot of numbers, that it's a new date, and therefore should start a new event. If this is the case, then my change should take care of the problem.

If that doesn't fix the problem, then we'll have to test further parameter definitions in the props.conf.