Multi line NFS monitoring file cause multiple events to be created

[jdagenais]

Hello,

We are monitoring application files that are mounted as read-only NFS drives, and sometimes multi-lines messages are processed as multiple events.

These are the format of the multi lines messages, when processes as single events.

Event #1
[LOG|INFO|2010 December 07, 09:59:43 (208)|TRACK_MD|RMI TCP Connection(61072)-100.200.10.22|100.200.10.21 (host.net.com)]
Saving MD:  DataInfo.1891568 LAST 12/7/10 10:59:31.000 AM EST       FROM:   100.200.10.20 (host.net.com)    @ 12/7/10 9:59:43.208 AM EST
[END]

Event #2
[LOG|ERROR|2010 December 07, 09:59:55 (8)|SQL|RMI TCP Connection(61294)-100.200.10.21|100.200.10.21 (host.net.com)]
Trying to saves duplicate points in data filed 1287/OBRC on 12/7/10 9:59:48.000 AM EST
[END]

This is the format, when they are processes as multiple events

Event #1
[LOG|INFO|2010 December 07, 09:59:43 (208)|TRACK_MD|RMI TCP Connection(61072)-100.200.10.22|100.200.10.21 (host.net.com)]

Event #2
Saving MD:  DataInfo.1891568 LAST 12/7/10 10:59:31.000 AM EST       FROM:   100.200.10.20 (host.net.com)    @ 12/7/10 9:59:43.208 AM EST
    [END]

Is there any ways to resolve this multi line nfs issue?

We tried different options such as:

BREAK_ONLY_BEFORE=\[LOG\|

MAX_TIMESTAMP_LOOKAHEAD=50

Thanks, Jean

[jdagenais]

Thanks for ths suggestion!

The problem is related to using NFS with high latency, and applications taking a long time (e.g. seconds) to write a complete message.

For example, the application can start to write the first part, and then 5 seconds later write the second part which cause the full messages to be written.

I found this post which explain a solution that seems to work quite well:

http://answers.splunk.com/questions/9750/multline-events-with-pauses-between-lines

This is the description of the change:

time_before_close = * Modtime delta required before Splunk can close a file on EOF. * Tells the system not to close files that have been updated in past seconds. * Defaults to 3.

For my test environment, I am using a value of time_before_close = 60

Thanks, Jean

[ziegfried]

You could try to use a LINE_BREAKER instead of line merging:

[<your sourcetype>]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)\[LOG\|
MAX_TIMESTAMP_LOOKAHEAD = 50

or

[<your sourcetype>]
SHOULD_LINEMERGE = false
LINE_BREAKER = \[END\](\s+)
MAX_TIMESTAMP_LOOKAHEAD = 50