Getting Data In

Multi line NFS monitoring file cause multiple events to be created

jdagenais
Explorer

Hello,

We are monitoring application files that are mounted as read-only NFS drives, and sometimes multi-lines messages are processed as multiple events.

These are the format of the multi lines messages, when processes as single events.

Event #1
[LOG|INFO|2010 December 07, 09:59:43 (208)|TRACK_MD|RMI TCP Connection(61072)-100.200.10.22|100.200.10.21 (host.net.com)]
Saving MD:  DataInfo.1891568 LAST 12/7/10 10:59:31.000 AM EST       FROM:   100.200.10.20 (host.net.com)    @ 12/7/10 9:59:43.208 AM EST
[END]

Event #2
[LOG|ERROR|2010 December 07, 09:59:55 (8)|SQL|RMI TCP Connection(61294)-100.200.10.21|100.200.10.21 (host.net.com)]
Trying to saves duplicate points in data filed 1287/OBRC on 12/7/10 9:59:48.000 AM EST
[END]

This is the format, when they are processes as multiple events

Event #1
[LOG|INFO|2010 December 07, 09:59:43 (208)|TRACK_MD|RMI TCP Connection(61072)-100.200.10.22|100.200.10.21 (host.net.com)]

Event #2
Saving MD:  DataInfo.1891568 LAST 12/7/10 10:59:31.000 AM EST       FROM:   100.200.10.20 (host.net.com)    @ 12/7/10 9:59:43.208 AM EST
    [END]

Is there any ways to resolve this multi line nfs issue?

We tried different options such as:

BREAK_ONLY_BEFORE=\[LOG\|

MAX_TIMESTAMP_LOOKAHEAD=50

Thanks, Jean

jdagenais
Explorer

Thanks for ths suggestion!

The problem is related to using NFS with high latency, and applications taking a long time (e.g. seconds) to write a complete message.

For example, the application can start to write the first part, and then 5 seconds later write the second part which cause the full messages to be written.

I found this post which explain a solution that seems to work quite well:

http://answers.splunk.com/questions/9750/multline-events-with-pauses-between-lines

This is the description of the change:

time_before_close = * Modtime delta required before Splunk can close a file on EOF. * Tells the system not to close files that have been updated in past seconds. * Defaults to 3.

For my test environment, I am using a value of time_before_close = 60

Thanks, Jean

ziegfried
Influencer

You could try to use a LINE_BREAKER instead of line merging:

[<your sourcetype>]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)\[LOG\|
MAX_TIMESTAMP_LOOKAHEAD = 50

or

[<your sourcetype>]
SHOULD_LINEMERGE = false
LINE_BREAKER = \[END\](\s+)
MAX_TIMESTAMP_LOOKAHEAD = 50
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...