Getting Data In

Is there a way to tell Splunk how long to wait for the beginning of the next event?

a212830
Champion

Hi,

I have a multi-line feed that appears to be having issues when the "next event" is delayed. Each event starts with a timestamp, and we have the line_breaker configured to break on those lines. It appears that when the feed gets slow, and additional lines are added to the existing event, Splunk is turning them into new events. (Hope this makes sense). Is there a way to tell Splunk how long to wait for the beginning of the next event?

For example: "AP" lines below took their time coming in , they then appeared as separate events. Other multi-line events that use the same format, but that aren't slow, worked fine.

20:21:56.143 [IDX]: >> GET >> FMID=01MSGO2AM0A9F7MAJJ45U2LAES001E29 NOT FOUND
20:21:56.143 [ORSCallMonitor] OnPartyDeleted
20:21:56.143 [IDX]: >> GET >> FMID=01MSGO2AM0A9F7MAJJ45U2LAES001E29 NOT FOUND
20:21:56.143 [ORSCallMonitor] OnCallDeleted
20:21:56.143 [IDX]: >> GET >> FMID=01MSGO2AM0A9F7MAJJ45U2LAES001E29 NOT FOUND
-AP[94739]->-16 @20:21:56.8953
-Ap[94739]-<-16 @20:21:56.8958
-AP[98780]->-4 @20:22:01.2976
-Ap[98780]-<-4 @20:22:01.3170
20:22:03.332 <<<=== 'EventAgentNotReady'(76) seq=aa73d6

Our props.conf:

ANNOTATE_PUNCT = false
FIELD_HEADER_REGEX = ^File:
KV_MODE = auto
LINE_BREAKER = ([\r\n]+)\d{2}:\d{2}:\d{2}.\d{3}
MAX_TIMESTAMP_LOOKAHEAD = 25
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
TIME_FORMAT = %H:%M:%S.%3N
TIME_PREFIX = ^
TRUNCATE = 999999

0 Karma

tom_frotscher
Builder

Hi,

looks like your line break pattern somehow also matches your AP lines. For me it works with this LINE_BREAKER:

^\d{2}:\d{2}:\d{2}\.\d{3}

alt text

0 Karma

a212830
Champion

Thanks. It worked for me (as did the other one) as well, but in prod it's not working. I actually tailed the file, and in once scenario, it took 15 seconds for the next "AP" line to appear (despite the timestamp on that line). Splunk is making these separate events. I'm guessing that the time between lines on the multi-event is causing the issue.

0 Karma

a212830
Champion

Anyone? I've looked through the doc, but nothing stands out.

0 Karma

Ayn
Legend

If this is a file monitor input, have a look at the "time_before_close" directive in inputs.conf. http://docs.splunk.com/Documentation/Splunk/6.2.1/admin/inputsconf

0 Karma

a212830
Champion

Is that for waiting for the next event, for for determining that the file is closed? My issue is the delay between lines on a multi-line event.

0 Karma

a212830
Champion

Anyone?

0 Karma

a212830
Champion

Any idea on this?

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Per definition, it says "time_before_close" is the time in second that Splunk will wait before considering data saved in file is completed. So, As @Ayn suggested, set this value to 1800 (30 mins) OR 1200 (20 min) based on your requirement, then Splunk will not start reading the data unless the last modification time for a version is atleast that seconds old. There will be delay though with this approach in getting your data to Splunk.

0 Karma

a212830
Champion

I think that we are talking two different things. The issue isn't that the file has closed or rolled-over - it's all in the same file. It goes like this (examples above):

First line of multi-line vent comes in:
Next line of multi-line event comes in - some seconds after the above line
Additional line of multi-line event comes in - some seconds after previous line

The first line is it's own event, and the next two are combined as a single multi-line event. If I test this input with my props, it works fine, so I think it's timing related.

0 Karma

a212830
Champion

Bump......

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...