Hi,
I have a multi-line feed that appears to be having issues when the "next event" is delayed. Each event starts with a timestamp, and we have the line_breaker configured to break on those lines. It appears that when the feed gets slow, and additional lines are added to the existing event, Splunk is turning them into new events. (Hope this makes sense). Is there a way to tell Splunk how long to wait for the beginning of the next event?
For example: "AP" lines below took their time coming in , they then appeared as separate events. Other multi-line events that use the same format, but that aren't slow, worked fine.
20:21:56.143 [IDX]: >> GET >> FMID=01MSGO2AM0A9F7MAJJ45U2LAES001E29 NOT FOUND
20:21:56.143 [ORSCallMonitor] OnPartyDeleted
20:21:56.143 [IDX]: >> GET >> FMID=01MSGO2AM0A9F7MAJJ45U2LAES001E29 NOT FOUND
20:21:56.143 [ORSCallMonitor] OnCallDeleted
20:21:56.143 [IDX]: >> GET >> FMID=01MSGO2AM0A9F7MAJJ45U2LAES001E29 NOT FOUND
-AP[94739]->-16 @20:21:56.8953
-Ap[94739]-<-16 @20:21:56.8958
-AP[98780]->-4 @20:22:01.2976
-Ap[98780]-<-4 @20:22:01.3170
20:22:03.332 <<<=== 'EventAgentNotReady'(76) seq=aa73d6
Our props.conf:
ANNOTATE_PUNCT = false
FIELD_HEADER_REGEX = ^File:
KV_MODE = auto
LINE_BREAKER = ([\r\n]+)\d{2}:\d{2}:\d{2}.\d{3}
MAX_TIMESTAMP_LOOKAHEAD = 25
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
TIME_FORMAT = %H:%M:%S.%3N
TIME_PREFIX = ^
TRUNCATE = 999999
Thanks. It worked for me (as did the other one) as well, but in prod it's not working. I actually tailed the file, and in once scenario, it took 15 seconds for the next "AP" line to appear (despite the timestamp on that line). Splunk is making these separate events. I'm guessing that the time between lines on the multi-event is causing the issue.
Anyone? I've looked through the doc, but nothing stands out.
If this is a file monitor input, have a look at the "time_before_close" directive in inputs.conf. http://docs.splunk.com/Documentation/Splunk/6.2.1/admin/inputsconf
Is that for waiting for the next event, for for determining that the file is closed? My issue is the delay between lines on a multi-line event.
Anyone?
Any idea on this?
Per definition, it says "time_before_close" is the time in second that Splunk will wait before considering data saved in file is completed. So, As @Ayn suggested, set this value to 1800 (30 mins) OR 1200 (20 min) based on your requirement, then Splunk will not start reading the data unless the last modification time for a version is atleast that seconds old. There will be delay though with this approach in getting your data to Splunk.
I think that we are talking two different things. The issue isn't that the file has closed or rolled-over - it's all in the same file. It goes like this (examples above):
First line of multi-line vent comes in:
Next line of multi-line event comes in - some seconds after the above line
Additional line of multi-line event comes in - some seconds after previous line
The first line is it's own event, and the next two are combined as a single multi-line event. If I test this input with my props, it works fine, so I think it's timing related.
Bump......