Getting issue while parsing event which have no timestamp in logs

[ashikuma]

Getting issue while parsing events which have no timestamp in logs, it should use date\time from last log event timestamp but it is not updating, can you please help...

[FrankVl]

Please share sample raw data and relevant Splunk inputs.conf props.conf etc. that you have at the moment.

[ashikuma]

Issue is after timestamp logs , logs without timestamp starting coming into file and below props are taking those event timestamp in splunk as index time (indexer server time) which shouldn't. it should take timestamp from last event which has already timestamp in logs

props.conf

[sourcetype]
BREAK_ONLY_BEFORE = [\w+\s\w+\s\d+\s\d+[:]\d+[:]\d+[.]\d+\s\d+]
TIME_FORMAT = %a %b %d %H:%M:%S.%6 %Y

and sample log files -

[Tue Apr 17 04:11:05.445891 2018] [core:warn] [pid 18022468:tid 1] AH00045: child process 13107268 still did not exit, sending a SIGTERM
[Tue Apr 17 04:11:05.445915 2018] [core:warn] [pid 18022468:tid 1] AH00045: child process 10289284 still did not exit, sending a SIGTERM
[Tue Apr 17 04:11:05.445937 2018] [core:warn] [pid 18022468:tid 1] AH00045: child process 25296994 still did not exit, sending a SIGTERM
Couldn't get http://stores.intranet.********/ITSupport/redesign/is_anything_broken.aspx 404 Not Found at /website/cgi-bin/apps/saletools.cgi line 592.
Couldn't get http://stores.*******/ITSupport/redesign/is_anything_broken.aspx 404 Not Found at /website/cgi-bin/apps/saletools.cgi line 592.
Couldn't get http://stores.intranet.*********/ITSupport/redesign/is_anything_broken.aspx 404 Not Found at /website/cgi-bin/apps/saletools.cgi line 592.

[imthesplunker]

Try this,

[sourcetype]
TIME_PREFIX = ^[
MAX_TIMESTAMP_LOOKAHEAD = 32
TIME_FORMAT = %a %b %d %H:%M:%S.%6N %Y
LINE_BREAKER = ([\r\n]+)[\w+\s\w+\s\d{2}\s\d{2}:\d{2}:\d{2}.\d{6}\s\d{4}]
SHOULD_LINEMERGE = false

[ashikuma]

Hey, I tried this, only issue with this it is breaking event for each line ...
I want - if any event has two lines then it should not break till it ends written automatically
But your regex is breaking each line separately

[cpetterborg]

Do you want each line separated into different events? If you want the non-timestamped lines be part of the last timestamped line, then you should add:

SHOULD_LINEMERGE=true

If that is not what you want, then what is the timestamp you are getting for those non-timestamped lines?

Your TIME_FORMAT should also be:

TIME_FORMAT = %a %b %d %H:%M:%S.%6N %Y

I'd also add:

TIME_PREFIX = ^\[

[ashikuma]

I need all events to be parsed in correct way , if some events are with no timestamp then I want them to use last event which contains timestamp

I tried your suggestion, it is using time from last event (with timestamp) but it should not take all log events which already parsed , it is taking all the events from timestamp startup point and displaying them as well. so it is getting duplicate events

What I need- if any event with no stamp come, then I need them to consolidate under the old event which has timestamp, if this is not possible then I need them to create new event but use timestamp from last event.

[cpetterborg]

This configuration in props.conf works just fine for me.:

[sourcetype]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
BREAK_ONLY_BEFORE=^\[\w+\s\w+\s\d+\s\d+[:]\d+[:]\d+[.]\d+\s\d+\]
TIME_FORMAT=%a %b %d %H:%M:%S.%6N %Y
TIME_PREFIX=^\[
MAX_TIMESTAMP_LOOKAHEAD=32

I did have to make a change to your BREAK_ONLY_BEFORE to escape the square brackets and I added the beginning of line anchor. This will result in the following three events from your sample data:

1       4/17/18
4:11:05.445 AM  
[Tue Apr 17 04:11:05.445891 2018] [core:warn] [pid 18022468:tid 1] AH00045: child process 13107268 still did not exit, sending a SIGTERM
2       4/17/18
4:11:05.445 AM  
[Tue Apr 17 04:11:05.445915 2018] [core:warn] [pid 18022468:tid 1] AH00045: child process 10289284 still did not exit, sending a SIGTERM
3       4/17/18
4:11:05.445 AM  
[Tue Apr 17 04:11:05.445937 2018] [core:warn] [pid 18022468:tid 1] AH00045: child process 25296994 still did not exit, sending a SIGTERM
Couldn't get http://stores.intranet./ITSupport/redesign/is_anything_broken.aspx 404 Not Found at /website/cgi-bin/apps/saletools.cgi line 592.
Couldn't get http://stores./ITSupport/redesign/is_anything_broken.aspx 404 Not Found at /website/cgi-bin/apps/saletools.cgi line 592.
Couldn't get http://stores.intranet.**/ITSupport/redesign/is_anything_broken.aspx 404 Not Found at /website/cgi-bin/apps/saletools.cgi line 592.

[ashikuma]

that is fine, but issue started after that when new log comes in without timestamp then it should take timestamp from previous event and should look like below:-

4/17/18 4:11:05.445 AM Couldn't get http://stores.intranet./ITSupport/redesign/is_anything_broken.aspx 404 Not Found at /website/cgi-bin/apps/saletools.cgi line 592.

But it is coming like below ,it is taking timestamp from last stored event (that is good) , but along with that it is taking old timestamped event data also and displaying that as new event which is making duplicate entries:

[Tue Apr 17 04:11:05.445937 2018] [core:warn] [pid 18022468:tid 1] AH00045: child process 25296994 still did not exit, sending a SIGTERM
Couldn't get http://stores.intranet./ITSupport/redesign/is_anything_broken.aspx 404 Not Found at /website/cgi-bin/apps/saletools.cgi line 592.
Couldn't get http://stores./ITSupport/redesign/is_anything_broken.aspx 404 Not Found at /website/cgi-bin/apps/saletools.cgi line 592.

[ashikuma]

that is fine, issue starts after that , if next single event will come without timestamp then it should be like below :-

4/17/18
4:11:05.445 AM -Couldn't get http://stores.intranet./ITSupport/redesign/is_anything_broken.aspx 404 Not Found at /website/cgi-bin/apps/saletools.cgi line 592.

but in my case it is coming into next event but with old event data also, like below... it should not contain old data with itself

[Tue Apr 17 04:11:05.445937 2018] [core:warn] [pid 18022468:tid 1] AH00045: child process 25296994 still did not exit, sending a SIGTERM
Couldn't get http://stores.intranet./ITSupport/redesign/is_anything_broken.aspx 404 Not Found at /website/cgi-bin/apps/saletools.cgi line 592.
Couldn't get http://stores./ITSupport/redesign/is_anything_broken.aspx 404 Not Found at /website/cgi-bin/apps/saletools.cgi line 592.
Couldn't get http://stores.intranet.**/ITSupport/redesign/is_anything_broken.aspx 404 Not Found at /website/cgi-bin/apps/saletools.cgi line 592