How to configure extraction for multiple timestamps from different message formats going to the same index?

[_smp_]

Hi.

I have two message formats falling into the same index. One of these message formats only contains one timestamp without milliseconds, but the second format contains a second timestamp with milliseconds. For reference:

Message format #1 with two timestamps, the second timestamp has milliseconds:

<181>Jan 27 15:15:26 monitor2 CISE_System_Statistics 0000006805 1 0 2017-01-27 15:15:26.036 +00:00 0000036144 70001 NOTICE System-Stats:

Message format #2 with one timestamp:

<180>Jan 27 15:15:32 CISE_Alarm WARN: RADIUS Authentication Request dropped :

I have configured the sourcetype to correctly pull out the second timestamp:

TIME_FORMAT = %Y-%m-%d %H:%M:%S.%3N %:z
TIME_PREFIX = \d{10}\s(?:\d\s){2}
SHOULD_LINEMERGE = true

This correctly processes format #1, but now events with message format #2 are merged into a single event. Is there a way to configure Splunk to pick the second timestamp from format #1, but the first timestamp for format #2?

[woodcock]

You can do this using a custom datetime.xml file; read about it here:

http://www.function1.com/2013/01/oh-no-splunking-log-files-with-multiple-formats-no-problem

[coltwanger]

Perhaps you can configure the second event to route to a new sourcetype, where you can specify the second time format?

You could have:

[sourcetype:withMS]
 TIME_FORMAT = %Y-%m-%d %H:%M:%S.%3N %:z
 TIME_PREFIX = \d{10}\s(?:\d\s){2}

[sourcetype:withoutMS]
TIME_FORMAT = %b %d %H:%M:%S  #just a guess
TIME_PREFIX = \<\d\d\d\>

Of course you would have to find a pattern in your events to match on in order to route it to a different sourcetype.

http://docs.splunk.com/Documentation/Splunk/6.5.1/Forwarding/Routeandfilterdatad

[gcusello]

Hi scottprigge,
I don't think that is possible to have two distinct formats for timestamp of one sourcetype, the only way is to use two differente sourcetypes with two different timestamp formats.

If milliseconds of the first type are important for you, you could use the common format for index both the events types and manage time with millisecond of the first type in a dedicated field (not timestamp).

Bye.
Giuseppe