Splunk Search

How to configure extraction for multiple timestamps from different message formats going to the same index?

_smp_
Builder

Hi.

I have two message formats falling into the same index. One of these message formats only contains one timestamp without milliseconds, but the second format contains a second timestamp with milliseconds. For reference:

Message format #1 with two timestamps, the second timestamp has milliseconds:

<181>Jan 27 15:15:26 monitor2 CISE_System_Statistics 0000006805 1 0 2017-01-27 15:15:26.036 +00:00 0000036144 70001 NOTICE System-Stats:

Message format #2 with one timestamp:

<180>Jan 27 15:15:32 CISE_Alarm WARN: RADIUS Authentication Request dropped :

I have configured the sourcetype to correctly pull out the second timestamp:

TIME_FORMAT = %Y-%m-%d %H:%M:%S.%3N %:z
TIME_PREFIX = \d{10}\s(?:\d\s){2}
SHOULD_LINEMERGE = true

This correctly processes format #1, but now events with message format #2 are merged into a single event. Is there a way to configure Splunk to pick the second timestamp from format #1, but the first timestamp for format #2?

0 Karma

woodcock
Esteemed Legend

You can do this using a custom datetime.xml file; read about it here:

http://www.function1.com/2013/01/oh-no-splunking-log-files-with-multiple-formats-no-problem

0 Karma

coltwanger
Contributor

Perhaps you can configure the second event to route to a new sourcetype, where you can specify the second time format?

You could have:

[sourcetype:withMS]
 TIME_FORMAT = %Y-%m-%d %H:%M:%S.%3N %:z
 TIME_PREFIX = \d{10}\s(?:\d\s){2}

[sourcetype:withoutMS]
TIME_FORMAT = %b %d %H:%M:%S  #just a guess
TIME_PREFIX = \<\d\d\d\>

Of course you would have to find a pattern in your events to match on in order to route it to a different sourcetype.

http://docs.splunk.com/Documentation/Splunk/6.5.1/Forwarding/Routeandfilterdatad

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi scottprigge,
I don't think that is possible to have two distinct formats for timestamp of one sourcetype, the only way is to use two differente sourcetypes with two different timestamp formats.

If milliseconds of the first type are important for you, you could use the common format for index both the events types and manage time with millisecond of the first type in a dedicated field (not timestamp).

Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...