Getting Data In

Timestamp Problem

liviab
Explorer

Hi,

I'm using Splunk to index logs which timestamp is in the format Y2010M09D17H10N07S00. As Splunk couldn't understand it alone, I tried three differents ways to configure it:

1) using TIME_PREFIX = \s+Y and MAX_TIMESTAMP_LOOKAHEAD=20 in props.conf - FAILED!

2) using TIME_FORMAT = Y%YM%mD%dH%HN%MS%S in props.conf - FAILED!

3) using DATETIME_CONFIG = /etc/system/local/datetime.xml in props.conf and datetime.xml as follows (only defines is showed):

<define name="softxtimestamp_date" extract="year,month,day">
    <text><![CDATA[[\s\S]+Y(\d{4})M(\d{2})D(\d{2})[\s\S]+]]></text>
</define>

<define name="softxtimestamp_time" extract="hour,minute,second">
    <text><![CDATA[[\s\S]+H(\d{2})N(\d{2})S(\d{2})[\s\S]+]]></text>
</define>

The last try (using date) partially works and I can't undestand why it works to some lines and not for others. 😕 Following it is showed some examples of what Splunk understood for each timestamp:

6/4/10 11:55:46.000 PM <- Y2010M06D04H23N55S50 NOK (sec)
6/4/10 11:55:46.000 PM <- Y2010M06D04H23N55S50 NOK (sec)
6/4/10 11:55:46.000 PM <- Y2010M06D04H23N55S49 NOK (sec)
6/4/10 11:55:46.000 PM <- Y2010M06D04H23N55S49 NOK (sec)
6/4/10 11:55:46.000 PM <- Y2010M06D04H23N55S49 NOK (sec)
6/4/10 11:55:46.000 PM <- Y2010M06D04H23N55S48 NOK (sec)
6/4/10 11:55:46.000 PM <- Y2010M06D04H23N55S48 NOK (sec)
6/4/10 11:55:46.000 PM <- Y2010M06D04H23N55S48 NOK (sec)
6/4/10 11:55:46.000 PM <- Y2010M06D04H23N55S47 NOK (sec)
6/4/10 11:55:46.000 PM <- Y2010M06D04H23N55S47 NOK (sec)
6/4/10 11:55:46.000 PM <- Y2010M06D04H23N55S46 OK
6/4/10 2:00:01.000 PM <- Y2010M06D04H15N58S07 NOK (hour, min, sec)
6/4/10 2:00:01.000 PM <- Y2010M06D04H15N56S52 NOK (hour, min, sec)
6/4/10 10:31:48.000 AM <- Y2010M06D04H10N31S48 OK
6/4/10 10:31:47.000 AM <- Y2010M06D04H10N31S47 OK
6/4/10 10:31:46.000 AM <- Y2010M06D04H10N31S46 OK
6/4/10 10:31:46.000 AM <- Y2010M06D04H10N31S46 OK
6/4/10 10:31:46.000 AM <- Y2010M06D04H10N31S46 OK
6/4/10 10:31:46.000 AM <- Y2010M06D04H10N31S46 OK
6/4/10 10:31:45.000 AM <- Y2010M06D04H10N31S45 OK
6/4/10 10:30:48.000 AM <- Y2010M06D04H10N30S48 OK
6/4/10 10:30:47.000 AM <- Y2010M06D04H10N30S47 OK

Can anyone help me? Please!

1 Solution

Stephen_Sorkin
Splunk Employee
Splunk Employee

The following configuration should work in props.conf:

[<sourcetype>]
TIME_PREFIX = \s+Y(?=\d{4})
TIME_FORMAT = %YM%mD%dH%HN%MS%S

Using just (1), you will find the right location to look for the timestamp, but none of our default rules will apply. Using just (2), strptime parsing will try to match that pattern at the beginning of the line.

View solution in original post

liviab
Explorer

"[\s\S]+" is to match with any character (space or not) before and after the timestamp field, since there are more fields in the line

0 Karma

Stephen_Sorkin
Splunk Employee
Splunk Employee

The following configuration should work in props.conf:

[<sourcetype>]
TIME_PREFIX = \s+Y(?=\d{4})
TIME_FORMAT = %YM%mD%dH%HN%MS%S

Using just (1), you will find the right location to look for the timestamp, but none of our default rules will apply. Using just (2), strptime parsing will try to match that pattern at the beginning of the line.

Stephen_Sorkin
Splunk Employee
Splunk Employee

The "(?=\d{4})" is a positive lookahead assertion. It's not strictly necessary, but just ensures that we're only looking for \s+Y immediately followed by four numbers. Your TIME_PREFIX would work equally well.

0 Karma

liviab
Explorer

Stephen, thanks a lot!!!
It works beautifully!! 😉

Can you only explain the TIME_PREFIX regex? Why did you use "(?=\d{4})"?

0 Karma

Lowell
Super Champion

Can you explain the purpose of the "[\s\S]+" before and after all of your regexes?

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...