Getting Data In

Splunk is getting confused Australian Timestamps thinking its US when Days (DD) are from 01 to 10 (Splunk6)

Dark_Ichigo
Builder

**Please Note:* This works fine with Splunk V4 but not Splunk V6.*

If the day of the month is below the 10th digit as the DD of a timestamp splunk will assume the date to be in US format (MM/DD/YYYY), otherwise it will correctly index the date as per the props Config as DD/MM/YYYY eg:

When splunk indexes the following log, it’s being indexed as being dated 10th of September

09/10/2013 04:30:28.115036 TRACE m 913   41    "[P00861257] FROM_LOCKER LCK;OW="hhhhhh,M,41";RE=OK;LO="xxxxxxxx""
09/10/2013 04:30:28.115569 TRACE m 913   41    "[P00861257] TO_EXCHANGE:MSDVXX=xxxxxx"
09/10/2013 04:30:28.156988 TRACE d 950   1     "[P00861257] COMMS_QUEUING:MSDVXX: RECV='08/10/2013 04:30:28.116353' QUEUE_TIME='         ' TO_DEV='08/10/2013 04:30:28.116878' FROM_DEV='08/10/2013 04:30:28.156627' DEV_TIME='00000.039'"

When splunk indexes the following log, it’s being indexed as being dated 13th of October (as per Config in props.conf)

13/10/2013 04:30:28.115036 TRACE m 913   41    "[P00861257] FROM_LOCKER LCK;OW="hhhhhh,M,41";RE=OK;LO="xxxxxxxx""
13/10/2013 04:30:28.115569 TRACE m 913   41    "[P00861257] TO_EXCHANGE:MSDVXX=xxxxxx"
13/10/2013 04:30:28.156988 TRACE d 950   1     "[P00861257] COMMS_QUEUING:MSDVXX: RECV='08/10/2013 04:30:28.116353' QUEUE_TIME='         ' TO_DEV='08/10/2013 04:30:28.116878' FROM_DEV='08/10/2013 04:30:28.156627' DEV_TIME='00000.039'"

######################################

Indexes.conf

 [index_name]
    homePath = $SPLUNK_DB/index_name/db
    coldPath = $SPLUNK_DB/index_name/colddb
    thawedPath = $SPLUNK_DB/index_name/thaweddb
    disabled = 0
    frozenTimePeriodInSecs = 2147483647

#####################################

props.conf

[source2]
MAX_EVENTS = 30000
TRUNCATE = 0
TIME_FORMAT=%d/%m/%Y %H:%M:%S.%6N
MAX_TIMESTAMP_LOOKAHEAD = 20
TIME_PREFIX = ^
TZ = Australia/Victoria

######################################

Forwarder Config (inputs.conf)

[default]
host = ABC  
[monitor://home/file/location/]
index = index
sourcetype = source2
crcSalt = <SOURCE>
disabled = false
1 Solution

mchang_splunk
Splunk Employee
Splunk Employee

The root cause should be MAX_TIMESTAMP_LOOKAHEAD = 20.
The total length of the time stamp in the event is 26, to identify microsecond correctly, you should set:
MAX_TIMESTAMP_LOOKAHEAD = 26
In this way, this issue should be resolved in 6.0 and the correct microsecond will be shown in 4.x and 5.x.

View solution in original post

mchang_splunk
Splunk Employee
Splunk Employee

The root cause should be MAX_TIMESTAMP_LOOKAHEAD = 20.
The total length of the time stamp in the event is 26, to identify microsecond correctly, you should set:
MAX_TIMESTAMP_LOOKAHEAD = 26
In this way, this issue should be resolved in 6.0 and the correct microsecond will be shown in 4.x and 5.x.

Dark_Ichigo
Builder

We deployed it and it worked fine.

Cheers!

0 Karma

mchang_splunk
Splunk Employee
Splunk Employee

Please try having props.conf on the forwarder, I guess you have installed the heavy weight forwarder, not universal forwarder.

0 Karma

Dark_Ichigo
Builder

I'm using a Universal forwarder

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...