Getting Data In

How to index the correct timestamp from a log that has two dates?

doubleIQ
Engager

Hi guys,
I have a log file that occasionally logs an event which contains two dates. For example, like this:

2014-10-02 00:25:28.592991+10~^~log_name~^~WARNING: Message for 01st October   2014 etc etc there is missing data~^~2~^~function_name~^~~^~database_name~^~role_name~^~~^~server~^~

The first part of the log entry is the exact time that the message gets logged, so in this case, it's 2nd October at 12:25am.
The next part starting with "Warning..." is just an error message that i want to report on, and it contains a date (1st October 2014).

The problem i am having is that Splunk is reading "01st October 2014" from the information, and indexing it for that time, and not the correct time on 2nd October.
So basically, being brief, the log entry should be getting indexed for the 2nd October, 2014 at 00:25:28am, but it is instead getting indexed for 1st October, 2014 at 00:25:28am. This is causing issues with my alerts that are running etc

Does anyone know how i can get Splunk to not read second date and to index on the first date occurance only?

If i haven't made something clear please let me know.

Your help is much appreciated.

thanks

Tags (3)
1 Solution

linu1988
Champion

Hello,
you have to mention the timeformat which splunk needs to take.

Mention in Props.conf in indexer end.

[Sourcetype]
TIME_FORMAT=%Y-%m-%d %H:%M:%S.%6q

That should be enough. You need to restart the indexer after making this entry.

Thanks,
L

View solution in original post

ibob0304
Communicator

Along with @linu1988 answer you could also add Time_Prefix to make little solid.

 [Sourcetype]
TIME_PREFIX= \d{4}-\d{0,2}-\d{0,2}\s
TIME_FORMAT=%Y-%m-%d %H:%M:%S.%6q
MAX_TIMESTAMP_LOOKAHEAD = 10
0 Karma

linu1988
Champion

Hello,
you have to mention the timeformat which splunk needs to take.

Mention in Props.conf in indexer end.

[Sourcetype]
TIME_FORMAT=%Y-%m-%d %H:%M:%S.%6q

That should be enough. You need to restart the indexer after making this entry.

Thanks,
L

kbecker
Communicator

,You should also set max_timestamp_lookahead in props which tells splunk to only look x amount of characters into the message for a time stamp.

0 Karma

kbecker
Communicator

You are correct it will still work with just time_format but it is best practice to also add max_timestamp_lookahead to reduce processing the message as without it splunk will still look 150 characters into the message when in this example it only needs to look 26. Good blog post on the subject,

http://blogs.splunk.com/2010/03/02/guess-what-time-it-is/

jrodman
Splunk Employee
Splunk Employee

FWIW, MAX_TIMESTAMP_LOOKAHEAD is only important when TIME_FORMAT fails.
It is a good safeguard, but it will not increase performance when TIME_FORMAT works correctly.

linu1988
Champion

not needed once the timestamp rule matched once. It's helpful when you only want to take a specific portion of the time data.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...