Getting Data In

How does Splunk get date from file

tgow
Splunk Employee
Splunk Employee

I have a file that lists the date at the top and also in the name. Here is a snippet of the first 10 lines of the file.


Log file opened: 05/09/2011 08:15:28.889 CDT



Director 14644: 08:15:28.890 User [admin] tries to log in



Director 14644: 08:15:28.910 validate.asp: calling SM_CreateNewSession()



Director 14644: 08:15:28.911 validate.asp: logon_ip:::1user_id:adminbrowser_ip:::1



Director 14644: 08:15:28.912 ::1:admin:::1:changeme:0:::0



Director 14644: 08:15:28.917 sm.js: SM_CreateNewSession, json_data: {"request-id":1,"topic":"system","message":"session","actions":[{"action":"create","id":1,"user-id":"admin","browser-ip":"::1","client-nonce":1756958650,"sha1":"63180aca427f33649593b87ee48b58c2a53399d9"}]}



Director 14644: 08:15:28.920 sm.js: SM_CreateNewSession, caught error:



Director 14644: 08:15:28.921 sm.js: WinHttp returned error: 12005 The URL is invalid


The name of the file is the following: Director-110509.081528.log

I setup a props.conf file with the following to create single events for each line of the file because originally Splunk was not recognizing the new lines and only created one event for the whole file.


[director]



SHOULD_LINEMERGE = False


The events in Splunk have the date of 11/05/09 but it should be 5/09/2011.

How does Splunk determine the date of the events in the file and how can I change the date to be the correct one.

Thanks in advance

Tags (2)
0 Karma
1 Solution

bbingham
Builder

Splunk looks at the start of each line for the date information and matches to any known set date format, in your case, it looks to be matching and failing, so it looks to the file name, Director-110509.081528.log.

Here's the order splunk looks for a timestamp:

Splunk uses the following precedence to assign timestamps to events:

 1. Look for a time or date in the event itself using an explicit TIME_FORMAT if provided.

 Use positional timestamp extraction for events that have more than one timestamp value in the raw data.

  2. If no TIME_FORMAT is provided, or no match is found, attempt to automatically identify a time or date in the event itself.

  3. If an event doesn't have a time or date, use the timestamp from the most recent previous event of the same source.

  4. If no events in a source have a time or date, look in the source (or file) name.

  5. For file sources, if no time or date can be identified in the file name, use the modification time on the file.

  6. If no other timestamp is found, set the timestamp to the current system time (the time at which the event is indexed by Splunk).

You may want to simply remove the date out of the filename and let splunk use the modification time on the file. Hope this helps!

View solution in original post

bbingham
Builder

Splunk looks at the start of each line for the date information and matches to any known set date format, in your case, it looks to be matching and failing, so it looks to the file name, Director-110509.081528.log.

Here's the order splunk looks for a timestamp:

Splunk uses the following precedence to assign timestamps to events:

 1. Look for a time or date in the event itself using an explicit TIME_FORMAT if provided.

 Use positional timestamp extraction for events that have more than one timestamp value in the raw data.

  2. If no TIME_FORMAT is provided, or no match is found, attempt to automatically identify a time or date in the event itself.

  3. If an event doesn't have a time or date, use the timestamp from the most recent previous event of the same source.

  4. If no events in a source have a time or date, look in the source (or file) name.

  5. For file sources, if no time or date can be identified in the file name, use the modification time on the file.

  6. If no other timestamp is found, set the timestamp to the current system time (the time at which the event is indexed by Splunk).

You may want to simply remove the date out of the filename and let splunk use the modification time on the file. Hope this helps!

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...