Index time not same as log message time

[jalfrey]

I just setup another splunk server. Foolishly I forgot to turn on NTP and the system clock was way off. The first chunk of log messages came in via Syslog and are indexed on when the were received by syslog (local time) not the correct date/time that appears in the log message.

Jun 12 03:59:58 10.0.59.59 id=firewall sn=0017C569F354 time="2013-06-12 10:59:59" fw=10.0.59.59 pri=6 c=1024 m=537 msg="Connection Closed" app=49176 sess=Web n=3268362 usr="admin" src=10.0.0.236:54609:X1 dst=10.0.59.59:80:X1 proto=tcp/http sent=775 rcvd=1659

As you can see from the log above the local time is: "12 03:59:58" where the log message time (remote) is time="2013-06-12 10:59:59"

do I need to teach splunk how to extract that date/time field or do I need to switch splunk to index based on log source time instead? If so how do I do that?

[chris · Accepted solution]

Splunk tries to find the timestamp in an event automatically. This works sometimes, but not in your case.

You can read about time stamp assignement here and there is even an example of what can be done if events contain more than one timestamp here

What you have to do to get started is create a props.conf file in $SPLUNK_HOME/etc/system/local
Insert the following:

[source::/whatever/is/displayed/in/the/source/field/of/your/events]
TIME_PREFIX = time="
TIME_FORMAT = %Y-%m-%d %H:%M:%S 

And then restart Splunk

[chris]

Hi jalfrey, you can apply props stanzas to sources or sourcetypes. The source is the path to a file the sourcetype is used to classify different sources with the same format. It is a good thing to work with sourcetypes. Splunk comes with predefined sourcetypes but you can also create your own if you have home grown applications with special log formats. I posted the example with the [source::xy] stanza because I thought it's the easiest way to start. Here is more Info:
http://docs.splunk.com/Documentation/Splunk/latest/Data/Whysourcetypesmatter

[jalfrey]

I'm confused about the /whaterver/is/displayed/in ...
Reading the article they had [source::/Applications/splunk/var/spool/splunk] but the source types documents says things like:
access_combined
access_combined_wcookie
asterisk_queue

Currently I think my sourcetype is syslog