Wrong time stamp for splunk search events

[rreddy012c]

Please see the below events timestamp with _raw time stamp it seems like _raw time stamp assigning minutes as HRS and seconds as minutes to _time event time stamp.

          _time                                           _raw

2017-10-10T16:09:00.000-0400 [10/10/2017 9:16:09] insert into #temp_ord_version values ( **, *, 169, 169 )

2017-10-10T16:09:00.000-0400 [10/10/2017 9:16:09] insert into #temp_ord_version values ( **, *, 18, 18 )

2017-10-10T16:09:00.000-0400 [10/10/2017 9:16:09] insert into #temp_ord_version values ( **, ****, 20, 20 )

_time time stamp -> 2017-10-10T16:09:00.000-0400 -> minutes as HRS and seconds as minutes to _time event time stamp from _raw
_raw time stamp -> [10/10/2017 9:16:09]

[gcusello]

Hi rreddy012c,
what is the TIME_FORMAT of your props.conf?
it should be

%Y-%m-%dT%H:%M:%S.%3N-%Z

Bye.
Giuseppe

[cpetterborg]

Not quite, Guiseppe. The data above it the table of data and 2017-10-10T16:09:00.000-0400 is _time, while [10/10/2017 9:16:09] is the timestamp in the data, so the TIME_FORMAT should be either:

%d/%m/%Y %H:%M:%S

or

%m/%d/%Y %H:%M:%S

depending on locale (because from the data you can't tell if it is day/month/year or month/day/year).

I'm wondering if there is an app that has been put in place (like a TA) that is causing the error, like you have stated.

For what it's worth, @rreddyo12c, defining things in props for the different sourcetypes is Splunk best practice and is well worth it when you run into problems exactly like this one. Additionally your indexing performance will be much better as well.

[gcusello]

Hi rreddy012c,
you could modify your datetime.xml (see https://answers.splunk.com/answers/147950/can-i-have-different-timestamp-formats-using-the-same-sour... ) so you don't need to ust TIME_FORMAT.
Bye.
Giuseppe

[rreddy012c]

We recently deployed splunk in our organization. Actually we have not define any TIME_FORMAT in props.conf

[gcusello]

Hi rreddy012c,
Check what is the sourcetype of the described logs and then insert in indexers props.conf this TIME_FORMAT and after Splunk restart check if you still have the problem.

Bye.
Giuseppe

[rreddy012c]

Actually we have 1500+ applications so we are using dynamic sourcetype I think if we update TIME_FORMAT that would effect for all applications.

[gcusello]

Surely, but if without using TIME_FORMAT you have the described problem, you should check if you have the same problem also in other uses of the same log.
Only for my information: what do you mean with more than 1500 applications? do you mean that you have logs from more of 1500 applications?
Anyway I suggest to perform a check and think to fix the time format.
Bye.
Giuseppe