Getting Data In

TIME_FORMAT, it's May, not January

howyagoin
Contributor

Overnight I noticed that my Splunk was suspiciously empty for a specific CSV file which was read in daily.

Upon further investigation it turned out this was due to the fact that the file now has data for May 1, or, 01/05/2011....the CSV auto-parsing saw the format for the last half of last month and worked out, properly, that it was Day/Month/Year, (16/04/2011 was pretty obvious) but now, well, it's a bit fuzzier.

Reading up in the manuals, it looks like the answer is to set the TIME_FORMAT in props.conf, which I have done with:

[source::/A/B/C/data/MyFile*]
TIME_FORMAT = %d/%m/%y %H:%M

The source is already being indexed with a inputs.conf entry for "/A/B/C/data"

The data looks like:

3812305781230123    , 7773213          , 9099, B, 75, INTERNET, 01/05/2011 23:58

However, Splunk is still seeing this as data for January.

What should I do to fix this?

Thanks!

Tags (1)
1 Solution

dmlee
Communicator

Hi, I think you should use %Y instead of %y , because the year is "2011" (four digits).
and , if possible, you can use TIME_PREFIX to tell splunk where is the exact date field you want splunk to recognize.

View solution in original post

0 Karma

dmlee
Communicator

Hi, I think you should use %Y instead of %y , because the year is "2011" (four digits).
and , if possible, you can use TIME_PREFIX to tell splunk where is the exact date field you want splunk to recognize.

0 Karma

howyagoin
Contributor

For the benefit of those reading who may want a concrete example:


TIME_PREFIX = ^(?:[^\,]+,){6}\s*
TIME_FORMAT = %d/%m/%Y %H:%M

0 Karma

howyagoin
Contributor

Hah, thanks, great catch on the %Y. That didn't actually fix my issue, but, the TIME_PREFIX seems to get me closer. There are six fields, comma separated, before my date/time field, so I just need to work out the regex on this -- as some of the fields are empty sometimes, othertimes not (annoying).

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...