Hello All,
we have started working with splunk to deal with a pile of date. for that we have created a custom source type and put it in the props.conf file. it looks like this:
[mk_csv]
CHECK_FOR_HEADER = true
KV_MODE = none
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = False
TIME_FORMAT = %Y-%jT%H:%M:%S.%3N
pulldown_type = 1
TZ = UTC
that is in ./splunk/etc/system/local/props.conf and yes we restarted the server
So far so good. we added the source type to the data inputs. we built a new index mk_mission.
Now from the search window if I run a sourcetype="mk_csv" nothing shows up. however I do find that there is now a mk_csv-3 with 37 events in it, it even correctly displays the julian dates (the %j in the time_format)
last problem. I configured the data inputs to use julian time as seen above however everything indexed is showing up with the wrong dates, always a bit early
examples:
search index for sourcetype="mk_csv-3"
record returned:
4/13/12 6:32:52.138 PM with the time stamp from the entry as: 2012-104T18:32:52.138
that looks like the sourcetype in the props.conf is working correctly
search for:
index="mk_mission_gra_eng"
get:
4/6/12 9:11:09.467 PM from 2012-088T21:11:09.467
4/6/12 5:42:38.170 PM from 2012-081T17:42:38.170
4/6/12 4:05:10.097 PM from 2012-102T16:05:10.097
4/5/12 11:17:30.163 PM from 2012-101T23:17:30.163
as near as I can tell this is not what I would expect since we set the source type as the data input level.
I generally would advise not to use CHECK_FOR_HEADER
. As far as the time stamp issue, that's tricky to debug, but my guess is Splunk is for whatever reason not reading your Julian days and getting the date from the last-mod time on the files.
how do you eliminate the static column headers from the csv files without CHECK_FOR_HEADERS?