Inconsistent indexing time on CSV monitor

[swamysanjanaput]

We have been using daily CSV exports from our "X" monitoring servers that we then display on our performance board each morning.
The" X" server runs an export of current tickets at 06:30 each morning which is exported to a CSV in a location monitored by Splunk.

Recently, the records have been indexed at inconsistent times, causing issues with our graphs.
Although the report always runs at 0630, some records are not being indexed until 12:00 the same day

Note: The csv files are consistently created at 0630 and then not touched until they're rotated out after 7 days.

Eg. 34 events are indexed at 6:30 everyday and 8 events are being indexed at 12 the same day, when further analysed we noticed fields were truncated for one of those 8 events. so tried adding truncate =0 in props.conf and could see all fields being indexed correctly however still facing issues with timestamp for those 8 events.

Could anyone please help or guide me to resolve this timestamp issue? Thanks in advance

The below defined props deployed to HF and UF(note: we are not deploying any configs to our indexers)
[sourcetype]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)
NO_BINARY_CHECK=true
CHARSET=UTF-8
INDEXED_EXTRACTIONS=csv
KV_MODE=none
category=Structured
FIELD_DELIMITER=,
disabled=false
pulldown_type=true

[skalliger · Accepted solution]

What does that CSV look like? Do you have a header in that file with a timestamp field?
If so, make sure to add the parameters TIMESTAMP_FIELDS as well as TIME_FORMAT in the props.conf stanza.

Skalli

[swamysanjanaput]

LastWriteTime Length Name
28/01/2020 6:30 AM 166693 x_alerts_20200128_0630.csv
29/01/2020 6:30 AM 123079 x_alerts_20200129_0630.csv

There is a field in the CSV called TimeRaised but we would like this to be a field in the resultant event, with each individual event having its ingest time recorded as _time. Not sure why only those 8 events having a time stamp as 1/29/2012:00:00.000 PM

[skalliger]

Then you'd rather want to add DATETIME_CONFIG = CURRENT to your props.conf stanza.

[swamysanjanaput]

Thank you @skalliger it worked!

[skalliger]

Thank you for your feedback! Glad it worked.