Getting Data In

Adding a new CSV file every day as a new index, how do I configure Splunk to use a field in the file as the event timestamp?

Sukisen1981
Champion

Hi -

I am currently adding a new CSV file every day as a new index in Splunk for some requirement.
Issue is: The event time stamp is being picked as the time when we upload the CSV to Splunk, while we want to use the starting field in the CSV, say for example, field reportrundt as the event time stamp.
Format for reportrundt is: 16.08.2016 00:00:00.

However, while setting the Splunk index field while adding the new index, based on this CSV file, Splunk is assigning the event time stamp as the date-time of upload and NOT the field (reportrundt) which is what we want Splunk to base its event timestamp on...

0 Karma
1 Solution

lguinn2
Legend

How are you setting up this input?

If you are using INDEXED_EXTRACTIONS = CSV in props.conf, then the way to specify the timestamp is

[yoursourcetypehere]
INDEXED_EXTRACTIONS = CSV
TIMESTAMP_FIELDS = reportrundt
TIME_FORMAT = %d.%m.%Y %H:%M:%S

Although I am not positive that Splunk will actually look at the TIME_FORMAT field...

If you are not using indexed extractions, then props.conf will be different. If you could show the header line (and maybe one line of the data, obfuscated), that would really help. But even this much might work:

[yoursourcetypehere]
TIME_FORMAT = %d.%m.%Y %H:%M:%S

Personally, I tend to avoid using indexed extractions, and would do this instead:

# in props.conf
[yoursourcetypehere]
TIME_FORMAT = %d.%m.%Y %H:%M:%S
REPORT-ext-fields = extract-CSV-fields

# in transforms.conf
[extract-CSV-fields]
DELIMS = ","
FIELDS = fieldName1, fieldName2, fieldName3   #copied from csv file heading with quotation marks as needed

View solution in original post

lguinn2
Legend

What is the rationale for creating a new index every day? I want to say "this is a really bad idea" - because it is usually a really bad idea, but perhaps there is an important reason for doing it this way.

0 Karma

lguinn2
Legend

How are you setting up this input?

If you are using INDEXED_EXTRACTIONS = CSV in props.conf, then the way to specify the timestamp is

[yoursourcetypehere]
INDEXED_EXTRACTIONS = CSV
TIMESTAMP_FIELDS = reportrundt
TIME_FORMAT = %d.%m.%Y %H:%M:%S

Although I am not positive that Splunk will actually look at the TIME_FORMAT field...

If you are not using indexed extractions, then props.conf will be different. If you could show the header line (and maybe one line of the data, obfuscated), that would really help. But even this much might work:

[yoursourcetypehere]
TIME_FORMAT = %d.%m.%Y %H:%M:%S

Personally, I tend to avoid using indexed extractions, and would do this instead:

# in props.conf
[yoursourcetypehere]
TIME_FORMAT = %d.%m.%Y %H:%M:%S
REPORT-ext-fields = extract-CSV-fields

# in transforms.conf
[extract-CSV-fields]
DELIMS = ","
FIELDS = fieldName1, fieldName2, fieldName3   #copied from csv file heading with quotation marks as needed
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...