Getting Data In

How to use the timestamp from a file name at index time?

sarit_s
Communicator

hello

I have log files that does not contain timestamp
i want splunk to take the time stamp from the file name
file name format is like :

2019-04-03T07:33:05.929Z_1.91.0.192_1.88.0.0_5.9.6418.0.zip

Can you help me out with it ?
Please note that my files are indexed from S3 bucket using aws app
Should i use transforms file or something in aws app ?

thanks

0 Karma

kmorris_splunk
Splunk Employee
Splunk Employee

You will want to use a custom datetime.xml file. Make a copy of the default datetime.xml and rename it. In the renamed copy, look for one of the existing definitions like _masheddate and copy the whole section. It will look something like the following:

<define name="_masheddate3" extract="year, month, day">
        <text><![CDATA[source::.*?/sampledata/\w+(\d{4})_(\d{2})_(\d{2})\.txt]]></text>
</define>

Change the extract= portion to reflect the components of the date and/or time that you will be pulling from the filename, in the order they exist. So your example might look something like:

You will want to use a custom datetime.xml file. Make a copy of the default datetime.xml and rename it. In the renamed copy, look for one of the existing definitions like _masheddate and copy the whole section. It will look something like the following:

<define name="_masheddate3" extract="year, month, day, hour, minute, second">
        <text><![CDATA[source::.*?/yourpathgoeshere/(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2}):(\d{2})\.*\.txt]]></text>
</define>

You may have to play around with this to get the regex correct. I am just working off of an example I did a while back, that only had the date, not the time. But this should point you in the correct direction.

The last bit you need to do is point at the custom datetime.xml in your props.conf file.

[your_sourcetype_goes_here]
BREAK_ONLY_BEFORE = \(\d{2}:\d{2}:\d{2}\.\d{5}
DATETIME_CONFIG = /etc/system/local/datetime_yourcopy.xml
NO_BINARY_CHECK = true
category = Custom

I hope this is helpful. If you run into problems, take a look at the documentation around datetime.xml.

0 Karma

sarit_s
Communicator

thank you for you detailed answer !
i saw somewhere that in splunk 7.2 there is an option to do it with INGEST_EVAL
im just not sure how to convert my file name format into this

this is the post i saw :
https://answers.splunk.com/answers/311452/how-to-use-date-in-filename-as-the-timestamp-for-e.html

0 Karma

woodcock
Esteemed Legend

I second the vote for datetime.xml.

0 Karma

kmorris_splunk
Splunk Employee
Splunk Employee

Here is a good blog post that might also be helpful:

https://www.splunk.com/blog/2014/04/23/its-that-time-again.html

0 Karma

sarit_s
Communicator

any idea anyone ?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...