Getting Data In

How to use the timestamp from a file name at index time?

sarit_s
Communicator

hello

I have log files that does not contain timestamp
i want splunk to take the time stamp from the file name
file name format is like :

2019-04-03T07:33:05.929Z_1.91.0.192_1.88.0.0_5.9.6418.0.zip

Can you help me out with it ?
Please note that my files are indexed from S3 bucket using aws app
Should i use transforms file or something in aws app ?

thanks

0 Karma

kmorris_splunk
Splunk Employee
Splunk Employee

You will want to use a custom datetime.xml file. Make a copy of the default datetime.xml and rename it. In the renamed copy, look for one of the existing definitions like _masheddate and copy the whole section. It will look something like the following:

<define name="_masheddate3" extract="year, month, day">
        <text><![CDATA[source::.*?/sampledata/\w+(\d{4})_(\d{2})_(\d{2})\.txt]]></text>
</define>

Change the extract= portion to reflect the components of the date and/or time that you will be pulling from the filename, in the order they exist. So your example might look something like:

You will want to use a custom datetime.xml file. Make a copy of the default datetime.xml and rename it. In the renamed copy, look for one of the existing definitions like _masheddate and copy the whole section. It will look something like the following:

<define name="_masheddate3" extract="year, month, day, hour, minute, second">
        <text><![CDATA[source::.*?/yourpathgoeshere/(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2}):(\d{2})\.*\.txt]]></text>
</define>

You may have to play around with this to get the regex correct. I am just working off of an example I did a while back, that only had the date, not the time. But this should point you in the correct direction.

The last bit you need to do is point at the custom datetime.xml in your props.conf file.

[your_sourcetype_goes_here]
BREAK_ONLY_BEFORE = \(\d{2}:\d{2}:\d{2}\.\d{5}
DATETIME_CONFIG = /etc/system/local/datetime_yourcopy.xml
NO_BINARY_CHECK = true
category = Custom

I hope this is helpful. If you run into problems, take a look at the documentation around datetime.xml.

0 Karma

sarit_s
Communicator

thank you for you detailed answer !
i saw somewhere that in splunk 7.2 there is an option to do it with INGEST_EVAL
im just not sure how to convert my file name format into this

this is the post i saw :
https://answers.splunk.com/answers/311452/how-to-use-date-in-filename-as-the-timestamp-for-e.html

0 Karma

woodcock
Esteemed Legend

I second the vote for datetime.xml.

0 Karma

kmorris_splunk
Splunk Employee
Splunk Employee

Here is a good blog post that might also be helpful:

https://www.splunk.com/blog/2014/04/23/its-that-time-again.html

0 Karma

sarit_s
Communicator

any idea anyone ?

0 Karma
Get Updates on the Splunk Community!

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...