hello
I have log files that does not contain timestamp
i want splunk to take the time stamp from the file name
file name format is like :
2019-04-03T07:33:05.929Z_1.91.0.192_1.88.0.0_5.9.6418.0.zip
Can you help me out with it ?
Please note that my files are indexed from S3 bucket using aws app
Should i use transforms file or something in aws app ?
thanks
You will want to use a custom datetime.xml file. Make a copy of the default datetime.xml and rename it. In the renamed copy, look for one of the existing definitions like _masheddate and copy the whole section. It will look something like the following:
<define name="_masheddate3" extract="year, month, day">
<text><![CDATA[source::.*?/sampledata/\w+(\d{4})_(\d{2})_(\d{2})\.txt]]></text>
</define>
Change the extract= portion to reflect the components of the date and/or time that you will be pulling from the filename, in the order they exist. So your example might look something like:
You will want to use a custom datetime.xml file. Make a copy of the default datetime.xml and rename it. In the renamed copy, look for one of the existing definitions like _masheddate and copy the whole section. It will look something like the following:
<define name="_masheddate3" extract="year, month, day, hour, minute, second">
<text><![CDATA[source::.*?/yourpathgoeshere/(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2}):(\d{2})\.*\.txt]]></text>
</define>
You may have to play around with this to get the regex correct. I am just working off of an example I did a while back, that only had the date, not the time. But this should point you in the correct direction.
The last bit you need to do is point at the custom datetime.xml in your props.conf file.
[your_sourcetype_goes_here]
BREAK_ONLY_BEFORE = \(\d{2}:\d{2}:\d{2}\.\d{5}
DATETIME_CONFIG = /etc/system/local/datetime_yourcopy.xml
NO_BINARY_CHECK = true
category = Custom
I hope this is helpful. If you run into problems, take a look at the documentation around datetime.xml.
thank you for you detailed answer !
i saw somewhere that in splunk 7.2 there is an option to do it with INGEST_EVAL
im just not sure how to convert my file name format into this
this is the post i saw :
https://answers.splunk.com/answers/311452/how-to-use-date-in-filename-as-the-timestamp-for-e.html
I second the vote for datetime.xml
.
Here is a good blog post that might also be helpful:
https://www.splunk.com/blog/2014/04/23/its-that-time-again.html
any idea anyone ?