Any way to extract date information from file name and time information from message ?

[cweiliou_splunk]

I have some source files which the messages have only time information without date information as below.
[ xxxxx20171003174502.txt ]

host pts/10 mmmm01 Wed Oct 03 11:46 xxxxxxxx
host ftp wwww01 Wed Oct 03 11:47 yyyyyyy

[ yyyyy20181114174607.txt ]

host2 www/10 ppppp05 Wed Nov 14 12:46 -----
zzzz ftp uuuu01 Wed Nov 14 16:31 nnnnn

The file name itself has the date information.
Is there a way to extract date information from file name and time information from messages within the file ?
For example, for above xxxxx20171003174502.txt's case, I would like to extract date information which is 2017/10/03 from file name xxxxx20171003174502.txt and extract time information "11:46" from message "host pts/10 mmmm01 Wed Oct 03 11:46 xxxxxxxx".

[cweiliou_splunk · Accepted solution]

You should be able to extract date information (year, month and day) from your file name and time information (hour, minute) from your message saved inside your log file by following below steps.
Note that this is just an example of how to do it. You should adjust the regex accordingly to fix your needs.

1) Create a new datetime.xml with below content and save it under SPLUNK_HOME/etc/system/local. Note that you need to add "source::" if you would like to extract information from source file.

<datetime>
  <define name="my_custom_date" extract="year,month,day">
     <text><![CDATA[source::.*?(\d{4})(\d{2})(\d{2})]]></text>
  </define>
  <define name="my_custom_time" extract="hour,minute">
     <text><![CDATA[(\d{2}):(\d{2})]]></text>
  </define>
  <timePatterns>
     <use name="my_custom_time"/>
  </timePatterns>
  <datePatterns>
     <use name="my_custom_date"/>
  </datePatterns>
</datetime>

2) Create below props.conf and save it under SPLUNK_HOME/etc/system/local/

[my_custom_sourcetype]
DATETIME_CONFIG=/etc/system/local/datetime.xml
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=false

3) Restart splunk and use oneshot to test the settings

opt/splunk/bin/splunk add oneshot /tmp/xxxxx20171003174502.txt -sourcetype my_custom_sourcetype
opt/splunk/bin/splunk add oneshot /tmp/yyyyy20181114174607.txt -sourcetype my_custom_sourcetype

4) Confirm the extracted date and time information by running below search

index=main | eval idxtime=_indextime | convert ctime(idxtime) | table idxtime _time _raw source