Getting Data In

Why is my datetime.xml configuration not extracting two separate timestamp formats from a single log file?

mookiie2005
Communicator

I am trying to extract two separate timestamp formats from a single log file. Here is a sample of the logfile:

[16 Mar 23:33:49.967] INFO com.hp.exstream.cc.server.core.phase.ExternalProgramPhase:pool-15-thread-14  - (See the entire program output in the file [var/jobs/5127/java.155941.out].)
[16 Mar 23:33:49.967] INFO com.hp.exstream.cc.server.core.phase.ExternalProgramPhase:pool-15-thread-14  - External program finished OK with return code [0].
[16 Mar 23:33:50.012] INFO com.hp.exstream.cc.server.core.job.JobDriver:pool-15-thread-14   - Execution of [SFTP to OPC] phase finished successfully.
[16 Mar 23:33:50.042] INFO com.hp.exstream.cc.server.core.job.JobDriver:pool-15-thread-14   - Starting execution of [SFTP hal to OPC] phase.
[16 Mar 23:33:50.044] INFO com.hp.exstream.cc.server.core.phase.ExternalProgramPhase:pool-15-thread-14  - Executing program [java] with arguments [-jar /opt/hp/ccdm/delivery-manager-2.3.0/command-center-2.3.0/lib/java/CCDeliveryClient.jar -c sftp -t put -u USWPLA  -p PLA2014highway -v cachi1-bh001.amer.zurich.corp -n 22 -l /opt/hp/ccdm/delivery-manager-2.3.0/command-center-2.3.0/var/jobs/5127 -r /PROD/PLA -x *.hal] in directory [/opt/hp/ccdm/delivery-manager-2.3.0/command-center-2.3.0/var/jobs/5127].
[16 Mar 23:33:56.121] INFO com.hp.exstream.cc.server.core.phase.ExternalProgramPhase:pool-15-thread-14  - Program output: 
(last 1024 chars) ... ession - is present.  The value is: *.hal
[INFO ]20150316@23:33:50,496:com.hpexstream.client.Sftp - Start FTP process.
[DEBUG]20150316@23:33:50,502:com.hpexstream.util.Find - Use file filter: WildcardFileFilter(*.hal)
[DEBUG]20150316@23:33:50,803:com.hpexstream.util.Find - Found file: VI0H150316.hal
[DEBUG]20150316@23:33:50,803:com.hpexstream.util.Find - Found file: VT0H150316.hal
[INFO ]20150316@23:33:50,803:com.hpexstream.client.Sftp - Found files count: 2
[INFO ]20150316@23:33:51,877:com.hpexstream.client.Sftp - Open SFTP channel
[INFO ]20150316@23:33:51,877:com.hpexstream.client.Sftp - Put local file: /opt/hp/ccdm/delivery-manager-2.3.0/command-center-2.3.0/var/jobs/5127/VI0H150316.hal to remote file: /PROD/PLA/VI0H150316.hal
[INFO ]20150316@23:33:55,694:com.hpexstream.client.Sftp - Put local file: /opt/hp/ccdm/delivery-manager-2.3.0/command-center-2.3.0/var/jobs/5127/VT0H150316.hal to remote file: /PROD/PLA/VT0H150316.hal

My props.conf looks like the below:

[datetime]
DATETIME_CONFIG = D:\Program Files\SplunkUniversalForwarder\etc\apps\FORMS\local\datetime.xml
LINE_BREAKER = ([\r\n])+(?=(\[\d{2}\s(\w{3})\s(\d{2}):(\d{2}):(\d{2}).(\d{3})]|\[(\d{4})(\d{2})(\d{2})@(\d{2}):(\d{2}):(\d{2})\,(\d{3})\]))
SHOULD_LINEMERGE=false

Here is my datetime.xml:

\[\d{2}\s(\w{3})\s(\d{2}):(\d{2}):(\d{2}).(\d{3})\]

\[(\d{4})(\d{2})(\d{2})@(\d{2}):(\d{2}):(\d{2})\,(\d{3})\]
0 Karma

cpetterborg
SplunkTrust
SplunkTrust

The problem is that you have square brackets around the date in your second format. Here is what you probably should have in your datetime.xml file:

<datetime>
<define name="_mydatetimeformat1" extract="day, month, hour, minute, second, subsecond">
    <text>\[\d{2}\s(\w{3})\s(\d{2}):(\d{2}):(\d{2}).(\d{3})\]</text>
</define>
<define name="_mydatetimeformat2" extract="year, month, day, hour, minute, second, subsecond">
     <text>(\d{4})(\d{2})(\d{2})@(\d{2}):(\d{2}):(\d{2})\,(\d{3})</text>
</define>
<timePatterns>
     <use name="_mydatetimeformat1"/>
     <use name="_mydatetimeformat2"/>
</timePatterns>
<datePatterns>
     <use name="_mydatetimeformat1"/>
     <use name="_mydatetimeformat2"/>
</datePatterns>
</datetime>

I think you may have a problem not having a year in the first date format.

0 Karma

nitesh218ss
Communicator

Hi in my log event and filename date is not present i want give a fix date to log so what is do ?

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...