Getting Data In

Added a new entry into datetime.xml and I can't seem to get it to work

MedralaG
Communicator

I need to add an additional time format for syslog data into the datetime.xml.
Following the docs I have created a copy of datetime.xml and copied it to /system/local,
added the following stanza to it:

define name="custom_date" extract="year, month, day">
<![CDATA[^<\d+>(\d{4})-(\d{2})-(\d{2})]]>
/define>
define name="custom_time" extract="hour, minute, second, zone">
<![CDATA[^<\d+>\d{4}-\d{2}-\d{2}T(\d{2}):(\d{2}):(\d{2}).\d{4}(\S\d{2}:\d{2})]]>
/define>

timePatterns>
use name="custom_time"/>

datePatterns>
use name="custom_date"/>

I have also created a props.conf in system/local with the following stanza:
[syslog]
DATETIME_CONFIG = /etc/system/local/datetime.xml

Below are example events that I am trying to get the extraction to work on:

-86-2019-01-25T18:26:30.4563+04:00 This is a test event
-86-2019-01-27T19:28:31.4563+00:00 This is a test event

When I try to ingest a file (via the gui) with those events in the file and select syslog as sourcetype I get a warning saying that it could not use strptime to parse the timestamp.
There is a highlight on the even that starts at the beginning of the line but ends at "+04:" , so for some reason it does not include the last two characters (00) in the timezone , even though the regex states that it should include it.

Could someone help out?

0 Karma
1 Solution

chrisyounger
SplunkTrust
SplunkTrust

make sure your TIME_PREFIX and TIME_STAMPLOOKAHEAD are both set correctly in props.conf

You custom_time should look like this:

^<\d+>\d{4}-\d{2}-\d{2}T(\d{2}):(\d{2}):(\d{2}).\d{4}(\S\+?\-?\d{2}:\d{2})

View solution in original post

0 Karma

MedralaG
Communicator

OK, I eventually got to the bottom of this.
The regex that I have put in there is fine , so is the remaining format of the xml file.
I initially discovered that if I create a new sourcetype (and not use "syslog") and get it to use the datetime.xml then it works, so it must have been something in the syslog props definition.
I eventually realised it was the MAX_TIMESTAMP_LOOKAHEAD which is set to 30 for syslog but for any new sourcetype (which will inherit the default settings) it will be set to 128.
The timestamp in the log example that I have ends on the 43rd character of the event.

0 Karma

chrisyounger
SplunkTrust
SplunkTrust

Nice work! Its extra confusing when the sourcetypes are changed during parsing or with a rename...

0 Karma

chrisyounger
SplunkTrust
SplunkTrust

make sure your TIME_PREFIX and TIME_STAMPLOOKAHEAD are both set correctly in props.conf

You custom_time should look like this:

^<\d+>\d{4}-\d{2}-\d{2}T(\d{2}):(\d{2}):(\d{2}).\d{4}(\S\+?\-?\d{2}:\d{2})

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...