Getting Data In

Custom datetime.xml for x12 format

hogan24
Path Finder

Trying to get datetime.xml configured to recognize a timestamp in x12 file format with no success...

Here are the possibilities of what the data could look like within the file:
[~GS*HS*123456*ASDF*20150519*0642896109*X*005010X279~
[~GS*HS*123456*ASDF
20150519*064201896109*X*005010X279~
[~GS*HS*123456*ASDF
20150519*06420123*896109*X*005010X279~

<datetime>
    <define name="_x12date" extract="year, month, day">
        <text><![CDATA[(?:~GS\*(?:.*?\*){3})(19\d\d|20\d\d)(0?[1-9]|1[012])(0[1-9]|[12]\d|3[01])(?:\*)]]></text>
    </define>
    <define name="_x12time1" extract="hour, minute, second, subsecond">
        <text><![CDATA[(?:~GS\*(?:.*?\*){4})(\d{2})(\d{2})(\d{2})(\d{2})(?:\*)]]></text>
    </define>
    <define name="_x12time2" extract="hour, minute, second">
        <text><![CDATA[(?:~GS\*(?:.*?\*){4})(\d{2})(\d{2})(\d{2})(?:\*)]]></text>
    </define>
    <define name="_x12time3" extract="hour, minute">
        <text><![CDATA[(?:~GS\*(?:.*?\*){4})(\d{2})(\d{2})(?:\*)]]></text>
    </define>
    <timePatterns>
          <use name="_x12time1"/>
          <use name="_x12time2"/>
          <use name="_x12time3"/>
    </timePatterns>
    <datePatterns>
          <use name="_x12date"/>
    </datePatterns>
</datetime>

props.conf looks like this:

[x12:270]
TRUNCATE = 0
DATETIME_CONFIG = /etc/apps/x12/local/datetime.xml

Any help would be appreciated as to why the timestamp is not being picked up. Thanks.

Tags (1)
1 Solution

hogan24
Path Finder

I figured this one out...I was trying to do my prefix in the regex which I believe was causing an issue. So I removed the everything in the regex up to the actual timestamp itself and put the prefix regex stanza in the TIME_PREFIX var leaving my props.conf to look like this:

[x12:270]
TRUNCATE = 0
DATETIME_CONFIG = /etc/apps/x12/local/datetime.xml
MAX_TIMESTAMP_LOOKAHEAD = 20
TIME_PREFIX = ~GS\*(?:.*?\*){3}

Here is what datetime.xml ended up looking like:

<datetime>

<define name="_year" extract="year">
    <text><![CDATA[(20\d\d|19\d\d|[901]\d(?!\d))]]></text>
</define>

<define name="_month" extract="month">
    <text><![CDATA[(0?[1-9]|1[012])(?!:)]]></text>
</define>

<define name="_day"  extract="day">
    <text><![CDATA[(0?[1-9]|[12]\d|3[01])]]></text> 
</define>

<define name="_hour" extract="hour">
    <text><![CDATA[([01]?[1-9]|[012][0-3])(?!\d)]]></text>
</define>

<define name="_minute" extract="minute">
    <text><![CDATA[([0-6]\d)(?!\d)]]></text>
</define>

<define name="_second" extract="second">
    <text><![CDATA[([0-6]\d)(?!\d)]]></text>
</define>

<define name="_x12date1" extract="year, month, day, hour, minute, second">
    <text><![CDATA[(19\d\d|20\d\d)(0?[1-9]|1[012])(0[1-9]|[12]\d|3[01])\*(\d{2})(\d{2})(\d{2})]]></text>
</define>

<define name="_x12date2" extract="year, month, day, hour, minute">
    <text><![CDATA[(19\d\d|20\d\d)(0?[1-9]|1[012])(0[1-9]|[12]\d|3[01])\*(\d{2})(\d{2})]]></text>
</define>

<timePatterns>
      <use name="_x12date1"/>
      <use name="_x12date2"/>
</timePatterns>
<datePatterns>
      <use name="_x12date1"/>
      <use name="_x12date2"/>
</datePatterns>

</datetime>

View solution in original post

hogan24
Path Finder

I figured this one out...I was trying to do my prefix in the regex which I believe was causing an issue. So I removed the everything in the regex up to the actual timestamp itself and put the prefix regex stanza in the TIME_PREFIX var leaving my props.conf to look like this:

[x12:270]
TRUNCATE = 0
DATETIME_CONFIG = /etc/apps/x12/local/datetime.xml
MAX_TIMESTAMP_LOOKAHEAD = 20
TIME_PREFIX = ~GS\*(?:.*?\*){3}

Here is what datetime.xml ended up looking like:

<datetime>

<define name="_year" extract="year">
    <text><![CDATA[(20\d\d|19\d\d|[901]\d(?!\d))]]></text>
</define>

<define name="_month" extract="month">
    <text><![CDATA[(0?[1-9]|1[012])(?!:)]]></text>
</define>

<define name="_day"  extract="day">
    <text><![CDATA[(0?[1-9]|[12]\d|3[01])]]></text> 
</define>

<define name="_hour" extract="hour">
    <text><![CDATA[([01]?[1-9]|[012][0-3])(?!\d)]]></text>
</define>

<define name="_minute" extract="minute">
    <text><![CDATA[([0-6]\d)(?!\d)]]></text>
</define>

<define name="_second" extract="second">
    <text><![CDATA[([0-6]\d)(?!\d)]]></text>
</define>

<define name="_x12date1" extract="year, month, day, hour, minute, second">
    <text><![CDATA[(19\d\d|20\d\d)(0?[1-9]|1[012])(0[1-9]|[12]\d|3[01])\*(\d{2})(\d{2})(\d{2})]]></text>
</define>

<define name="_x12date2" extract="year, month, day, hour, minute">
    <text><![CDATA[(19\d\d|20\d\d)(0?[1-9]|1[012])(0[1-9]|[12]\d|3[01])\*(\d{2})(\d{2})]]></text>
</define>

<timePatterns>
      <use name="_x12date1"/>
      <use name="_x12date2"/>
</timePatterns>
<datePatterns>
      <use name="_x12date1"/>
      <use name="_x12date2"/>
</datePatterns>

</datetime>

timothywatson
Path Finder

Hogan24, I am very impressed with your efforts to parse EDI X12. This appears to be a 270. Have you been successful at parsing the full transaction? Have you worked with any other HIPAA Transactions, like the 271 or the 276/277? This solution is the only hit I got regarding EDI/X12 in all of Splunkland. Can you direct me to guidance? Pretty-Please???

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...