Trying to get datetime.xml configured to recognize a timestamp in x12 file format with no success...
Here are the possibilities of what the data could look like within the file:
[~GS*HS*123456*ASDF*20150519*0642896109*X*005010X279~
[~GS*HS*123456*ASDF20150519*064201896109*X*005010X279~
[~GS*HS*123456*ASDF20150519*06420123*896109*X*005010X279~
<datetime>
<define name="_x12date" extract="year, month, day">
<text><![CDATA[(?:~GS\*(?:.*?\*){3})(19\d\d|20\d\d)(0?[1-9]|1[012])(0[1-9]|[12]\d|3[01])(?:\*)]]></text>
</define>
<define name="_x12time1" extract="hour, minute, second, subsecond">
<text><![CDATA[(?:~GS\*(?:.*?\*){4})(\d{2})(\d{2})(\d{2})(\d{2})(?:\*)]]></text>
</define>
<define name="_x12time2" extract="hour, minute, second">
<text><![CDATA[(?:~GS\*(?:.*?\*){4})(\d{2})(\d{2})(\d{2})(?:\*)]]></text>
</define>
<define name="_x12time3" extract="hour, minute">
<text><![CDATA[(?:~GS\*(?:.*?\*){4})(\d{2})(\d{2})(?:\*)]]></text>
</define>
<timePatterns>
<use name="_x12time1"/>
<use name="_x12time2"/>
<use name="_x12time3"/>
</timePatterns>
<datePatterns>
<use name="_x12date"/>
</datePatterns>
</datetime>
props.conf looks like this:
[x12:270]
TRUNCATE = 0
DATETIME_CONFIG = /etc/apps/x12/local/datetime.xml
Any help would be appreciated as to why the timestamp is not being picked up. Thanks.
I figured this one out...I was trying to do my prefix in the regex which I believe was causing an issue. So I removed the everything in the regex up to the actual timestamp itself and put the prefix regex stanza in the TIME_PREFIX var leaving my props.conf to look like this:
[x12:270]
TRUNCATE = 0
DATETIME_CONFIG = /etc/apps/x12/local/datetime.xml
MAX_TIMESTAMP_LOOKAHEAD = 20
TIME_PREFIX = ~GS\*(?:.*?\*){3}
Here is what datetime.xml ended up looking like:
<datetime>
<define name="_year" extract="year">
<text><![CDATA[(20\d\d|19\d\d|[901]\d(?!\d))]]></text>
</define>
<define name="_month" extract="month">
<text><![CDATA[(0?[1-9]|1[012])(?!:)]]></text>
</define>
<define name="_day" extract="day">
<text><![CDATA[(0?[1-9]|[12]\d|3[01])]]></text>
</define>
<define name="_hour" extract="hour">
<text><![CDATA[([01]?[1-9]|[012][0-3])(?!\d)]]></text>
</define>
<define name="_minute" extract="minute">
<text><![CDATA[([0-6]\d)(?!\d)]]></text>
</define>
<define name="_second" extract="second">
<text><![CDATA[([0-6]\d)(?!\d)]]></text>
</define>
<define name="_x12date1" extract="year, month, day, hour, minute, second">
<text><![CDATA[(19\d\d|20\d\d)(0?[1-9]|1[012])(0[1-9]|[12]\d|3[01])\*(\d{2})(\d{2})(\d{2})]]></text>
</define>
<define name="_x12date2" extract="year, month, day, hour, minute">
<text><![CDATA[(19\d\d|20\d\d)(0?[1-9]|1[012])(0[1-9]|[12]\d|3[01])\*(\d{2})(\d{2})]]></text>
</define>
<timePatterns>
<use name="_x12date1"/>
<use name="_x12date2"/>
</timePatterns>
<datePatterns>
<use name="_x12date1"/>
<use name="_x12date2"/>
</datePatterns>
</datetime>
I figured this one out...I was trying to do my prefix in the regex which I believe was causing an issue. So I removed the everything in the regex up to the actual timestamp itself and put the prefix regex stanza in the TIME_PREFIX var leaving my props.conf to look like this:
[x12:270]
TRUNCATE = 0
DATETIME_CONFIG = /etc/apps/x12/local/datetime.xml
MAX_TIMESTAMP_LOOKAHEAD = 20
TIME_PREFIX = ~GS\*(?:.*?\*){3}
Here is what datetime.xml ended up looking like:
<datetime>
<define name="_year" extract="year">
<text><![CDATA[(20\d\d|19\d\d|[901]\d(?!\d))]]></text>
</define>
<define name="_month" extract="month">
<text><![CDATA[(0?[1-9]|1[012])(?!:)]]></text>
</define>
<define name="_day" extract="day">
<text><![CDATA[(0?[1-9]|[12]\d|3[01])]]></text>
</define>
<define name="_hour" extract="hour">
<text><![CDATA[([01]?[1-9]|[012][0-3])(?!\d)]]></text>
</define>
<define name="_minute" extract="minute">
<text><![CDATA[([0-6]\d)(?!\d)]]></text>
</define>
<define name="_second" extract="second">
<text><![CDATA[([0-6]\d)(?!\d)]]></text>
</define>
<define name="_x12date1" extract="year, month, day, hour, minute, second">
<text><![CDATA[(19\d\d|20\d\d)(0?[1-9]|1[012])(0[1-9]|[12]\d|3[01])\*(\d{2})(\d{2})(\d{2})]]></text>
</define>
<define name="_x12date2" extract="year, month, day, hour, minute">
<text><![CDATA[(19\d\d|20\d\d)(0?[1-9]|1[012])(0[1-9]|[12]\d|3[01])\*(\d{2})(\d{2})]]></text>
</define>
<timePatterns>
<use name="_x12date1"/>
<use name="_x12date2"/>
</timePatterns>
<datePatterns>
<use name="_x12date1"/>
<use name="_x12date2"/>
</datePatterns>
</datetime>
Hogan24, I am very impressed with your efforts to parse EDI X12. This appears to be a 270. Have you been successful at parsing the full transaction? Have you worked with any other HIPAA Transactions, like the 271 or the 276/277? This solution is the only hit I got regarding EDI/X12 in all of Splunkland. Can you direct me to guidance? Pretty-Please???