How can I correctly parse time from the XML field?

[rsreese]

I am attempting to extract Time using TIME_FORMAT and TIME_PREFIX in props.conf. Would like to understand how to correctly parse the Time from the GMTTime XML field. The original message is read from a file and sent using a universal forwarder. The inputs.conf on the universal forwarder looks like this:

[monitor:///opt/xml/events.txt]
disabled = false
sourcetype = epo
host = lab-epo

The original message looks like this:

<29>1 2017-08-18T02:50:19.0Z LAB-EPO EPOEvents - EventFwd [agentInfo@3401 tenantId="1"] <?xml version="1.0" encoding="UTF-8"?><EPOevent><MachineInfo><MachineName>LAB-WIN7-02</MachineName><AgentGUID>{b37ff914-XXXX-XXXX-8740-91aa851f0e3d}</AgentGUID><IPAddress>192.XXX.XXX.XXX</IPAddress><OSName>Windows 7</OSName><UserName>SYSTEM</UserName><TimeZoneBias>240</TimeZoneBias><RawMACAddress>XXXXXXXX</RawMACAddress></MachineInfo><SoftwareInfo ProductName="McAfee Endpoint Security" ProductVersion="10.5.0" ProductFamily="TVD"><CommonFields><Analyzer>ENDP_AM_1050</Analyzer><AnalyzerName>McAfee Endpoint Security</AnalyzerName><AnalyzerVersion>10.5.0</AnalyzerVersion><AnalyzerHostName>LAB-WIN7-02</AnalyzerHostName><AnalyzerEngineVersion>XXXX.7806</AnalyzerEngineVersion><AnalyzerDetectionMethod>On-Access Scan</AnalyzerDetectionMethod><AnalyzerDATVersion>3075.0</AnalyzerDATVersion></CommonFields><Event><EventID>1278</EventID><Severity>3</Severity><GMTTime>2017-08-18T14:48:53</GMTTime><CommonFields><ThreatCategory>av.detect</ThreatCategory><ThreatEventID>1278</ThreatEventID><ThreatSeverity>2</ThreatSeverity><ThreatName>EICAR test file</ThreatName><ThreatType>test</ThreatType><DetectedUTC>2017-08-18T14:48:53Z</DetectedUTC><ThreatActionTaken>IDS_ALERT_ACT_TAK_DEL</ThreatActionTaken><ThreatHandled>True</ThreatHandled><SourceHostName>LAB-WIN7-02</SourceHostName><SourceProcessName>C:\PROGRAM FILES (X86)\GOOGLE\CHROME\APPLICATION\CHROME.EXE</SourceProcessName><TargetHostName>LAB-WIN7-02</TargetHostName><TargetUserName>LAB-WIN7-02\xadmin</TargetUserName><TargetFileName>C:\USERS\XADMIN\DOWNLOADS\Unconfirmed 408214.crdownload</TargetFileName></CommonFields><CustomFields target="EPExtendedEventMT"><BladeName>IDS_BLADE_NAME_SPB</BladeName><AnalyzerContentCreationDate>2017-08-16T13:00:00Z</AnalyzerContentCreationDate><AnalyzerGTIQuery>False</AnalyzerGTIQuery><ThreatDetectedOnCreation>False</ThreatDetectedOnCreation><TargetName>Unconfirmed 408214.crdownload</TargetName><TargetPath>C:\USERS\XADMIN\DOWNLOADS</TargetPath><TargetHash>44d88612fea8a8f36de82e1278abb02f</TargetHash><TargetFileSize>68</TargetFileSize><TargetModifyTime>2017-08-18T14:48:53Z</TargetModifyTime><TargetAccessTime>2017-08-18T14:48:53Z</TargetAccessTime><TargetCreateTime>2017-08-18T14:48:53Z</TargetCreateTime><Cleanable>False</Cleanable><TaskName>IDS_OAS_TASK_NAME</TaskName><FirstAttemptedAction>IDS_ALERT_THACT_ATT_CLE</FirstAttemptedAction><FirstActionStatus>False</FirstActionStatus><SecondAttemptedAction>IDS_ALERT_THACT_ATT_DEL</SecondAttemptedAction><SecondActionStatus>True</SecondActionStatus><AttackVectorType>4</AttackVectorType><DurationBeforeDetection>0</DurationBeforeDetection><NaturalLangDescription>IDS_NATURAL_LANG_OAS_DETECTION_DEL|TargetName=Unconfirmed 408214.crdownload|TargetPath=C:\USERS\XADMIN\DOWNLOADS|ThreatName=EICAR test file|SourceProcessName=C:\PROGRAM FILES (X86)\GOOGLE\CHROME\APPLICATION\CHROME.EXE|ThreatType=test|TargetUserName=LAB-WIN7-02\xadmin</NaturalLangDescription><AccessRequested></AccessRequested><DetectionMessage>IDS_OAS_DEFAULT_THREAT_MESSAGE</DetectionMessage><AMCoreContentVersion>3075.0</AMCoreContentVersion></CustomFields></Event></SoftwareInfo></EPOevent>

The props.conf on the receiving indexer looks like this:

[epo]
SEDCMD-replace = s/\<29\>[^\>]*\>\n*//g
KV_MODE = xml
TIME_PREFIX = \<EPOevent.SoftwareInfo.Event.GMTTime\>
TIME_FORMAT = %Y-%m-%dT%H:%M:%S

I have also tried:

[epo]
SEDCMD-replace = s/\<29\>[^\>]*\>\n*//g
KV_MODE = xml
TIME_PREFIX = \<GMTTime\>
TIME_FORMAT = %Y-%m-%dT%H:%M:%S

Via search, the event looks like the following after SEC_CMD as parse the message:

<EPOevent><MachineInfo><MachineName>LAB-WIN7-02</MachineName><AgentGUID>{b37ff914-XXXX-xxxx-XXXX-91aa851fXXXX}</AgentGUID><IPAddress>192.xXX.xxx.102</IPAddress><OSName>Windows 7</OSName><UserName>SYSTEM</UserName><TimeZoneBias>240</TimeZoneBias><RawMACAddress>000c29xxxxxx</RawMACAddress></MachineInfo><SoftwareInfo ProductName="McAfee Endpoint Security" ProductVersion="10.5.0" ProductFamily="TVD"><CommonFields><Analyzer>ENDP_AM_1050</Analyzer><AnalyzerName>McAfee Endpoint Security</AnalyzerName><AnalyzerVersion>10.5.0</AnalyzerVersion><AnalyzerHostName>LAB-WIN7-02</AnalyzerHostName><AnalyzerEngineVersion>5900.7806</AnalyzerEngineVersion><AnalyzerDetectionMethod>On-Access Scan</AnalyzerDetectionMethod><AnalyzerDATVersion>3075.0</AnalyzerDATVersion></CommonFields><Event><EventID>1278</EventID><Severity>3</Severity><GMTTime>2017-08-18T14:48:53</GMTTime><CommonFields><ThreatCategory>av.detect</ThreatCategory><ThreatEventID>1278</ThreatEventID><ThreatSeverity>2</ThreatSeverity><ThreatName>EICAR test file</ThreatName><ThreatType>test</ThreatType><DetectedUTC>2017-08-18T14:48:53Z</DetectedUTC><ThreatActionTaken>IDS_ALERT_ACT_TAK_DEL</ThreatActionTaken><ThreatHandled>True</ThreatHandled><SourceHostName>LAB-WIN7-02</SourceHostName><SourceProcessName>C:\PROGRAM FILES (X86)\GOOGLE\CHROME\APPLICATION\CHROME.EXE</SourceProcessName><TargetHostName>LAB-WIN7-02</TargetHostName><TargetUserName>LAB-WIN7-02\xadmin</TargetUserName><TargetFileName>C:\USERS\XADMIN\DOWNLOADS\Unconfirmed 408214.crdownload</TargetFileName></CommonFields><CustomFields target="EPExtendedEventMT"><BladeName>IDS_BLADE_NAME_SPB</BladeName><AnalyzerContentCreationDate>2017-08-16T13:00:00Z</AnalyzerContentCreationDate><AnalyzerGTIQuery>False</AnalyzerGTIQuery><ThreatDetectedOnCreation>False</ThreatDetectedOnCreation><TargetName>Unconfirmed 408214.crdownload</TargetName><TargetPath>C:\USERS\XADMIN\DOWNLOADS</TargetPath><TargetHash>44d88612fea8a8f36de82e1278abb02f</TargetHash><TargetFileSize>68</TargetFileSize><TargetModifyTime>2017-08-18T14:48:53Z</TargetModifyTime><TargetAccessTime>2017-08-18T14:48:53Z</TargetAccessTime><TargetCreateTime>2017-08-18T14:48:53Z</TargetCreateTime><Cleanable>False</Cleanable><TaskName>IDS_OAS_TASK_NAME</TaskName><FirstAttemptedAction>IDS_ALERT_THACT_ATT_CLE</FirstAttemptedAction><FirstActionStatus>False</FirstActionStatus><SecondAttemptedAction>IDS_ALERT_THACT_ATT_DEL</SecondAttemptedAction><SecondActionStatus>True</SecondActionStatus><AttackVectorType>4</AttackVectorType><DurationBeforeDetection>0</DurationBeforeDetection><NaturalLangDescription>IDS_NATURAL_LANG_OAS_DETECTION_DEL|TargetName=Unconfirmed 408214.crdownload|TargetPath=C:\USERS\XADMIN\DOWNLOADS|ThreatName=EICAR test file|SourceProcessName=C:\PROGRAM FILES (X86)\GOOGLE\CHROME\APPLICATION\CHROME.EXE|ThreatType=test|TargetUserName=LAB-WIN7-02\xadmin</NaturalLangDescription><AccessRequested></AccessRequested><DetectionMessage>IDS_OAS_DEFAULT_THREAT_MESSAGE</DetectionMessage>
<AMCoreContentVersion>3075.0</AMCoreContentVersion></CustomFields></Event></SoftwareInfo></EPOevent>

The indexer is still applying a timestamp of when it receives the message verse using GMTTime. Here is a formatted view of what splunk sees, e.g. the nested XML:

EPOevent.MachineInfo.AgentGUID
    {b37ff914-83f4-4b48-8740-XXXXXXXXXX}    
EPOevent.MachineInfo.IPAddress
    192.xxx.xxx.XXX 
EPOevent.MachineInfo.MachineName
    LAB-WIN7-02 
EPOevent.MachineInfo.OSName
    Windows 7   
EPOevent.MachineInfo.RawMACAddress
    000c29fXXXXX
EPOevent.MachineInfo.TimeZoneBias
    240 
EPOevent.MachineInfo.UserName
    SYSTEM  
EPOevent.SoftwareInfo.CommonFields.Analyzer
    ENDP_AM_1050    
EPOevent.SoftwareInfo.CommonFields.AnalyzerDATVersion
    3075.0  
EPOevent.SoftwareInfo.CommonFields.AnalyzerDetectionMethod
    On-Access Scan  
EPOevent.SoftwareInfo.CommonFields.AnalyzerEngineVersion
    5900.7806   
EPOevent.SoftwareInfo.CommonFields.AnalyzerHostName
    LAB-WIN7-02 
EPOevent.SoftwareInfo.CommonFields.AnalyzerName
    McAfee Endpoint Security    
EPOevent.SoftwareInfo.CommonFields.AnalyzerVersion
    10.5.0  
EPOevent.SoftwareInfo.Event.CommonFields.DetectedUTC
    2017-08-18T14:48:53Z    
EPOevent.SoftwareInfo.Event.CommonFields.SourceHostName
    LAB-WIN7-02 
EPOevent.SoftwareInfo.Event.CommonFields.SourceProcessName
    C:\PROGRAM FILES (X86)\GOOGLE\CHROME\APPLICATION\CHROME.EXE 
EPOevent.SoftwareInfo.Event.CommonFields.TargetFileName
    C:\USERS\XADMIN\DOWNLOADS\Unconfirmed 408214.crdownload 
EPOevent.SoftwareInfo.Event.CommonFields.TargetHostName
    LAB-WIN7-02 
EPOevent.SoftwareInfo.Event.CommonFields.TargetUserName
    LAB-WIN7-02\xadmin  
EPOevent.SoftwareInfo.Event.CommonFields.ThreatActionTaken
    IDS_ALERT_ACT_TAK_DEL   
EPOevent.SoftwareInfo.Event.CommonFields.ThreatCategory
    av.detect   
EPOevent.SoftwareInfo.Event.CommonFields.ThreatEventID
    1278    
EPOevent.SoftwareInfo.Event.CommonFields.ThreatHandled
    True    
EPOevent.SoftwareInfo.Event.CommonFields.ThreatName
    EICAR test file 
EPOevent.SoftwareInfo.Event.CommonFields.ThreatSeverity
    2   
EPOevent.SoftwareInfo.Event.CommonFields.ThreatType
    test    
EPOevent.SoftwareInfo.Event.CustomFields.AMCoreContentVersion
    3075.0  
EPOevent.SoftwareInfo.Event.CustomFields.AnalyzerContentCreationDate
    2017-08-16T13:00:00Z    
EPOevent.SoftwareInfo.Event.CustomFields.AnalyzerGTIQuery
    False   
EPOevent.SoftwareInfo.Event.CustomFields.AttackVectorType
    4   
EPOevent.SoftwareInfo.Event.CustomFields.BladeName
    IDS_BLADE_NAME_SPB  
EPOevent.SoftwareInfo.Event.CustomFields.Cleanable
    False   
EPOevent.SoftwareInfo.Event.CustomFields.DetectionMessage
    IDS_OAS_DEFAULT_THREAT_MESSAGE  
EPOevent.SoftwareInfo.Event.CustomFields.DurationBeforeDetection
    0   
EPOevent.SoftwareInfo.Event.CustomFields.FirstActionStatus
    False   
EPOevent.SoftwareInfo.Event.CustomFields.FirstAttemptedAction
    IDS_ALERT_THACT_ATT_CLE 
EPOevent.SoftwareInfo.Event.CustomFields.NaturalLangDescription
    IDS_NATURAL_LANG_OAS_DETECTION_DEL|TargetName=Unconfirmed 408214.crdownload|TargetPath=C:\USERS\XADMIN\DOWNLOADS|ThreatName=EICAR test file|SourceProcessName=C:\PROGRAM FILES (X86)\GOOGLE\CHROME\APPLICATION\CHROME.EXE|ThreatType=test|TargetUserName=LAB-WIN7-02\xadmin 
EPOevent.SoftwareInfo.Event.CustomFields.SecondActionStatus
    True    
EPOevent.SoftwareInfo.Event.CustomFields.SecondAttemptedAction
    IDS_ALERT_THACT_ATT_DEL 
EPOevent.SoftwareInfo.Event.CustomFields.TargetAccessTime
    2017-08-18T14:48:53Z    
EPOevent.SoftwareInfo.Event.CustomFields.TargetCreateTime
    2017-08-18T14:48:53Z    
EPOevent.SoftwareInfo.Event.CustomFields.TargetFileSize
    68  
EPOevent.SoftwareInfo.Event.CustomFields.TargetHash
    44d88612fea8a8f36de82e1278abb02f    
EPOevent.SoftwareInfo.Event.CustomFields.TargetModifyTime
    2017-08-18T14:48:53Z    
EPOevent.SoftwareInfo.Event.CustomFields.TargetName
    Unconfirmed 408214.crdownload   
EPOevent.SoftwareInfo.Event.CustomFields.TargetPath
    C:\USERS\XADMIN\DOWNLOADS   
EPOevent.SoftwareInfo.Event.CustomFields.TaskName
    IDS_OAS_TASK_NAME   
EPOevent.SoftwareInfo.Event.CustomFields.ThreatDetectedOnCreation
    False   
EPOevent.SoftwareInfo.Event.CustomFields{@target}
    EPExtendedEventMT   
EPOevent.SoftwareInfo.Event.EventID
    1278    
EPOevent.SoftwareInfo.Event.GMTTime
    2017-08-18T14:48:53 
EPOevent.SoftwareInfo.Event.Severity
    3   
EPOevent.SoftwareInfo{@ProductFamily}
    TVD 
EPOevent.SoftwareInfo{@ProductName}
    McAfee Endpoint Security    
EPOevent.SoftwareInfo{@ProductVersion}
    10.5.0  
timestamp
    none    
Time            
_time   
    2017-08-18T10:50:32.000-04:00   
Default 
host
    lab-epo 
index
    main    
linecount
    1   
punct
    <><><>--</><>{----}</><>...</><>_</><></><></><></  
source
    /opt/xml/events.txt     
sourcetype
    epo
splunk_server
    lab-splunk-01

[richgalloway · Accepted solution]

Try without escape chars.

[epo]
SEDCMD-replace = s/\<29\>[^\>]*\>\n*//g
KV_MODE = xml
TIME_PREFIX = <GMTTime>
TIME_FORMAT = %Y-%m-%dT%H:%M:%S