Getting Data In

Is there a better way to parse DTS Compliant IAS logs (xml format)?

pjaguilarjr
New Member

DTS Compliant IAS logs are IAS logs wrapped in XML. Additional info: http://technet.microsoft.com/en-us/library/cc771748(v=ws.10).asp

I can extract field names at search time with |xmlkv but I that's not permanent. I've tried putting

KV_MODE=xml

into props.conf a few different ways and confirmed with btool, but apparently this format isn't true xml so splunk is ignoring it. Any suggestions? Here's a sample event:

<Timestamp data_type="4">12/22/2009 15:06:56.609</Timestamp> 
                <Computer-Name data_type="1">NAP-IAS2</Computer-Name>
                <Event-Source data_type="1">IAS</Event-Source>
                <Acct-Session-Id data_type="2">B3BA359F48CEDE4E9F78E5B3158F3B877E744D735B83CA01</Acct-Session-Id>
                <Class data_type="1">311 1 2001:4898:b0:3007:492e:957a:d44d:7093 12/16/2009 04:32:04 145361</Class>
                <MS-Extended-Quarantine-State data_type="0">0</MS-Extended-Quarantine-State>
                <MS-Quarantine-State data_type="0">0</MS-Quarantine-State>
                <Client-IPv6-Address data_type="5">2001:4898:b0:3007:6cc0:9514:d2ff:cdcf</Client-IPv6-Address>
                <Client-Vendor data_type="0">0</Client-Vendor>
                <Client-Friendly-Name data_type="1">NAP-HRA2</Client-Friendly-Name>
                <Proxy-Policy-Name data_type="1">HRA</Proxy-Policy-Name>
                <Provider-Type data_type="0">1</Provider-Type>
                <Quarantine-Session-Id data_type="1">{9F35BAB3-CE48-4EDE-9F78-E5B3158F3B87} - 2009-12-22 23:06:53.319Z</Quarantine-Session-Id>
                <Machine-Inventory data_type="1">6.1.7600 0.0 x86 Workstation</Machine-Inventory>
                <Fully-Qualified-Machine-Name data_type="1">CONTOSO\CLIENT1</Fully-Qualified-Machine-Name>
                <Authentication-Type data_type="0">7</Authentication-Type>
                <System-Health-Result data_type="1">Windows Security Health Validator:Compliant:No Data:None[]:(0x0 - ):(0x0 - ):(0x0 - ):(0x0 - ):(0x0 - ):(0x0 - ):(0x0 - ):(0x0 - )</System-Health-Result>
                <System-Health-ResultEx data_type="1">
                                <SHV-Name data_type="1">Windows Security Health Validator</SHV-Name>
                                <Config-ID data_type="0">0</Config-ID>
                                <Config-Friendly-Name data_type="1"></Config-Friendly-Name>
                                <Health-Result data_type="1">Compliant</Health-Result>
                                <Extended-Isolation-State data_type="1">No Data</Extended-Isolation-State>
                                <Failure-Category data_type="1">None</Failure-Category>
                                <Failure-Category-String data_type="1"></Failure-Category-String>
                                <Compliance-Results data_type="1"></Compliance-Results>
                </System-Health-ResultEx>
                <NP-Policy-Name data_type="1">ias2-HRA-NAPSTIR-Red-Compliant</NP-Policy-Name>
                <Quarantine-Update-Non-Compliant data_type="0">0</Quarantine-Update-Non-Compliant>
                <Framed-Protocol data_type="0">1</Framed-Protocol>
                <Service-Type data_type="0">2</Service-Type>
                <Packet-Type data_type="0">2</Packet-Type>
                <Reason-Code data_type="0">0</Reason-Code>
</Event>
0 Karma

hburton
Explorer

I'm seeing the same thing. 

props.conf looks like.

[DTS_Compliant_IAS]
BREAK_ONLY_BEFORE = <Event>
NO_BINARY_CHECK = 1
TIME_FORMAT = %m/%d/%Y %H:%M:%S.%3N
TIME_PREFIX = <Timestamp data_type="4">
pulldown_type = 1
KV_MODE=xml
 
inputs.conf looks like
[monitor://C:\Windows\System32\LogFiles\IN*.log]
sourcetype = DTS_Compliant_IAS
index = radius
disabled = 0
 
If I search with |xmlkv my field are extracted, same as original poster. Otherwise, it's just the full xml string.
 
 
0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

you should use KV_MODE on search time extractions and INDEXED_EXTRACTIONS on index time extractions.

https://docs.splunk.com/Documentation/Splunk/7.3.3/Admin/Propsconf

r. Ismo

hburton
Explorer

Thank you soutamo.

Edited Reply:

Now that I know the difference (thanks again. I'm brand new here) I see that it doesn't I considered writing a small python file monitor to convert the xml to json, and I may do that, but it looks like the Splunk proper way to do it is to use the KV_MODE=xml on a config file at the search head. We are using Cloud as our search head, so I don't know how to access my props.conf. Is there a way to do the same thing in settings somehow?

 

 

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

especially in cloud environment you must have your own test environment for do data on boarding tests. Install free version and use trial license or request developer license from Splunk. After you have do your tests successfully you should create apps for those props.conf etc and do service request to Splunk to install that app to your Splunk cloud environment. There are instructions how you show create that app etc. 

But as I said, you must have your own test instance where you develop these before you could add those to cloud.

r. Ismo

microwavelunch1
Engager

Did you ever resolve this? I have the same issue, despite putting KV_MODE=xml on both UF and indexer, the events do not get parsed as XML, but if you do a search time parse it works perfectly...

SplunkerPaul
Engager

You need to add KV_MODE=xml  in props.conf on your search head. This is a search time field extraction configuration.

0 Karma

norbert_hamel
Communicator

Hi,

I have checked this setting in props.conf

[DTS_Compliant_IAS]
BREAK_ONLY_BEFORE = <Event>
NO_BINARY_CHECK = 1
TIME_FORMAT = %m/%d/%Y %H:%M:%S.%3N
TIME_PREFIX = <Timestamp data_type="4">
pulldown_type = 1
KV_MODE=xml

Additionally, your example is missing the opening tag, so I have added this to the test log.

Using this settings the search for sourcetype=DTS_Compliant_IAS extracts all fields for me.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...