Getting Data In

how to parse an xml file as 1 event

itsnotcomplicat
Engager

I have an XML file I want to bring into splunk as a single event. It is the equivalent of an Excel file. The props.conf is set as:


[excelxml]  
SHOULD_LINEMERGE = true 
TRUNCATE = 0 
MAX_EVENTS = 500000 
TIME_PREFIX = \ 
TIME_FORMAT = %d:%m:%Y:%h:%m:%s 

This does result in an event that matches the file, and it has the correct timestamp. But I also get another event with some of the row data in it.

The event I want shows all the data. The event I don't want starts at the first tag.

I suspect that some default xml processing is breaking out the row data, but I am not sure how to suppress this. Any hints appreciated.

Thanks

Tags (1)

sloshburch
Splunk Employee
Splunk Employee

I think you're going to want to use LINE_BREAKER to specify where the event should break. http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf and http://docs.splunk.com/Documentation/Splunk/latest/Data/Configureeventlinebreaking elaborates on its usage.

Would you review that and reply back with an attempted solution that defines the line break and if it works or not? We can review from there.

0 Karma

sim_tcr
Communicator

this resolved.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

Sounds like that worked. If so, you can click the link to accept this answer OR, if something else solved it, let us know what.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

On the phone you mentioned this config was on the search head and the forwarder. Is it also on the indexer? That's where the event would get processed. This might help clarify: http://docs.splunk.com/Documentation/Splunk/latest/Admin/Configurationparametersandthedatapipeline#H...

If still no good, maybe post a sanitized example of the data so we can see how if the props are good.

0 Karma

sim_tcr
Communicator

Hello Burch,

I must have said wrong on the call. The config is on the forwarder and on indexers.
Below is a sample. It has more lines similar. Around 8000. We want that whole 8000 lines as one event.

<?xml version="1.0" encoding="UTF-8" ?>
<?xml-stylesheet type='text/xsl' href='freb.xsl'?>
<!-- saved from url=(0014)about:internet -->
<failedRequest url="https://xxxxx.fmr.com:443/xxxxxx/xxxxxxx.aspx"
               siteId="21"
               appPoolId="xxx_xx"
               processId="xxxxx"
               verb="POST"
               remoteUserName=""
               userName=""
               tokenUserName="NT AUTHORITY\IUSR"
               authenticationType="anonymous"
               activityId="{00000000-0000-0000-D07B-0080010000DB}"
               failureReason="STATUS_CODE"
               statusCode="500"
               triggerStatusCode="500"
               timeTaken="765"
               xmlns:freb="http://schemas.microsoft.com/win/2006/06/iis/freb"
               >
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
 <System>
  <Provider Name="WWW Server" Guid="{3A2A4E84-4C21-4981-AE10-3FDA0D9B0F83}"/>
  <EventID>0</EventID>
  <Version>1</Version>
  <Level>5</Level>
  <Opcode>2</Opcode>
  <Keywords>0x100</Keywords>
  <TimeCreated SystemTime="2017-02-08T18:02:20.952Z"/>
  <Correlation ActivityID="{00000000-0000-0000-D07B-0080010000DB}"/>
  <Execution ProcessID="12812" ThreadID="16488"/>
  <Computer>xxxxxxx</Computer>
 </System>
 <EventData>
  <Data Name="ContextId">{00000000-0000-0000-D07B-0080010000DB}</Data>
  <Data Name="ModuleName">FailedRequestsTracingModule</Data>
  <Data Name="Notification">1</Data>
  <Data Name="fIsPostNotificationEvent">false</Data>
  <Data Name="NotificationStatus">0</Data>
 </EventData>
 <RenderingInfo Culture="en-US">
  <Opcode>NOTIFY_MODULE_END</Opcode>
  <Keywords>
   <Keyword>RequestNotifications</Keyword>
  </Keywords>
  <freb:Description Data="Notification">BEGIN_REQUEST</freb:Description>
  <freb:Description Data="NotificationStatus">NOTIFICATION_CONTINUE</freb:Description>
 </RenderingInfo>
 <ExtendedTracingInfo xmlns="http://schemas.microsoft.com/win/2004/08/events/trace">
  <EventGuid>{002E91E3-E7AE-44AB-8E07-99230FFA6ADE}</EventGuid>
 </ExtendedTracingInfo>
</Event>
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
 <System>
  <Provider Name="WWW Server" Guid="{3A2A4E84-4C21-4981-AE10-3FDA0D9B0F83}"/>
  <EventID>0</EventID>
  <Version>1</Version>
  <Level>5</Level>
  <Opcode>1</Opcode>
  <Keywords>0x100</Keywords>
  <TimeCreated SystemTime="2017-02-08T18:02:20.952Z"/>
  <Correlation ActivityID="{00000000-0000-0000-D07B-0080010000DB}"/>
  <Execution ProcessID="12812" ThreadID="16488"/>
  <Computer>xxxxxxx</Computer>
 </System>
 <EventData>
  <Data Name="ContextId">{00000000-0000-0000-D07B-0080010000DB}</Data>
  <Data Name="ModuleName">ConfigurationValidationModule</Data>
  <Data Name="Notification">1</Data>
  <Data Name="fIsPostNotification">false</Data>
 </EventData>
 <RenderingInfo Culture="en-US">
  <Opcode>NOTIFY_MODULE_START</Opcode>
  <Keywords>
   <Keyword>RequestNotifications</Keyword>
  </Keywords>
  <freb:Description Data="Notification">BEGIN_REQUEST</freb:Description>
 </RenderingInfo>
 <ExtendedTracingInfo xmlns="http://schemas.microsoft.com/win/2004/08/events/trace">
  <EventGuid>{002E91E3-E7AE-44AB-8E07-99230FFA6ADE}</EventGuid>
 </ExtendedTracingInfo>
</Event>
</failedRequest>
0 Karma

sloshburch
Splunk Employee
Splunk Employee

Just to be safe, I want to make sure you see that there's multiple stanzas of the same type (example: EventData). As such, you might need to play with the sourcetype to allow multivalue fields to retain all data.

0 Karma

sim_tcr
Communicator

I tried to do this, but sill i am not getting it as one full event, i put below in my forwarder and index props.conf

[ name ]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
CHARSET=UTF-8
disabled=false
BREAK_ONLY_BEFORE=GobblyGook
MAX_EVENTS=100000
0 Karma

sloshburch
Splunk Employee
Splunk Employee

'GobblyGook' was meant to be text that you replace and not to be implemented literally.

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

Heh. It would work anyway, as long as the file never contains that literal.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

ha ha ha ha

0 Karma

chris
Motivator

I remember reading, that setting:

BREAK_ONLY_BEFORE = GobblyGook

in props.conf will make Splunk read the entire file since GobblyGook never appears in the source file. We use this for non-XML files I'm not sure if this works with XML Files.

Chris

itsnotcomplicat
Engager

Thanks Chris
I put GobblyGook in props.conf as suggested and now it is one event! As I wanted ...

I still wonder what was causing the extra event - I have seen reference to default XML file processing in other places and wonder if that kicked in as well - but that's maybe a question for another day.

BR
Steve

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...