Getting Data In

Single Line XML Log File

jheilman
Explorer

We have a system called Metaviewer that produces XML log files. I've read several answers about indexing and trapping events from an XML file. The problem in this case is that all of the events are combined into a single line. The only time a line break even enters the picture is when one of the <SQL> elements in the XML event contains a line break. Is there a way to index multiple events from a single line?

A stripped down sample log event from Metaviewer might look like this...

<?xml version="1.0" encoding="UTF-8"?><!--Log file generated byMetafile Event Logging Server--><EventLog xmlns="http://www.metafile.com/ns/eventlog/10" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><CreateDate>6/17/2010</CreateDate><CreateTime>13:20:00</CreateTime><Logger>servername</Logger><Events><Event><EventID>256</EventID><Description>Modify User</Description><Category>Audit</Category><Source>MetaViewer Enterprise</Source><SubSource>MVENT</SubSource><UserName>DOMAIN\user</UserName><UserID>562</UserID><Computer>servername</Computer><Date>06/17/2010</Date><Time>13:24:33</Time><ObjectType>User</ObjectType><Details><SQL>UPDATE UserInfo SET CMAnnTlbx = 0,CMArchMaint = 0,CMArchSearch = 0,CMDocClass = 0,CMDocForms = 0,CMMetaViews = 0,CMMoveStrg = 0,CMPgStore = 0,CMPrefetch = 0,CMPurgeDoc = 0,CMQueries = 0,CMShortcuts = 0,CMSysInfo = 0,CMTasks = 0,CMUsers = 0,CMWorkGrps = 0,GroupID = 36,Notes = NULL,SIDType = 0,SIDValue = NULL,SigCode = 0,Attribs = &apos;&lt;Variables&gt;&lt;/Variables&gt;&apos;,InactivityTO = 0,NameFirst = NULL,NameLast = NULL,NameUser = &apos;UserName&apos; WHERE UserID =575</SQL><UserID>575</UserID><UserName>UserName</UserName></Details></Event></Events></EventLog>

In a full log, multiple <Event> elements would be slapped together on the same line. The <SQL> element may contain line breaks, but that's the only time you'll ever see them. So the challenge is to index each of the <Event>s that exist on the same line, while still trapping multiple lines when necessary.

I can't even seem to get Splunk to index multiple events on a single line. Here's what I'm trying right now...

[metaviewer_audit_xml]
TIME_PREFIX = <Date>
MAX_TIMESTAMP_LOOKAHEAD = 200
BREAK_ONLY_BEFORE = <Event>
BREAK_ONLY_BEFORE_DATE = False
MUST_BREAK_AFTER = </Event>
SHOULD_LINEMERGE = True

I feel like I know why this doesn't work, but I haven't found another way of doing it. Any suggestions?

0 Karma
1 Solution

southeringtonp
Motivator

LINEMERGE and related settings by nature operate on single lines and try to reassemble them into multi-line events. When all your data is on one line, the assumption is that it's all part of the same event.

Instead, take a look at LINE_BREAKER, which will allow you to break events up before line merging takes place. As a bonus, it's generally faster since you can turn merging off.

Fair warning -- I haven't tested this regex -- but it should give you the general idea:

[metaviewer_audit_xml]
LINE_BREAKER = [\>\s]((?=\<Event\>))
SHOULD_LINEMERGE = False
TIME_PREFIX = <Date>
MAX_TIMESTAMP_LOOKAHEAD = 350

You can find a little more information here:
     http://www.splunk.com/base/Documentation/4.1.6/Admin/Indexmulti-lineevents

View solution in original post

southeringtonp
Motivator

LINEMERGE and related settings by nature operate on single lines and try to reassemble them into multi-line events. When all your data is on one line, the assumption is that it's all part of the same event.

Instead, take a look at LINE_BREAKER, which will allow you to break events up before line merging takes place. As a bonus, it's generally faster since you can turn merging off.

Fair warning -- I haven't tested this regex -- but it should give you the general idea:

[metaviewer_audit_xml]
LINE_BREAKER = [\>\s]((?=\<Event\>))
SHOULD_LINEMERGE = False
TIME_PREFIX = <Date>
MAX_TIMESTAMP_LOOKAHEAD = 350

You can find a little more information here:
     http://www.splunk.com/base/Documentation/4.1.6/Admin/Indexmulti-lineevents

jheilman
Explorer

This worked perfectly. Thank you!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...