Splunk Search

Speeding up XML searches

bhiley
Explorer

It seems I need to use 'xmlkvrecursive' to properly parse XML log files where the tags may contain many attributes. However this parsing is during the search, which is consequently v. slow.
How can I parse (using xmlkvrecursive or similar) at index time - then search on the tag or attribute names via the indexes ?

Example of current search :-

index=test1 sourcetype="mmsgw" | xmlkvrecursive | search ActivityLogRecord_Common-ServerID="mmsgw1.xxx.com" ActivityLogRecord_Common-UserID="999999999@h1.xxx.com"

A single data record typically looks like :-



















































































Tags (1)

lguinn2
Legend

I don't think there is any way to parse XML prior to indexing.

However, what might make your search run more quickly: does the value "mmsgw1.xxx.com" appear anywhere in the text other than the ActivityLogRecord_Common-ServerID? If this value does NOT appear anywhere else, then you could simply search

index=test1 sourcetype="mmsgw" mmsgw1.xxx.com

for example. Once you have retrieved only the needed events, then you could apply the xmlkvrecursive. This would probably be much faster, if you can eliminate a significant number of events in the initial search. And perhaps you don't need to parse the XML at all -- you only need to create the fields from the XML if you want to run searches, statistics or reports on the fields that are created by xmlkvrecursive.

Based on the example event, the following search is also valid

index=test1 sourcetype=mmsgw serverid="mmsgw1.xxx.com" userid="999999999"

and also doesn't appear to require the XML to be parsed. But of course, I don't really understand the data...

0 Karma

bhiley
Explorer

Seems slightly counter-intuitive to me to index before you parse but then I'm a newbie with Splunk. Your method certainly speeds up the search hugely. Many thanks.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...