Dashboards & Visualizations

TIME_FORMAT and XML

colinj
Path Finder

Howdy,

I'm running in to a problem getting some XML to parse in to events properly. The log has multi-line entries as you might expect and in the XML are date strings that I want Splunk to ignore. So, each log entry begins with the following time stamp (which I want to use):

[03/12/12 16:15:30.103]: <Some log data goes here>
<possibly followed by a bunch of XML>
[03/12/12 16:15:30.112]: <Some more log data goes here>

The entries in the log file might only a single line long or might be as long as several hundred lines of XML before the next so-called real time spamp. Part of the XML data being generated are time strings like this: 20120312161445.247Z, which I don't want.

What I want is for all of the lines between one time stamp and the next to be grouped in to a single event.

Here's what I have in my props.conf file:

TIME_PREFIX = \[
TIME_FORMAT = %m/%d/%y %H:%M:%S.%3N

However Splunk is still splitting on the other time stamps within the XML. So, what am I doing wrong?

0 Karma
1 Solution

kristian_kolb
Ultra Champion

Are you editing the correct props.conf? (i.e. where the parsing occurs).
If you have a Universal Forwarder -> Indexer OR Lightweight Forwarder -> Indexer setup, the props.conf to edit is on the Indexer.

If you are using a full/heavy forwarder, the settings should go there.

Apart from that your settings look good. You could possibly further qualify your TIME_PREFIX regex with (if tour 'real' timestamps are the actually in the beginning of the line);

TIME_PREFIX=^\[

How have you configured linebreaking? I'd recommend that you use

SHOULD_LINEMERGE=false
LINE_BREAKER = ([\r\n]+)\[\d\d

which explicitly tells splunk to break events after a newline followed by the beginning of a 'real' timestamp.

Hope this helps. If not, please give us more of your props.conf, and perhaps some more sample events.

/Kristian

View solution in original post

kristian_kolb
Ultra Champion

Are you editing the correct props.conf? (i.e. where the parsing occurs).
If you have a Universal Forwarder -> Indexer OR Lightweight Forwarder -> Indexer setup, the props.conf to edit is on the Indexer.

If you are using a full/heavy forwarder, the settings should go there.

Apart from that your settings look good. You could possibly further qualify your TIME_PREFIX regex with (if tour 'real' timestamps are the actually in the beginning of the line);

TIME_PREFIX=^\[

How have you configured linebreaking? I'd recommend that you use

SHOULD_LINEMERGE=false
LINE_BREAKER = ([\r\n]+)\[\d\d

which explicitly tells splunk to break events after a newline followed by the beginning of a 'real' timestamp.

Hope this helps. If not, please give us more of your props.conf, and perhaps some more sample events.

/Kristian

colinj
Path Finder

Thanks! That did the trick.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...