Splunk Search

Trouble splitting multi-line events

muppetlegs
Engager

I have some suricata stats logs which are in the following format:

------------------------------------------------------------------
Date: 9/26/2012 -- 16:57:53 (uptime: 0d, 00h 00m 24s)
------------------------------------------------------------------
Counter                   | TM Name                  | Value
------------------------------------------------------------------
tcp.sessions              | Detect                   | 2932
tcp.blah                  | Whatever                 | 42
...
string.string             | string                   | 0
------------------------------------------------------------------

I'm trying to :

  1. extract four values: Counter, Module(for TM Name), Value, and Date
  2. have the Counter, TM Name, and Value for each line within the event associate only with what's on its own line
  3. except to take on the date value of the entire event

...but I'm not getting it done

My props.conf looks like:

[suricata_stats]
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE_DATE = false
BREAK_ONLY_BEFORE = \-*\r\nDate
EXTRACT-counter = ^(?<counter>[a-z]+\.[a-z]+)\s+|
EXTRACT-module = ^[a-z]+\.[a-z]+\s+|\s+(?<module>[^ ])\s+|
EXTRACT-value = ^[a-z]+\.[a-z]+\s+|\s+[^ ])\s+|\s+(?P<value>.+)

I know the regex is .. messy, but the results I'm getting are not as intended.

With sample data containing what i intend to be 5 results, I get 6.

One event is :
--------------------------------------------------------------------------

The next five events begin with "Date" and display all of the subsequent lines.

My extractions are also wrong:

  1. the only value for "Counter" is "tcp.sessions" - no "tcp.blah", or others.
  2. the only value for "Module" is "9" (from after "Date:")
  3. i have five values for "Value", but each value is a multi-line value containing everything from "9/26/2012" down to "string.string"

Where do we start?

Tags (1)

dazole
New Member

I'm working on the same thing, actually. I'm curious if you've managed to get a good setup for this. Here's what I have in my props.conf:

 [suricata_stats_log2]
BREAK_ONLY_BEFORE = Date:
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = False
TIME_PREFIX = Date:
pulldown_type = 1
EXTRACT-Counter = (?i)(?P<Counter>[^ ]+)\s+\|\s+\w+\s+\|\s+\d+
EXTRACT-TM_Name = (?i)\..*?   \| (?P<TM_Name>\w+)(?= )
EXTRACT-Value = (?i) .*?         \| (?P<Value>\d+)

Each line (tcp.sessions | Detect | 2932, etc) is it's own event with the correct timestamp. However, I'm not sure I like this particular solution yet. 272 actual stats turn into over 27000 events in Splunk! I'm still futzing with it to try to find something I like.

I've tried multikv and either it didn't work at all, or I was totally broken in my search implementation of it, heh.

0 Karma

emiller42
Motivator

There is the multikv command to parse out tabular data at search-time, which would likely be easier than trying to do that in props.conf. But there is still the issue of getting the event in correctly in the first place, which I'm not sure about...

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...