Splunk Search

Help extracting multiline events using rex

jambajuice
Communicator

We have events that look like this:

edit 4
    set srcintf "port1"
    set dstintf "port2"
        set srcaddr "0.0.0.0" "127.0.0.0" "169.254.0.0" "172.16.0.0" "192.0.2.0" "192.168.0.0" "255.255.255.255"             
        set dstaddr "ANY"             
    set schedule "always"
        set service "ANY"             
    set logtraffic enable
    set comments "Blocks inbound traffic from illegal networks"
next
edit 56
    set srcintf "port1"
    set dstintf "port1"
        set srcaddr "0.0.0.0" "127.0.0.0" "169.254.0.0" "172.16.0.0" "192.0.2.0" "192.168.0.0" "255.255.255.255"             
        set dstaddr "ANY"             
    set schedule "always"
        set service "ANY"             
    set logtraffic enable
next
edit 1
    set srcintf "port1"
    set dstintf "port2"
        set srcaddr "ANY"             
        set dstaddr "jnetwebcluster_VIP"             
    set action accept
    set utm-status enable
    set comments "Permit inbound ICMP to JNET DMZ"
    set schedule "always"
        set service "Allowed-ICMP"             
    set av-profile "strict"
    set ips-sensor "all_default_pass"
    set dlp-sensor "Credit-Card"
    set profile-protocol-options "strict"
    set logtraffic enable
next

I am trying to extract each individual rule as a separate event using rex. I've tried the following:

rex field=_raw max_match=400 "(?msi)edit\s(?P<rule>.*next)"  
rex field=_raw max_match=400 "(?msi)edit\s(?P<rule>.*)next"  
rex field=_raw max_match=400 "(?msi)edit\s(?P<rule>.*)(?=next)"  
rex field=_raw max_match=400 "(?msi)edit\s(?P<rule>.*)\n\s+next"  

Each one of these extractions will put all of the individual rules into a single field. It won't terminate at the "next" line that ends each individual rule.

What am I doing wrong here?

Thx.

Tags (1)

gkanapathy
Splunk Employee
Splunk Employee

For your current data, you need to use a non-greedy regex match:

(?msi)edit\s(?P<rule>.*?)\vnext

is probably what you need. Note I use .*? instead of .*.


But it seems that you would probably be a lot better off breaking this at index time with:

[mysourcetype]
SHOULD_LINEMERGE = false
LINE_BREAKER = (?i)([\r\n]+)(?=edit)

Or (less efficiently but maybe easier to understand):

[mysourcetype]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = (?i)^edit

I feel like I've answered this question before.

Ron_Naken
Splunk Employee
Splunk Employee

From what I can gather, you would be best off adding an entry to props.conf to allow Splunk to recognize each entry as a single event. You could do that like this:

$SPLUNK_HOME/etc/system/local/PROPS.CONF:

[mysourcetype]
SHOULD_LINEMERGE = true
MUST_BREAK_AFTER =^next$

This won't affect existing data, but it will allow Splunk to recognize each edit clause as a single entry for new data.

gkanapathy
Splunk Employee
Splunk Employee

much better would be to use SHOULD_LINEMERGE = false and LINE_BREAKER. See my response.

0 Karma

Ayn
Legend

Is there a specific reason for wanting to use rex for this? Otherwise you have transaction which is made just for grouping events together like this.

<yoursearch> | transaction startswith="edit" endswith="next"

More information on transaction: http://www.splunk.com/base/Documentation/latest/SearchReference/Transaction

0 Karma

jambajuice
Communicator

I thought transaction was for grouping multiple events together, not for expanding a single event into multiple fields/events.

I tried using your example and it returns no results. The problem is that there are a number of nested edit/next statements within each policy. I've got the event breaking configured to break the config file into events where the edit/next lines are at the start of the line. This creates a single event for each major config section. I need to break one of those sections into individual events.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...