Splunk Search

Field splitting on structured data

cdstealer
Contributor

Hi All, I'm struggling with a data input from the EMC Recoverpoint devices. I may be making things hard for myself, but I'm sure someone will have come across this scenario on their travels 😉

An event looks like this:

SITE_VPLEX: 
    RPAs: 
      RPA 1: 
        Version: 4.1.SP1.P1(h.167)
        WAN IP: 000.000.000.000
        RPA LAN IPv4: 000.000.000.000
        RPA LAN IPv6:N/A
        iSCSI interface IPs: None
        Interfaces: 
          Type: FC
          Initiator ID: 50012481006bexxx
          Type: FC
          Initiator ID: 50012481006bexxx
          Type: FC
          Initiator ID: 50012481006bexxx
          Type: FC
          Initiator ID: 50012481006bexxx
        Hardware details: 
          Hardware type: Intel Corporation S2600GZ GEN5
          Adapter type: 2564
          Vendor: Intel Corporation
          Hardware Serial ID: FC6RP133000229_00000000002_FFF
          Hardware Platform: Intel Corporation S2600GZ GEN5
          Amount of memory: 16269416 KB
          Number of CPUs: 12
      RPA 2: 
        Version: 4.1.SP1.P1(h.167)
        WAN IP: 000.000.000.000
        RPA LAN IPv4: 000.000.000.000
        RPA LAN IPv6:N/A
        iSCSI interface IPs: None
        Interfaces: 
          Type: FC
          Initiator ID: 50012481006bdxxx
          Type: FC
          Initiator ID: 50012481006bdxxx
          Type: FC
          Initiator ID: 50012481006bdxxx
          Type: FC
          Initiator ID: 50012481006bdxxx
        Hardware details: 
          Hardware type: Intel Corporation S2600GZ GEN5
          Adapter type: 2564
          Vendor: Intel Corporation
          Hardware Serial ID: FC6RP133000135_0000000000_FFF
          Hardware Platform: Intel Corporation S2600GZ GEN5
          Amount of memory: 16269416 KB
          Number of CPUs: 12

I have the fields extracting without issue, but the issue I have is each field needs to belong to either RPA1 or RPA2.

Thanks in advance
Steve

0 Karma
1 Solution

woodcock
Esteemed Legend

I would handle this by splitting each one of these into 2 events on the way in.

View solution in original post

woodcock
Esteemed Legend

I would handle this by splitting each one of these into 2 events on the way in.

cdstealer
Contributor

Hi,
Thanks. I did consider doing this before I posted. However, it does seem to fit better and is simple to implement. I've had to apply a little extra config to get the data how it's needed. But it's all working 🙂 So the props.conf I'm now using on the heavyforwarder is below, the props above lives on the search head for search time extraction:

[emc_sourcetype]
DATETIME_CONFIG=CURRENT
SHOULD_LINEMERGE=true
BREAK_ONLY_BEFORE=^\s+RPA\s\d:
NO_BINARY_CHECK=true
disabled=false
TRUNCATE = 999999
MAX_EVENTS = 15000
SEDCMD-topline1 = s/string//g
SEDCMD-topline2 = s/string//g
SEDCMD-topline3 = s/string//g
SEDCMD-topline4 = s/string//g
SEDCMD-RPA1 = s/RPA 1:/RPA: 1/g
SEDCMD-RPA2 = s/RPA 2:/RPA: 2/g

So this splits the event at RPA X:, removes the top 4 lines as these are now nolonger required and then replaces the RPA id for auto extraction. Job done 🙂

Thanks to you both for your help.
Steve

woodcock
Esteemed Legend

You should actually get rid of this:

SHOULD_LINEMERGE=true
BREAK_ONLY_BEFORE=^\s+RPA\s\d:

And use this instead:

SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n\s]+)RPA\s*\d:

The latter is FAR more efficient.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi cdstealer,
you should try with two regex like the following (for Initiator_ID field both in RPA1 and RPA2):

(?ms)(RPA\s1:).+?Initiator\sID:\s(?<RPA1Initiator>[^ ]*)
(?ms)(RPA\s2:).+?Initiator\sID:\s(?<RPA2Initiator>[^ ]*)

see an example in https://regex101.com/r/OY794T/1

Bye.
Giuseppe

0 Karma

cdstealer
Contributor

Hi Giuseppe,
Thanks for the reply. Apologies, I probably should have stated how I'm extracting the fields.

transforms:

[emc_sourcetype]
DELIMS = "\"{\n}\"", ":"
FORMAT = $1::$2
MV_ADD = true

props:

[emc_sourcetype]
KV_MODE = none
KV_TRIM_SPACES = true
REPORT-extractions = emc_sourcetype

Thanks

0 Karma

gcusello
SplunkTrust
SplunkTrust

Sorry I didn't understand your need: can you explain again?
with the regex I gave, you can extract both the Initiator_ID field for RPA1 and RPA2.
Bye.
Giuseppe

0 Karma

cdstealer
Contributor

Hi, no problem 🙂 So I'm trying to make this dynamic so that if the data changes, the extractions still work and also any new fields get extracted without having to manually create them. So the props/transforms I gave extract all the fields almost as I need. This also includes new fields. So what I need is a way to get splunk to enrich the fields so that we know which field belongs to which RPA.
In the example above, I have 8 "Initiator ID" values which anyone searching this data will have no idea what RPA they belong to. So I want splunk to assign the 4 under the RPA1 heading and the second 4 to belong to RPA2. So if a user searches for RPA1, they only get the fields extracted under the RPA1 heading. I hope I'm explaining this ok.

Thanks
Steve

0 Karma

gcusello
SplunkTrust
SplunkTrust

Ok, Using a regex like the one I sent, you'll have different Initiator_ID field for RPA1 and RPA2.
If you need all the four values of Initiator_ID both for RPA1 and RPA2, you could use a regex like this:

(?ms)(RPA\s1:).+?Initiator\sID:\s(?<RPA1Initiator1>[^ ]*)\s+Type:\s.+?Initiator\sID:\s(?<RPA1Initiator2>[^ ]*)\s+Type:\s.+?Initiator\sID:\s(?<RPA1Initiator3>[^ ]*)\s+Type:\s.+?Initiator\sID:\s(?<RPA1Initiator4>[^ ]*)

Bye.
Giuseppe

0 Karma

cdstealer
Contributor

Hi, Thanks. The regex way is what I was wanting to avoid as it means any changes to the event will break it. But if I have no other option, then I'll have to go down that road 😞

Thanks again for your help.
Steve

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...