Getting Data In

How to make multivalue fields parse in props.conf and transforms.conf?

reswob4
Builder

So I've gotten stuck trying to get multivalue field working.

I have, in general, two type of events.

The first:

Jan 20 16:46:47 1.11.2.23 xsuite[30512]: Source IP: 1.11.2.2, Nat/Proxy IP: 1.11.2.2, User: CN=Bugs Bunny+UID=1111111,OU=Warner Bros,O=Acme,C=US, Transaction: connection, Address: 1.11.2.1, Device name: car.toon.LOCAL, Port: 3389, Access/Protocol: RDP, Service/App: - -, Details: xsuite[30512]: CN=Bugs Bunny+UID=1111111,OU=Warner Bros,O=Acme,C=US connected to 1.11.2.1:3389; Idle time out: 900;

and the second:

Jan 20 16:35:32 1.11.2.23 gkpsyslog[11895]: Private IP: , Public IP: , Nat/Proxy IP: 1.11.2.2, User: unknown, Transaction: login, Address: - -, Device Name: - -, User Group: --Port: - -, Access/Protocol: - -, Service/App: - -, Details: Message 18007:  Email alert sent to user Bugs Bunny+UID=1111111,OU=Warner Bros,O=Acme,C=US:  Message 18016:  User Bugs Bunny+UID=1111111,OU=Warner Bros,O=Acme,C=US has logged into the CA Privileged Access Manager appliance car.toon.local.

The following search works pretty well parsing all the fields including the multivalue fields:

index="xceedium" | rex field=_raw "User: (?<user>[^(|^,]*)()" | rex field=_raw "Details: (?<Details>.+)" | rex max_match=10 field=Details "Message (?<message_id>\d+):" | rex max_match=10 field=Details "Message \d+:(?<message>.+?)(:|$)" | rex field=_raw "Service/App: (?<ServiceApp>[^,]*)" | rex field=_raw "Access/Protocol: (?<AccessProtocol>[^,]*)" | rex field=_raw "User Group: (?<group>.+?)Port" | rex field=_raw "Port: (?<port>[^,]*)" | rex field=_raw "Transaction: (?<Transaction>[^,]*),\s+Address: (?<Address>[^,]*),\s+Device (name|Name): (?<DeviceName>[^,]*)" | rex field=_raw "Private IP: (?<priv_ip>[^,]*),\s+Public IP: (?<pub_ip>[^,]*),\s+Nat/Proxy IP: (?<Nat_ProxyIP>[^,]*)" | rex field=_raw "Source IP: (?<src>[^,]*),"

but I can't get the props.conf and transforms.conf correctly set up to do the same for the multivalue fields. The multivalue fields don't parse out.

I've been looking at the following:

https://answers.splunk.com/answers/112311/multi-value-field-extraction.html
http://docs.splunk.com/Documentation/Splunk/6.0/Knowledge/Configureadvancedextractionswithfieldtrans...

the source is udp:22514 and the sourcetype is xml. The logs are sent to a Heavy Forwarder, which sends to my two indexes.

Currently, I have the props.conf and transforms.conf configured on the search head.

here is the section of my props.conf for this:

[source::udp:22514]
EXTRACT-user = User: (?<user>[^(|^,]*)()
EXTRACT-ServiceApp = Service/App: (?<ServiceApp>[^,]*)
EXTRACT-AccessProtocol = Access/Protocol: (?<AccessProtocol>[^,]*)
EXTRACT-group = User Group: (?<group>.+?)Port
EXTRACT-port = Port: (?<port>[^,]*)
EXTRACT-Transaction,Address,DeviceName = Transaction: (?<Transaction>[^,]*),\s+Address: (?<Address>[^,]*),\s+Device (name|Name): (?<DeviceName>[^,]*)
EXTRACT-priv_ip,pub_ip = Private IP: (?<priv_ip>[^,]*),
EXTRACT-pub_ip = Public IP: (?<pub_ip>[^,]*),
EXTRACT-nat_proxyIP = Nat/Proxy IP: (?<nat_proxyIP>[^,]*)
EXTRACT-src = Source IP: (?<src>[^,]*),
#EXTRACT-Details = Details: (?<Details>.+)
TRANSFORMS-Details = Details
TRANSFORMS-message_id = message_id
TRANSFORMS-message = message

and here is the section of my transforms.conf:

[Details]
REGEX = Details: (?<Details>.+)
FORMAT = Details::$1

[message_id]
SOURCE_KEY = Details
REGEX = Message (?<message_id>\d+):
FORMAT = message_id::$1
MV_ADD = true

[message]
SOURCE_KEY = Details
REGEX = Message \d+:(?<message>.+?)(:|$)
FORMAT = message::$1
MV_ADD = true

Do I need to create a fields.conf?

https://docs.splunk.com/Documentation/Splunk/6.5.1/Knowledge/ConfigureSplunktoparsemulti-valuefields

If so, can I get some help with the syntax? The example they give doesn't mesh in my mind with what I'm dealing with....

Thanks

0 Karma
1 Solution

somesoni2
SplunkTrust
SplunkTrust

Try removing SOURCE_KEY from both the MV field extraction. It should be able to extract based off raw data as well.

View solution in original post

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Try removing SOURCE_KEY from both the MV field extraction. It should be able to extract based off raw data as well.

0 Karma

reswob4
Builder

Changed to

 [message_id]
 REGEX = Message (?<message_id>\d+):
 FORMAT = message_id::$1
 MV_ADD = true

 [message]
 REGEX = Message \d+:(?<message>.+?)(:|$)
 FORMAT = message::$1
 MV_ADD = true

and to

 [message_id]
 SOURCE_KEY = _raw
 REGEX = Message (?<message_id>\d+):
 FORMAT = message_id::$1
 MV_ADD = true

 [message]
 SOURCE_KEY = _raw
 REGEX = Message \d+:(?<message>.+?)(:|$)
 FORMAT = message::$1
 MV_ADD = true

neither worked.

At least right away.....

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Also, change the TRANSFORM- to REPORT-. This is a search-time field extraction and should be REPORT not TRANSFORM which is for index-time field extraction. (in props.conf)

0 Karma

reswob4
Builder

SWEET.

That was it.

Final config for props.conf:

props.conf:

 [source::udp:22514]
 EXTRACT-user = User: (?[^(|^,]*)()
 EXTRACT-ServiceApp = Service/App: (?[^,]*)
 EXTRACT-AccessProtocol = Access/Protocol: (?[^,]*)
 EXTRACT-group = User Group: (?.+?)Port
 EXTRACT-port = Port: (?[^,]*)
 EXTRACT-Transaction,Address,DeviceName = Transaction: (?[^,]*),\s+Address: (?[^,]*),\s+Device (name|Name): (?[^,]*)
 EXTRACT-priv_ip,pub_ip = Private IP: (?[^,]*),
 EXTRACT-pub_ip = Public IP: (?[^,]*),
 EXTRACT-nat_proxyIP = Nat/Proxy IP: (?[^,]*)
 EXTRACT-src = Source IP: (?[^,]*),
 #EXTRACT-Details = Details: (?.+)
 REPORT-Details = Details
 REPORT-message_id = message_id
 REPORT-message = message

Final config for transforms.conf:

  [message_id]
  REGEX = Message (?\d+):
  FORMAT = message_id::$1
  MV_ADD = true

  [message]
  REGEX = Message \d+:(?.+?)(:|$)
  FORMAT = message::$1
  MV_ADD = true

Could you actually ANSWER my question so I can mark this as answered and you get cred?

Thanks

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...