Splunk Search

Transforms.conf and props.conf field extractions

Michael_Schyma1
Contributor

Hey fellow Splunker's. I am trying to figure out what i am doing wrong in the transforms.conf to create the proper field extractions. I feel like this should work. The format of my regex should be right, but i am not getting any results returned in the fields that I want to extract. Any help that can be given to fix this issue would be greatly appreachated. Thank you.

Below is an example of the transforms.conf and the props.conf files.

(Transforms.conf)

[vmware-host]
REGEX = ^(?:\S+\s){3}(?P<Hostname>\S+)(?:\S+\s){4}(?P<ANSI_Standard>[^:]+)(?:\S+\s){3}(?P<Message_ID>\S+)\s(?P<Priority>\S+)\s\'(?P<Category>\w+)\'\]\s\[(?P<Event_Type>[^\]]+)\]
FORMAT = fieldOne::$1, fieldTwo::$2

(props.conf)

[VMwareESXi]
pulldown_type = 1
TRANSFORMS = vmware-host
Tags (1)
0 Karma
1 Solution

lguinn2
Legend

Thanks for the updates. Now I can see a reason to use the transforms. But, as Bob pointed out in the comment below, you should use REPORT instead of TRANSFORM in props.conf. It is much better to build a search-time field extraction.

props.conf

[VMwareESXi]
pulldown_type = 1
REPORT-r1 = vmware-host

transforms.conf

[vmware-host]
REGEX=(\w+\s\d+\s\d+:\d+:\d+)\s(\S+)\s(\w+\s\d+\s\d+:\d+:\d+)\s(\S+)\[(\S+\s\S+)\s(\S+)\s(\S+)\'(\S+)\'\] \s*(.*)
FORMAT=date1::$1,hostname::$2, date2::$3, servicename::$4, date3::$5,hexcode::$6,reportinglevel::$7, reportingcomponent::$8,message::$9

Note that you don't want to break the lines. So if you copy-and-paste this (which is a good idea), put the lines for the REGEX and the FORMAT back together again!
I named your fields somewhat arbitrarily, but hopefully this will give you a good idea of how to finish the job.

View solution in original post

lguinn2
Legend

Thanks for the updates. Now I can see a reason to use the transforms. But, as Bob pointed out in the comment below, you should use REPORT instead of TRANSFORM in props.conf. It is much better to build a search-time field extraction.

props.conf

[VMwareESXi]
pulldown_type = 1
REPORT-r1 = vmware-host

transforms.conf

[vmware-host]
REGEX=(\w+\s\d+\s\d+:\d+:\d+)\s(\S+)\s(\w+\s\d+\s\d+:\d+:\d+)\s(\S+)\[(\S+\s\S+)\s(\S+)\s(\S+)\'(\S+)\'\] \s*(.*)
FORMAT=date1::$1,hostname::$2, date2::$3, servicename::$4, date3::$5,hexcode::$6,reportinglevel::$7, reportingcomponent::$8,message::$9

Note that you don't want to break the lines. So if you copy-and-paste this (which is a good idea), put the lines for the REGEX and the FORMAT back together again!
I named your fields somewhat arbitrarily, but hopefully this will give you a good idea of how to finish the job.

lguinn2
Legend

Bob - you are so right! I must have been brain-dead on this one! I have updated my answer to reflect your comment.

0 Karma

BobM
Builder

This should work but it is not a best practice as it extracts the field at index time. This places more load on your indexers as well as taking more space in your index. If you replace TRANSFORMS with REPORT in your props.conf this will run at search time which is more efficient.

Although switching to EXTRACT would be my preference. Your initial post was close to an extract regex.

lguinn2
Legend

I am not sure why you need the transforms.conf stanza at all, given the way you wrote your regular expression.

Why not

[VMwareESXi]
pulldown_type = 1
EXTRACT-e1 =^(?:\S+\s){3}(?P<Hostname>\S+)(?:\S+\s){4}(?P<ANSI_Standard>[^:]+)(?:\S+\s){3}(?P<Message_ID>\S+)\s(?P<Priority>\S+)\s\'(?P<Category>\w+)\'\]\s\[(?P<Event_Type>[^\]]+)\]

But I do see some strange things in the regular expression. Can you post a snippet of the ESXi log file that you are trying to use?

Michael_Schyma1
Contributor

If any additional information or logs are needed, just let me know. I will be keeping a close watch on this question.

0 Karma

Michael_Schyma1
Contributor

Thank you so much.
Via the ESXi sourcetype:

Sep 5 14:21:34 cam.company.com Sep 5 18:21:34 Vpxa: [2012-09-05 18:21:34.420 17C3DB90 verbose 'App'] Set internal stats for VM: 272 (vpxa VM id), 4181 (vpxd VM id). Is FT primary? 0
Sep 5 14:21:34 cam.company.com Sep 5 18:21:34 Vpxa: [2012-09-05 18:21:34.664 17B7AB90 verbose 'VpxaHalCnxHostagent'] Received callback in WaitForUpdatesDone
Sep 5 14:21:34 cam.company.com Sep 5 18:21:34 Vpxa: [2012-09-05 18:21:34.664 17B7AB90 verbose 'VpxaHalCnxHostagent'] [VpxaHalCnxHostagent::ProcessUpdate] Applying updates from 3675467 to 3675468 (at 3675467)

0 Karma

Michael_Schyma1
Contributor

I removed the parts of the configuration files that were not necessary.

0 Karma

landen99
Motivator

Keep in mind that every set of parenthesis is a capture group. $1 goes to the very first capture group, not the first named capture group.

0 Karma

Michael_Schyma1
Contributor

Thank you for responding. We are just commenting them out to try and figure out what the issue is of why we can not get the regex to work. We are trying to get the transforms.conf file to work the [vmware-host]

0 Karma

Ayn
Legend

Also, which of the settings are you referring to? I see multiple things going on, of which some parts are commented out. What are you expecting, and what's not working, specifically?

0 Karma

Ayn
Legend

Please indent code sections with 4 spaces at the beginning of the line. That way they will show correctly here on the site. Thank you.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...