Splunk Search

Transforms.conf and props.conf field extractions

Michael_Schyma1
Contributor

Hey fellow Splunker's. I am trying to figure out what i am doing wrong in the transforms.conf to create the proper field extractions. I feel like this should work. The format of my regex should be right, but i am not getting any results returned in the fields that I want to extract. Any help that can be given to fix this issue would be greatly appreachated. Thank you.

Below is an example of the transforms.conf and the props.conf files.

(Transforms.conf)

[vmware-host]
REGEX = ^(?:\S+\s){3}(?P<Hostname>\S+)(?:\S+\s){4}(?P<ANSI_Standard>[^:]+)(?:\S+\s){3}(?P<Message_ID>\S+)\s(?P<Priority>\S+)\s\'(?P<Category>\w+)\'\]\s\[(?P<Event_Type>[^\]]+)\]
FORMAT = fieldOne::$1, fieldTwo::$2

(props.conf)

[VMwareESXi]
pulldown_type = 1
TRANSFORMS = vmware-host
Tags (1)
0 Karma
1 Solution

lguinn2
Legend

Thanks for the updates. Now I can see a reason to use the transforms. But, as Bob pointed out in the comment below, you should use REPORT instead of TRANSFORM in props.conf. It is much better to build a search-time field extraction.

props.conf

[VMwareESXi]
pulldown_type = 1
REPORT-r1 = vmware-host

transforms.conf

[vmware-host]
REGEX=(\w+\s\d+\s\d+:\d+:\d+)\s(\S+)\s(\w+\s\d+\s\d+:\d+:\d+)\s(\S+)\[(\S+\s\S+)\s(\S+)\s(\S+)\'(\S+)\'\] \s*(.*)
FORMAT=date1::$1,hostname::$2, date2::$3, servicename::$4, date3::$5,hexcode::$6,reportinglevel::$7, reportingcomponent::$8,message::$9

Note that you don't want to break the lines. So if you copy-and-paste this (which is a good idea), put the lines for the REGEX and the FORMAT back together again!
I named your fields somewhat arbitrarily, but hopefully this will give you a good idea of how to finish the job.

View solution in original post

lguinn2
Legend

Thanks for the updates. Now I can see a reason to use the transforms. But, as Bob pointed out in the comment below, you should use REPORT instead of TRANSFORM in props.conf. It is much better to build a search-time field extraction.

props.conf

[VMwareESXi]
pulldown_type = 1
REPORT-r1 = vmware-host

transforms.conf

[vmware-host]
REGEX=(\w+\s\d+\s\d+:\d+:\d+)\s(\S+)\s(\w+\s\d+\s\d+:\d+:\d+)\s(\S+)\[(\S+\s\S+)\s(\S+)\s(\S+)\'(\S+)\'\] \s*(.*)
FORMAT=date1::$1,hostname::$2, date2::$3, servicename::$4, date3::$5,hexcode::$6,reportinglevel::$7, reportingcomponent::$8,message::$9

Note that you don't want to break the lines. So if you copy-and-paste this (which is a good idea), put the lines for the REGEX and the FORMAT back together again!
I named your fields somewhat arbitrarily, but hopefully this will give you a good idea of how to finish the job.

lguinn2
Legend

Bob - you are so right! I must have been brain-dead on this one! I have updated my answer to reflect your comment.

0 Karma

BobM
Builder

This should work but it is not a best practice as it extracts the field at index time. This places more load on your indexers as well as taking more space in your index. If you replace TRANSFORMS with REPORT in your props.conf this will run at search time which is more efficient.

Although switching to EXTRACT would be my preference. Your initial post was close to an extract regex.

lguinn2
Legend

I am not sure why you need the transforms.conf stanza at all, given the way you wrote your regular expression.

Why not

[VMwareESXi]
pulldown_type = 1
EXTRACT-e1 =^(?:\S+\s){3}(?P<Hostname>\S+)(?:\S+\s){4}(?P<ANSI_Standard>[^:]+)(?:\S+\s){3}(?P<Message_ID>\S+)\s(?P<Priority>\S+)\s\'(?P<Category>\w+)\'\]\s\[(?P<Event_Type>[^\]]+)\]

But I do see some strange things in the regular expression. Can you post a snippet of the ESXi log file that you are trying to use?

Michael_Schyma1
Contributor

If any additional information or logs are needed, just let me know. I will be keeping a close watch on this question.

0 Karma

Michael_Schyma1
Contributor

Thank you so much.
Via the ESXi sourcetype:

Sep 5 14:21:34 cam.company.com Sep 5 18:21:34 Vpxa: [2012-09-05 18:21:34.420 17C3DB90 verbose 'App'] Set internal stats for VM: 272 (vpxa VM id), 4181 (vpxd VM id). Is FT primary? 0
Sep 5 14:21:34 cam.company.com Sep 5 18:21:34 Vpxa: [2012-09-05 18:21:34.664 17B7AB90 verbose 'VpxaHalCnxHostagent'] Received callback in WaitForUpdatesDone
Sep 5 14:21:34 cam.company.com Sep 5 18:21:34 Vpxa: [2012-09-05 18:21:34.664 17B7AB90 verbose 'VpxaHalCnxHostagent'] [VpxaHalCnxHostagent::ProcessUpdate] Applying updates from 3675467 to 3675468 (at 3675467)

0 Karma

Michael_Schyma1
Contributor

I removed the parts of the configuration files that were not necessary.

0 Karma

landen99
Motivator

Keep in mind that every set of parenthesis is a capture group. $1 goes to the very first capture group, not the first named capture group.

0 Karma

Michael_Schyma1
Contributor

Thank you for responding. We are just commenting them out to try and figure out what the issue is of why we can not get the regex to work. We are trying to get the transforms.conf file to work the [vmware-host]

0 Karma

Ayn
Legend

Also, which of the settings are you referring to? I see multiple things going on, of which some parts are commented out. What are you expecting, and what's not working, specifically?

0 Karma

Ayn
Legend

Please indent code sections with 4 spaces at the beginning of the line. That way they will show correctly here on the site. Thank you.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...