Getting Data In

How to automatically extract key value pairs after using translatefix

isha_rastogi
Path Finder

I have used translatefix to decode the fix messages logs and it worked fine. But Splunk is not able to automatically extract key-value pairs. I don't want to write extract kvdelims and pairdelims in each search. How can I automatically extract key-values using configuration files

My Original data:
19:14:59.338308 outgoing: 8=FIX.4.X|9=12|35=0|34=123456|49=ABC1|52=20170406-23:14:59.338|56=XYZ1|10=123|

After using translatefix as command:
index=abc sourcetype=xyz | translatefix
data is decoded as:

19:14:59.338308 outgoing: BeginString=FIX.4.X BodyLength=12 MsgType=Heartbeat MsgSeqNum=123456 SenderCompID=ABC1 SendingTime=20170406-23:14:59.338 TargetCompID=XYZ1 CheckSum=123

To extract key value pair I have to use extract command in search like below:

ndex="abc" sourcetype=xyz | translatefix | extract pairdelim=" " kvdelim="=" .
Is there any way I can configure to extract automatically using conf files (props.conf, transforms.conf). So I dont have to write extract in each search query and data is extracted automatically.

I tried using delims and regex (\w+)=([^[\s]+) in transforms.conf. It worked for original data but not working after using translatefix command.

0 Karma
1 Solution

lguinn2
Legend

Okay, I gave a dumb answer because I didn't realize that you are using translatefix inline. If you are reformatting the data on the fly (which is how translatefix works), you can't put the field extractions in props.conf or transforms.conf based on the NEW format.

However, you can certainly add the field extractions based on the original/stored format. Of course, in the original format, you don't get the "real" field names, instead you get weird names like "10" and "56." To make the names pretty, you would need to do a bunch of renames, which doesn't really help either.

I suggest a macro. Let's call it infixed_xyz() and let the body of the macro be

sourcetype=xyz | translatefix | extract pairdelim=" " kvdelim="="

now you can write a search like this

index=abc `infixed_xyx` | where TargetCompID=XYZ1 | stats or whatever

View solution in original post

0 Karma

lguinn2
Legend

Okay, I gave a dumb answer because I didn't realize that you are using translatefix inline. If you are reformatting the data on the fly (which is how translatefix works), you can't put the field extractions in props.conf or transforms.conf based on the NEW format.

However, you can certainly add the field extractions based on the original/stored format. Of course, in the original format, you don't get the "real" field names, instead you get weird names like "10" and "56." To make the names pretty, you would need to do a bunch of renames, which doesn't really help either.

I suggest a macro. Let's call it infixed_xyz() and let the body of the macro be

sourcetype=xyz | translatefix | extract pairdelim=" " kvdelim="="

now you can write a search like this

index=abc `infixed_xyx` | where TargetCompID=XYZ1 | stats or whatever
0 Karma

isha_rastogi
Path Finder

As translatefix is configured in commands.conf and I'm using it as command. Can I configure it in configuration file so that it will be automatically applied instead of writing it inline.

0 Karma

lguinn2
Legend

In the props.conf (on the search head or wherever users log in), add the following:

[xyz]
KV_MODE = auto

And the key-value pairs (separated by 😃 will be extracted. You do not need the transforms.conf.

0 Karma

isha_rastogi
Path Finder

Tried to put KV_MODE =auto. No success :(. ( Tried on new indexed data)

0 Karma

adonio
Ultra Champion

can you share sample data after it was decoded? (in plain text)

0 Karma

isha_rastogi
Path Finder

Yes Sure.

My Original data:
19:14:59.338308 outgoing: 8=FIX.4.X|9=12|35=0|34=123456|49=ABC1|52=20170406-23:14:59.338|56=XYZ1|10=123|

After using translatefix as command:
index=abc sourcetype=xyz | translatefix
data is decoded as:

19:14:59.338308 outgoing: BeginString=FIX.4.X BodyLength=12 MsgType=Heartbeat MsgSeqNum=123456 SenderCompID=ABC1 SendingTime=20170406-23:14:59.338 TargetCompID=XYZ1 CheckSum=123

To extract key value pair I have to use extract command in search like below:

ndex="abc" sourcetype=xyz | translatefix | extract pairdelim=" " kvdelim="=" .
Is there any way I can configure to extract automatically using conf files (props.conf, transforms.conf)

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...