Automatic Field Extraction Using "Translatefix" App

[bcarr12]

Hi all,

I recently had the Translatefix app installed in my company's Splunk environment and it is working great, many thanks to Glenn for creating it! http://splunk-base.splunk.com/apps/22347/financial-information-exchange-fix-log-parsing

One question I have, is that when I take my FIX logs and pipe them to translatefix, the logs are transformed successfully into "plain english" fields, but Splunk never seems to auto-extract them so I can work with them (or it extracts some but not others). Is there anything I might be able to do to make this happen. Example of a translated log:
2013-05-22 12:55:04,078 INFO in.test_test1 - <10781 ExecutionReport (8=FIX.4.2 BodyLength=295 MsgType=Execution Report TargetSubID=ABC 129=123 TargetCompID=TESTCOMP SenderCompID=TESTCOMP2 SendingTime=20130522-16:55:04 MsgSeqNum=10781 TradeDate=20130522 OrderID=abc_123_456 ClOrdID=1234567890-1 ExecID=abc_2456435_123456 ExecTransType=New OrdStatus=Canceled Account=00000123 Symbol=TESTSYMBOL Side=2 OrderQty=1000 OrdType=Limit Price=8.50 TimeInForce=Day LastShares=0 LastPx=0.00 CumQty=400 AvgPx=8.499 TransactTime=20130522-16:55:04 OrigClOrdID=1234567890-0 ExecType=Canceled LeavesQty=0 CheckSum=092 )

Everything seems clearly seperated so I am not sure why Splunk is not automatically extracting any of the created fields. Any thoughts as to how I can make this happen?

[Glenn]

Hi bcarr12,

I'm not exactly sure why Splunk doesn't extract it automatically either, except that perhaps the automatic extraction takes place on the data further up the pipeline than where translatefix operates.

You can easily extract them manually with a few commands after the translatefix in your search string. Just add: | extract kvdelim="=" pairdelim=" "

See this blog post for more details: http://blogs.splunk.com/2010/10/04/splunk-in-financial-services/

This may be fixed in future when I get around to updating this add-on. It has many necessary improvements - FIX field coverage, efficiency, this problem, and the fact it doesn't even work on Splunk 5 apparently.

Cheers,

Glenn

[Glenn]

Thanks for the confirmation mmezei! I have since done a 5.0.3 upgrade on our environment and did not see any issues myself, so it's nice to hear of the same experience for someone else.

[mmezei]

Glenn,
Just wanted to let you know that it works fine with Splunk 5.0.3 - you just need to add one item to make it available from within other apps. I did it manually: Apps menu->Manage Apps->translatefix view objects->permissions

[Glenn]

No problem. I'll try to let you know when it's updated. Hey, if you feel like this answered your question, would you mind marking my answer as the correct one? I wouldn't normally ask... but this would finally put me over 1000 points 🙂

[bcarr12]

Glenn, thanks again! This one tip made your already awesome app even better! "Translatefix" has saved myself and my team so much work digging through logs and going to separate websites to translate one by one. Looking forward to any future updates you may have!

Brent