Splunk Search

Is there a logging or debug tool to identify all props and transforms that are applied to a particular sourcetype to isolate rogue field extractions?

phoenixdigital
Builder

Hi All,

This has happened to myself and other colleagues on more than one occasion. We go to resolve some issues with a customers Splunk installation and find that a field extraction somewhere in their nest of Splunk apps is breaking a critical field and it just can't be tracked down.

Usually after many hours of btooling and grepping to no avail, it is easier to start with a clean install and then add each app one by one until it breaks.

Is there some logging or debug tool that can be turned on to identify ALL the props and transforms that are applied to a particular sourcetype before it is presented to the user?

Something that lists the following for each field in a search
field, SplunkApp, Props.conf.stanza, transforms.conf.stanza

As I mentioned btool helps identify the easy ones that are acting on a sourcetype but it is harder to track down props that are acting on a src, host etc....

In the instance a colleague just encountered there is a Splunk ES install where the dest field is being "cleared" by a TA somewhere on the system. Btool and grepping the Splunk app directory yielded no obvious culprit. He has since started with a fresh install and is now adding each TA one by one until he can identify which one breaks it.

There has to be an easier way!

rjthibod
Champion

Maybe not the complete answer you are looking for, but another tool to add to your toolbox is the app Knowledge Object Explorer by @martin_mueller. It may reveal additional places to look based on what sourcetypes or eventtypes touch a given field.

App link: https://splunkbase.splunk.com/app/2871/

phoenixdigital
Builder

Looks pretty good but from what I can tell it just parses conf files so doesn't really report on a live search.

Been playing around with it for the last 30 mins and it could be useful for finding some problems though but probably wouldn't help in this situation. Thanks for the info though.

0 Karma

rjthibod
Champion

I am glad it could be at least somewhat useful to you even if it is not the complete answer you are looking for.

0 Karma

phoenixdigital
Builder

FWIW he found it. A previous Splunker had this in a TAs fields.conf

[dest]
TOKENIZER = (\d+\.\d+\.\d+\.\d+)

Looking for this with btool on just props and transforms means it would never have been found.

0 Karma
Get Updates on the Splunk Community!

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...