Hi All,
This has happened to myself and other colleagues on more than one occasion. We go to resolve some issues with a customers Splunk installation and find that a field extraction somewhere in their nest of Splunk apps is breaking a critical field and it just can't be tracked down.
Usually after many hours of btooling and grepping to no avail, it is easier to start with a clean install and then add each app one by one until it breaks.
Is there some logging or debug tool that can be turned on to identify ALL the props and transforms that are applied to a particular sourcetype before it is presented to the user?
Something that lists the following for each field in a search
field, SplunkApp, Props.conf.stanza, transforms.conf.stanza
As I mentioned btool helps identify the easy ones that are acting on a sourcetype but it is harder to track down props that are acting on a src, host etc....
In the instance a colleague just encountered there is a Splunk ES install where the dest field is being "cleared" by a TA somewhere on the system. Btool and grepping the Splunk app directory yielded no obvious culprit. He has since started with a fresh install and is now adding each TA one by one until he can identify which one breaks it.
There has to be an easier way!
Maybe not the complete answer you are looking for, but another tool to add to your toolbox is the app Knowledge Object Explorer by @martin_mueller. It may reveal additional places to look based on what sourcetypes or eventtypes touch a given field.
App link: https://splunkbase.splunk.com/app/2871/
Looks pretty good but from what I can tell it just parses conf files so doesn't really report on a live search.
Been playing around with it for the last 30 mins and it could be useful for finding some problems though but probably wouldn't help in this situation. Thanks for the info though.
I am glad it could be at least somewhat useful to you even if it is not the complete answer you are looking for.
FWIW he found it. A previous Splunker had this in a TAs fields.conf
[dest]
TOKENIZER = (\d+\.\d+\.\d+\.\d+)
Looking for this with btool on just props and transforms means it would never have been found.