aws:cloudtrail field extractiong not working on SHC env.

[sylim_splunk]

One item in particular that I'm seeing is the aws:cloudtrail sourcetype which is that the aws:cloudtrail sourcetype is not extracting fields as it is for other aws-related sourcetypes (e.g. aws:description).
This also appears to only occur on our SHC instead of the our standalone search head which does show the field extractions as expected.
Both search head environments are using the same TA content (being sourced by our Git repository) so the only difference is that the standalone search head is using the deployment server to get the TA while the search head cluster is getting the TA via deployer. I will attach screenshots of both environments as well as the correctly parsed aws:description data to demonstrate the issue.

[sylim_splunk · Accepted solution]

Typically this can happen due to the precedence between the apps & add-ons. As the different apps/add-ons are installed on SHC SH and the standalone SH the first thing is to try to find the differences.

Use find . -name "props.conf" -o -name "transforms.conf" | xargs grep "aws:cloudtrail" this will tell us what files have configurations for the sourcetype in question.
This time, no other "aws:cloudtrail" related configuration in the standalone search head. However, in the SHC there is another props.conf in SAI, which has many other aws:XYZ sourcetypes, also it has a suspicious one, i.e, kv_mode = none - this is basically disabling automatic key-value field extractions.

[aws:cloudtrail] in splunk_app_infrastructure/default/props.conf
KV_MODE = none

The props.conf in SAI has higher precedence than what is defined in splunk_app_aws, "kv_mode=json" .

https://docs.splunk.com/Documentation/Splunk/8.0.3/Admin/Wheretofindtheconfigurationfiles#How_app_di...
" In the app/user context, precedence is determined instead by reverse-lexicographical order. "
To solve this problem, comment out the kv_mode in SAI.

[johnansett]

Thanks, this solved that exact issue.  Only thing I would say it's better to create a props.conf in local in the SAI app and then added the stanza

[aws:cloudtrail]
KV_MODE = json

That way when you upgrade the app it won't revert back.