When you using a heavy forwarder, parsing happens on the forwarder. While I am pretty sure that the KV_MODE should apply only at search time, I wonder if you are missing something because it appears only on the search head / indexer.
I would try adding the KV_MODE=JSON to the heavy forwarder as well as the indexer/search head. It won't hurt to have it in two places. Try it and see if that works. There is really no down side.
You might also consider installing the free Splunk on Splunk app (SoS). This would help you see all the places that might be affecting the configuration of the alarm_log sourcetype. Or try btool
... View more