UPDATE:
So finally got a Splunk engineer and went through the whole upgrade process. Long story short, LDAP was causing my indexers to bomb out. LDAP!!!!!!!
From what Splunk said, the indexer should not be doing lookups on queries coming from the search heads. I also see more (almost constant) LDAP warnings in 6.5.x over 6.4.x. So LDAP lookups was causing timeouts on the indexers and causing blocking. I disabled LDAP on the indexers and no more blocking.
Food for thought guys.
This at least doesn't affect our end users since the search heads still do LDAP and also proves that once the search is passed to the indexer, the user does not need to be in the indexer (auth wise). So Splunk will have to figure out why the indexers are trying to do auth on search head queries but at least it's a quick fix.
... View more