Seems like a relatively simple issue but I'm stumped.
I've got peers setup on on a search head.. and if I do a search referencing an index ONLY available on the remote peer.. it will only work if I do it as follows.
index=INDEX_NAME splunk_server=*
if I don't invoke splunk_server= no results are returned.
Further, when in access control none of the indexes are shown from the peer.
By default only the main index is searched. I did try setting default searched indexes to all non_internal but that didn't work either unless i invoked splunk_server.
What am I missing?
This appears to be a result of a bug .."(SPL-95114) where instability in an indexing cluster can lead to distributed search groups losing the internal reference to their members. When this happens, the default target group "dmc_group_indexer" acts as if it had no members even though those are listed in distsearch.conf, and as a result your searches will not be dispatched to any remote peers unless you specify a splunk_server=* or splunk_server_group=* clause."
https://answers.splunk.com/answers/221468/search-returns-zero-results-searchlog-reports-dist.html
Cleaning up distsearch.conf with splunk restart resolved this issue
This appears to be a result of a bug .."(SPL-95114) where instability in an indexing cluster can lead to distributed search groups losing the internal reference to their members. When this happens, the default target group "dmc_group_indexer" acts as if it had no members even though those are listed in distsearch.conf, and as a result your searches will not be dispatched to any remote peers unless you specify a splunk_server=* or splunk_server_group=* clause."
https://answers.splunk.com/answers/221468/search-returns-zero-results-searchlog-reports-dist.html
Cleaning up distsearch.conf with splunk restart resolved this issue
Check your distsearch.conf on the search head.
Do you have anything odd in it?
servers = localhost:localhost
[distributedSearch:dmc_group_search_head]
servers = localhost:localhost
[distributedSearch:dmc_group_cluster_master]
[distributedSearch:dmc_group_deployment_server]
[distributedSearch:dmc_group_kv_store]
[distributedSearch:dmc_group_indexer]
default = true
servers = localhost:localhost
[distributedSearch:dmc_group_shc_deployer]
[distributedSearch]
servers = https://name_removed.com:8089
so basically nothing but the peer info
Looks like the stuff was "crazy" lol will answer my original question with the fix.