Why would 1 of 3 search heads in a search head cluster not show any results in the distributed management console?

[paulnshelly_200]

I have three search heads in a search head cluster and they are all listed in my Distributed Management Console as search heads. Only 2 of the 3 instances are showing data when viewing in the DMC dashboards. The introspection log on the search head not displaying has the data and the index=_introspection shows data for that search head. I am running Splunk Enterprise 6.3.2.

Why would that one search head not show data in the dashboards?

Thanks.

Paul

[ykou_splunk]

Can you be more specific about which dashboard is not working, or are all dashboards not working for that search head?

In addition, please double check three things:
1. make sure the search head is a distributed search peer of DMC, so that DMC can query the search head's REST APIs to get current data. If this is set up correctly, at least the dashboards' Snapshot section should show something.
2. make sure the search head is forwarding it's internal logs to the indexers that DMC can query. Since you mentioned "The introspection log on the search head not displaying has the data and the index=_introspection shows data for that search head." I assume you already forwarded the internal logs to the indexers.
3. This might be the actual issue. Go to DMC set up page, and make sure all server roles are correct, then click the Apply Changes button on the top right corner of the set up page. This will make sure that DMC knows about that search head.

[paulnshelly_200]

I have tried the search activity dashboard and the resource usage dashboard, When I looked at the KV Store dashboard data shows for this instance.

1. It is a peer of the DMC.
2. The search head is forwarding, it was showing the results when I first upgraded to 6.3.2.
3. It is listed as a search head, kv store in the DMC.

Thanks.

Paul