Deployment Architecture

Why would 1 of 3 search heads in a search head cluster not show any results in the distributed management console?

paulnshelly_200
Explorer

I have three search heads in a search head cluster and they are all listed in my Distributed Management Console as search heads. Only 2 of the 3 instances are showing data when viewing in the DMC dashboards. The introspection log on the search head not displaying has the data and the index=_introspection shows data for that search head. I am running Splunk Enterprise 6.3.2.

Why would that one search head not show data in the dashboards?

Thanks.

Paul

0 Karma

ykou_splunk
Splunk Employee
Splunk Employee

Can you be more specific about which dashboard is not working, or are all dashboards not working for that search head?

In addition, please double check three things:
1. make sure the search head is a distributed search peer of DMC, so that DMC can query the search head's REST APIs to get current data. If this is set up correctly, at least the dashboards' Snapshot section should show something.
2. make sure the search head is forwarding it's internal logs to the indexers that DMC can query. Since you mentioned "The introspection log on the search head not displaying has the data and the index=_introspection shows data for that search head." I assume you already forwarded the internal logs to the indexers.
3. This might be the actual issue. Go to DMC set up page, and make sure all server roles are correct, then click the Apply Changes button on the top right corner of the set up page. This will make sure that DMC knows about that search head.

0 Karma

paulnshelly_200
Explorer

I have tried the search activity dashboard and the resource usage dashboard, When I looked at the KV Store dashboard data shows for this instance.

  1. It is a peer of the DMC.
  2. The search head is forwarding, it was showing the results when I first upgraded to 6.3.2.
  3. It is listed as a search head, kv store in the DMC.

Thanks.

Paul

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...