Deployment Architecture

No logs from 2 SHC search heads

davebo1896
Communicator

Two of our three search heads have stopped forwarding logs.
It doesn't matter if they are captain or not.

I have verified server.conf and outputs.conf look good using splunk btool server all and splunk btool outputs all

I have tried cleaning all search heads (remove from cluster, remove non-standard apps and user, run splunk clean all, and put back in the cluster)

The logs are being written, but are not being forwarded to the indexers.

Any ideas what to check next?

0 Karma
1 Solution

davebo1896
Communicator

I got it figured out.

in /opt/splunk/etc/system/local/inputs.conf

  [default]
  host =<server>

host was set to the same hostname on all three search heads.
No idea how that happened, but setting host to the respective hostname and cluster restart fixed the problem

View solution in original post

0 Karma

davebo1896
Communicator

I got it figured out.

in /opt/splunk/etc/system/local/inputs.conf

  [default]
  host =<server>

host was set to the same hostname on all three search heads.
No idea how that happened, but setting host to the respective hostname and cluster restart fixed the problem

0 Karma

sk314
Builder

Does the outputs.conf have this line

[indexAndForward]
index = false

Ref: http://docs.splunk.com/Documentation/Splunk/latest/DistSearch/Forwardsearchheaddata

0 Karma

davebo1896
Communicator

Yes. They are using index=false

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...