Right as I posted that I found the issue - in case anyone is curious, the server.pem file had expired on a few of our searchheads. You can check that by running: openssl x509 -enddate -noout -in <splunklocation>/etc/auth/server.pem If it is expired, just rename the server.pem file to server.pem.bak<date> or something like that, and restart splunk, it will generate a new one.
... View more
with that many forwarders, you should be (IMHO) using GIT for managing your configs. and then you deploy the entire set of configs to all of the DS. think of managing them as a whole.
And with a deployment of that size, you should be talking to your Splunk Field Architect.
... View more