In our situation, the problem was actually the permissions on this one particular log file. It appears that when Splunk was upgraded, the permission on the log file was set to root only and splunk was not able to read the log file. We don't run Splunk as a root user, therefore we had no other choice but to change ownership of the file so Splunk could read it. We are running RHEL 8.x, so "chown -R splunk:splunk /opt/splunk" did the trick. Once we restarted Splunk the issue went away immediately. Just like several others had mentioned previously, we were only seeing the issue on our Cluster Master and no other Splunk application server. Hope this helps!
... View more