We use UF to forward events from a group of servers to our indexers. We also use deployment server to ensure consistent configs across the UFs. A few of the servers are NOT forwarding the Windows Application Log, but ARE forwarding the System and Security Logs... The inputs.conf and outputs.conf are IDENTICAL on each box. I've also noticed that on servers that are experiencing the problem, the application_checkpoint file is missing in the persistentstorage folder.
Turned out it was corrupt event logs.. Cleared the logs, restarted splunk... Voila!
Turned out it was corrupt event logs.. Cleared the logs, restarted splunk... Voila!