I have a 4.3.3 UF on a windows 2008r2 box that was forwarding windows event logs quite happily.
It's now stopped forwarding but, if I restart splunk on the forwarding server, the missing events are forwarded, but no new events until I restart splunk again.
Short of restarting splunk every 5 minutes, can any one suggest why this might be happening?
Accepted the answer by mistake - any way you can unaccept?
The issue went away when we upgraded, we are now splunk 6 and not an issue so far......
I had the same symptoms, it was a configuration issue.
Make sure you fully understand ignoreOlderThan=
https://docs.splunk.com/Documentation/Splunk/6.6.1/Admin/Inputsconf
In my case logs were not written to for 7 + days and then splunk will no longer try to read from that file even when new events appeard in the file..
In my case, it was a misunderstanding of ignoreOlderThan= in inputs.conf.
ignoreOlderThan will completely ignore files that ever reach this threshold.
From the inputs.conf documentation.
"Do NOT select a time that files you want to read could reach in age, even temporarily"
My files wouldn't write to the logs for several weeks and then begin writing again. Splunk would not even try to ingest them.
Accepted the answer by mistake - any way you can unaccept?
The issue went away when we upgraded, we are now splunk 6 and not an issue so far......
We are experiencing a few issues with our windows forwarders and one of them sounds like it might be the same. We have this issue where we get splunk internal logs constantly, but monitored files are only sent on shutdown of the universal forwarder. This problem appears to be that the splunk forwarder was trying to "restart" too quickly. When we tried to restart the forwarder we received an error message that the process was taking too long - but it appeared stopped in the windows serverice listing so we started it up again. No error message was received on startup. We then received the internal logs as expected, but didn't receive the application log file we were monitoring. We then stopped the forwarder (received the error message again) and waited about 5 minutes. After 5 minutes we started the forwarder and both the splunk internal logs and the monitored log files were continuously coming through. It appears that the stop, pause for a longer time, then start appeared to fix this issue.
This did not fix the issue where the splunkd logs only came through when the agent was stopped and the monitored logs never came through.
On the forwarder, have you looked at the splunkd log? You will find it in the subdirectory $SPLUNK_HOME\var\log\splunk
Let us know what you find there...