We are having an issue where a Universal Forwarder configured to forward a half dozen custom application logs is not forwarding any of them. It is a Windows server, and we are seeing Windows security information come across, and if I point to a known log, c:/\windows/\windowsupdate.log in inputs.conf, it sends that along. However, the other files are not coming across to the indexer.
They use custom sourcetypes that have been correctly specified and I can examine and appear to be set up correctly. The types are specified in inputs.conf for each file stanza and I can see them in Splunk's props.conf.
I cannot see a reason that only our custom files would be ignored. In splunkd.log on the forward it acknowledges their stanzas for each file:
03-31-2015 12:48:41.248 -0400 INFO TailingProcessor - Parsing configuration stanza: monitor://
There are no warnings or errors in the log file.
Splunk is running as a local system account and has permissions to the files.
I do not know what could be causing this issue and I am unsure where else I can look to diagnose the issue.
couple of things you can try, after seeing that message it sounds like parsing Que issue.
you can try increase
maxQueueSize = 200MB in outputs.conf.
some times the maxQueueSize setting causes events to be loaded into a queue in memory so Before increase check your memory and performance using top what ever OS command.
Note: check out metrics.log and parsingQueue or TCPque is full etc errors
Thanks, I've tried that but it does not seem to have changed anything. I'm pulling the log files on the forwarder and I will examine the metrics.log