Splunk forwarder loop

[dbond]

Hi,

We have a couple of instances where the splunk forwarder gets into a loop due to firewall logging.

The Forwarder is installed on windows 2008 R2, its a domain controller, firewall activity is logged (to the security event log). When The Splunk forwarder sends data to the splunk server it gets logged in the event log, this then triggers another send by splunk, which then get logged and triggered etc. This doesnt always happen, it happens after a reboot, or just after some time, it can be fine.

Why is it doing this? How can it be stopped? I have to stop the forwarder and test after a while to see if it still does it. At the moment it has sent 13GB of logs to splunk, containing mostly logs of the splunk forwarder sending logs to splunk.

Is there a way to get the splunk forwarder to exclude the log for the splunk forwarder, or to only send the data from the logs every 10 seconds, instead of right now when ever a new entry appears? Or is there another solution?

Thank You

David

[Drainy]

Aha, I had the same issue. It is a log message to say that a connection has been established. I think I just needed to turn down the logging for those sorts of messages.
Anyway to filter them and never index them read here;
http://splunk-base.splunk.com/answers/24000/how-do-i-exclude-some-windows-events-from-being-indexed

It has a good example on how to filter by eventcode

[Drainy]

Well this is true, but you can install a Splunk indexer on the remote machine and instead configure it as a forwarder, this was how things were done before the UF or where you have specific requirements (such as this), where the power of the Splunk indexer is required.

[dbond]

Thanks for your reply. From my understanding, the props.conf and transfoms.conf are only parsed on splunk, the forwarder ingnores them.
This would need to be done on the forwarder as when this happens, thousands of entries a second are added, pushing the CPU usage of splunk forwarder to close to 100%, it appears that the splunk forwarder isnt keeping the connection to the splunk server open.