Configure heavy forwarder to send data periodically

[moohkhol]

Hi Guys,

I set-up Heavy forwarder at Machine-1 and wants to send data on Machine-2, since network in involve in between so i want to minimize the network traffic to prevent certain type of events and send the imp event immediately but less imp event per day or per hours.
Many Places i read that using universal forwarder we cann't do filter as well as interval setting, that's why i moved to Heavy forwarder, I can able to filter out events in heavy forwarders but unable to set periodicity.

It would be great help if some one can tell me stanza for setting interval of polling,
lets say, I want to monitor, "testlog" all time and "testlog1" after every 30 mins and "testlog2" after every 1 hours.

Right now my inputs.conf is below,

[monitor:///usr/local/preview/splunk/testlog/*]
index = main
sourcetype = testlog

[splunktcp://9997]
connection_host = ip
_TCP_ROUTING = splunkindexer_9997

[jbrodsky_splunk]

Also, on the bandwidth consumption concern, note that if you enable useclientSSLcompression on the universal forwarder, you will achieve significant compression benefit. See here:

http://answers.splunk.com/answers/92067/forwarder-output-compression-ratio

The heavyweight forwarder is not nearly as efficient. General rule of thumb - if you are dropping more than about 60% of the traffic, then it warrants an HWF.

[gfuente]

Maybe, you could configure a regular monitor. Then use cron or any other scheduler tool to start and stop the UF as needed

You could install several instances of uf in the same host, if needed, one for real time monitoring and the other one for batch (with the start and stop trick...)

Regards

[moohkhol]

What's the best practice for splunk to achieve this, lets say i wants to use log for data analytics perspective, i don't need real time information, even if it's hours old or day old it would me fine for me, it would be very general requirement.
Any suggestion for that would help me a lots.

[Ayn]

This is not unique to Universal Forwarders - no Splunk instance has the option to send logs just periodically. So, moving to a heavy forwarder does not help.

[Ayn]

Commonly this is not a problem as most environments don't have so little bandwidth. If it's bandwidth you're concerned about, you can limit at which rate forwarders send their data in limits.conf. Default for a Universal Forwarder is 256kB/s.

[moohkhol]

What's the best practice for splunk to achieve this, lets say i wants to use log for data analytics perspective, i don't need real time information, even if it's hours old or day old it would me fine for me, it would be very general requirement.
Any suggestion for that would help me a lots.