Getting Data In

Configure heavy forwarder to send data periodically

moohkhol
New Member

Hi Guys,

I set-up Heavy forwarder at Machine-1 and wants to send data on Machine-2, since network in involve in between so i want to minimize the network traffic to prevent certain type of events and send the imp event immediately but less imp event per day or per hours.
Many Places i read that using universal forwarder we cann't do filter as well as interval setting, that's why i moved to Heavy forwarder, I can able to filter out events in heavy forwarders but unable to set periodicity.

It would be great help if some one can tell me stanza for setting interval of polling,
lets say, I want to monitor, "testlog" all time and "testlog1" after every 30 mins and "testlog2" after every 1 hours.

Right now my inputs.conf is below,

[monitor:///usr/local/preview/splunk/testlog/*]
index = main
sourcetype = testlog

[splunktcp://9997]
connection_host = ip
_TCP_ROUTING = splunkindexer_9997

Tags (2)
0 Karma

jbrodsky_splunk
Splunk Employee
Splunk Employee

Also, on the bandwidth consumption concern, note that if you enable useclientSSLcompression on the universal forwarder, you will achieve significant compression benefit. See here:

http://answers.splunk.com/answers/92067/forwarder-output-compression-ratio

The heavyweight forwarder is not nearly as efficient. General rule of thumb - if you are dropping more than about 60% of the traffic, then it warrants an HWF.

0 Karma

gfuente
Motivator

Maybe, you could configure a regular monitor. Then use cron or any other scheduler tool to start and stop the UF as needed

You could install several instances of uf in the same host, if needed, one for real time monitoring and the other one for batch (with the start and stop trick...)

Regards

moohkhol
New Member

What's the best practice for splunk to achieve this, lets say i wants to use log for data analytics perspective, i don't need real time information, even if it's hours old or day old it would me fine for me, it would be very general requirement.
Any suggestion for that would help me a lots.

0 Karma

Ayn
Legend

This is not unique to Universal Forwarders - no Splunk instance has the option to send logs just periodically. So, moving to a heavy forwarder does not help.

0 Karma

Ayn
Legend

Commonly this is not a problem as most environments don't have so little bandwidth. If it's bandwidth you're concerned about, you can limit at which rate forwarders send their data in limits.conf. Default for a Universal Forwarder is 256kB/s.

0 Karma

moohkhol
New Member

What's the best practice for splunk to achieve this, lets say i wants to use log for data analytics perspective, i don't need real time information, even if it's hours old or day old it would me fine for me, it would be very general requirement.
Any suggestion for that would help me a lots.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...