Getting Data In

wheater splunk forwarder can compress data brfore forward

perlish
Communicator

Hi, I use splunk forwarder to forward data.
But the data was too much.
I want compress data before forward, wheather splunk support this?

Thank you

Tags (2)
0 Karma

DaveSavage
Builder

Perlish - have you considered zip'ing the files and then using a heavy forwarder to send them to your back end indexer?
There are quite a few threads here which discuss it, not least being: http://splunk-base.splunk.com/answers/52976/indexing-log-files-which-are-in-zip-format
Good luck. I'm still struggling to get my head around the quantities you must be talking about!
Br
Dave

0 Karma

Ayn
Legend

Docs?

http://docs.splunk.com/Documentation/Splunk/5.0/Deploy/Configureforwarderswithoutputs.confd#Forwardi...

 compressed     false   global or target group stanza   Specifies whether the forwarder sends compressed data. 

MuS
SplunkTrust
SplunkTrust

hi perlish

if I understand your request correct, you want to filter out unneeded data on the forwarder, right?
if so, you can setup a so call heavy forwarder and setup filters and routes on it.

Filters and routes could also be setup on the indexer.

cheers,
MuS

0 Karma

kristian_kolb
Ultra Champion

Are you saying that you need sustained data transfer rates of 20 MB/s? That's just over 1,7 TB per day. Seriously, that is a LOT.

With those requirements (and that kind of license), I believe that Splunk Support will help you out. File a support case at splunk.com/support

0 Karma

perlish
Communicator

Everyday,every second,i need forward data,the rate will be 20M/s, So if i limit rate, i can`t forward all data in oneday。

0 Karma

MuS
SplunkTrust
SplunkTrust

when you are saying you are loosing data, what kind of data is it and why does it get lost if it is not forwarded with in one day?

0 Karma

perlish
Communicator

yes.
It is.

0 Karma

MuS
SplunkTrust
SplunkTrust

and you really need every single bit of the data you are forwarding?

0 Karma

perlish
Communicator

Because if I dont limit, the rate will be 20M/s
If i limit, for example limit 10M/s.
I can
t finished forward in everyday.

0 Karma

MuS
SplunkTrust
SplunkTrust

why should this happen?

0 Karma

perlish
Communicator

But if i limit the rate, the data will loss.

0 Karma

MuS
SplunkTrust
SplunkTrust

you could limit the transfer rate on the forwarder likt this http://splunk-base.splunk.com/answers/52066/data-transfer-rate-between-forward-and-indexer
but I'm not aware of any compress feature.

0 Karma

perlish
Communicator

Oh,sorry, it`s my fault

I want to compress data.

Sorry.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...