Perlish - have you considered zip'ing the files and then using a heavy forwarder to send them to your back end indexer?
There are quite a few threads here which discuss it, not least being: http://splunk-base.splunk.com/answers/52976/indexing-log-files-which-are-in-zip-format
Good luck. I'm still struggling to get my head around the quantities you must be talking about!
Br
Dave
Docs?
compressed false global or target group stanza Specifies whether the forwarder sends compressed data.
hi perlish
if I understand your request correct, you want to filter out unneeded data on the forwarder, right?
if so, you can setup a so call heavy forwarder and setup filters and routes on it.
Filters and routes could also be setup on the indexer.
cheers,
MuS
Are you saying that you need sustained data transfer rates of 20 MB/s? That's just over 1,7 TB per day. Seriously, that is a LOT.
With those requirements (and that kind of license), I believe that Splunk Support will help you out. File a support case at splunk.com/support
Everyday,every second,i need forward data,the rate will be 20M/s, So if i limit rate, i can`t forward all data in oneday。
when you are saying you are loosing data, what kind of data is it and why does it get lost if it is not forwarded with in one day?
yes.
It is.
and you really need every single bit of the data you are forwarding?
Because if I dont limit, the rate will be 20M/s
t finished forward in everyday.
If i limit, for example limit 10M/s.
I can
why should this happen?
But if i limit the rate, the data will loss.
you could limit the transfer rate on the forwarder likt this http://splunk-base.splunk.com/answers/52066/data-transfer-rate-between-forward-and-indexer
but I'm not aware of any compress feature.
Oh,sorry, it`s my fault
I want to compress data.
Sorry.