All Apps and Add-ons

How will thousands of syslog events sent to Hadoop affect a heavy forwarder?

a212830
Champion

Hi,

I am currently processing syslog events, using the hfw. This feed is pretty busy - hundreds of files, and I'm being asked to forward all the data to Hadoop. How will this affect the forwarder? These events are critical within Splunk, and I don't want any delay to be added. I'm not sure of the need for real-time here, so my suggestion is going to be Hadoop Connect.

0 Karma

hsesterhenn_spl
Splunk Employee
Splunk Employee

Hi,

indexing the data using Splunk Enterprise Core in combination with a HFW should not be influenced if you export the data off the indexer using Hadoop Connect.

Remember, Hadoop Connect will run a search and then export the result/data to Hadoop.

Maybe the Hadoop Data Roll feature is a better option if you want to archive buckets instead of exporting files.

https://docs.splunk.com/Documentation/Splunk/latest/Indexer/ArchivingindexestoHadoop

HTH,

Holger

0 Karma

a212830
Champion

thousands of events...

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...