All Apps and Add-ons

Does Splunk and Elastic Map Reduce work Together?

skoelpin
SplunkTrust
SplunkTrust

I have a few indexes which have around 2.5 billion events each. Unfortunately we don't have a lot of CPU to sort through this massive data and make it meaningful in a dashboard. We're currently in the process of setting up a summary index, but the requirements/fields can change at anytime which mean's we'd have to re-summerize that data.

So my question is, can we use Amazon EMR as a temporary boost in horsepower to Map and Reduce this data back into the summary index? How difficult would this be to do?

1 Solution

hsesterhenn
Path Finder

Hi,

I would assume your instance with 2,5 billion events is also running on AWS?

Why not use HUNK on AWS and export your data with the Splunk Hadoop Connect App?

https://aws.amazon.com/de/elasticmapreduce/hunk/

HTH,

Holger

View solution in original post

hsesterhenn
Path Finder

Hi,

I would assume your instance with 2,5 billion events is also running on AWS?

Why not use HUNK on AWS and export your data with the Splunk Hadoop Connect App?

https://aws.amazon.com/de/elasticmapreduce/hunk/

HTH,

Holger

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...