All Apps and Add-ons

Hunk jobs hang in Amazon Elastic MapReduce

csharp_splunk
Splunk Employee
Splunk Employee

When I run a search on Hunk against Amazon Elastic MapReduce the searches will run endlessly and never complete giving the full results. How can I fix this?

1 Solution

csharp_splunk
Splunk Employee
Splunk Employee

Generally Hunk by default will spin up 10 concurrent threads to read files from the Hadoop File System. In the case of Elastic MapReduce this is generally backed by S3. We have noticed that this level of concurrency sometimes creates issues and potential deadlock conditions when reading data from EMR and the s3n filesystem. The quick workaround for this is to minimize the number of concurrent threads reading data from the filesystem using the following setting on the External Results Provider:

vix.splunk.search.mr.threads = 1

This will set the concurrency down to 1 thread and will at least temporarily work around the issue. We are looking into a more permanent fix.

View solution in original post

csharp_splunk
Splunk Employee
Splunk Employee

Generally Hunk by default will spin up 10 concurrent threads to read files from the Hadoop File System. In the case of Elastic MapReduce this is generally backed by S3. We have noticed that this level of concurrency sometimes creates issues and potential deadlock conditions when reading data from EMR and the s3n filesystem. The quick workaround for this is to minimize the number of concurrent threads reading data from the filesystem using the following setting on the External Results Provider:

vix.splunk.search.mr.threads = 1

This will set the concurrency down to 1 thread and will at least temporarily work around the issue. We are looking into a more permanent fix.

Get Updates on the Splunk Community!

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Wednesday, May 29, 2024  |  11AM PST / 2PM ESTRegister now and join us to learn more about how you can ...

Get Your Exclusive Splunk Certified Cybersecurity Defense Engineer at Splunk .conf24 ...

We’re excited to announce a new Splunk certification exam being released at .conf24! If you’re headed to Vegas ...

Share Your Ideas & Meet the Lantern team at .Conf! Plus All of This Month’s New ...

Splunk Lantern is Splunk’s customer success center that provides advice from Splunk experts on valuable data ...