Getting Data In

Is it expected behavior for Hunk to delete files as they age from the dispatch directory used for MapReduce?

splunkIT
Splunk Employee
Splunk Employee

We are using Hunk with MapR. There is a dispatch directory that Hunk uses for the reduce of the map reduce. /mapr/tmp/dispatch.

We are seeing files deleted from this directory while the search is progressing: First the heartbeat file, and then the sub directories.
The search has not completed and this hangs the search job forever.

Is this part of Hunk to delete these files as they age? If so, what is the delete age and or setting to control it?

We are also seeing any non hunk file being deleted from this directory. Is Hunk deleting them?

0 Karma

rdagan_splunk
Splunk Employee
Splunk Employee

Overall – yes this is expected. Hunk deletes the files under the Hunk HDFS working directory.
What is not expected, is that it will delete these files before the job is completed and these files are still needed

0 Karma

ddrillic
Ultra Champion

How does the directory structure look under /mapr/tmp/dispatch? Each search job should have its own directory...

0 Karma

rdagan_splunk
Splunk Employee
Splunk Employee

Do you have two instances of Hunk pointing to the same Hunk working directory in HDFS?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...