All Apps and Add-ons

How to get Splunk Hadoop Connect output to write to the same CSV file instead of creating a new one for each job?

venkatsai123
New Member

I am trying a job using Splunk Hadoop Connect. When the job ran for the 1st time, output was written into a new .csv file.. after an hour, when the job reran, I see it create a new file instead of append the output to the old .csv file.

I am confused.

Can anyone tell me how write output to same .csv file instead of creating new one?

0 Karma

rdagan_splunk
Splunk Employee
Splunk Employee

What you see is the default behavior of Hadoop Connect. As the job runs, the Splunk platform processes chunks of data received from the search and creates compressed files, locally on the search head. These files are moved to HDFS or the mounted file system if they reach 64MB or if cumulatively they consume more than 1GB, or the search finishes successfully. You can change these limits in the Export Defaults configuration section.
The reason most people select to stay with these default settings - 64MB or 128MB - is that Hadoop itself likes to process the file based on the Hadoop Block size

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...