Getting Data In

Why is batch processing not removing files after indexing them in Splunk 6.2.1?

jizzmaster
Path Finder

I have an app that is not removing/deleting the files after consuming them. They are indexed appropriately, but just not deleted afterwards.

inputs.conf

[batch:///opt/splunk/etc/apps/my-special-app/pickup/*.json]
index = test
sourcetype = nessus_json
move_policy = sinkhole

I have tested this on a second Splunk box and the exact same app will correctly remove the files after indexing them. I can't tell where the issue may be on this main Splunk box, however. Any suggestions?

On Splunk v6.2.1. This worked a month or so ago. I'd rather figure out the cause before moving to upgrade the Splunk instance.

0 Karma

micahkemp
Champion

I forced a permissions issue with a file such that the Splunk user had read, not write permissions to a file that was configured as a batch input. It resulted in this line in splunkd.log:

11-24-2017 22:49:10.062 +0000 ERROR TailReader - Unable to remove sinkhole file: path=/tmp/batch_del_fail.log, errno=Operation not permitted

Can you look for a similar message to verify if it is a simple permissions issue?

0 Karma

samhays
Path Finder

Sounds like the permissions on the filesystem got screwed up. Check to make sure the account splunk is running as can actually delete the files.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...