Getting Data In

Batch large amount of zipped logs

mattmorning
Explorer

I'm trying to run a batch process for zipped log files. Splunk can read the total number of files (displayed in the Data Input Manager, but it doesn't index them.

Here's my listing of the inputs.conf:

[batch://c:\logs\2010-10\] 
move_policy = sinkhole 
crcSalt = <SOURCE> 
disabled = false

Log file location on HDD:

c:\logs\2010-10\2010-10-01\http_access.log

Thanks for any help!

Tags (2)
0 Karma
1 Solution

Mick
Splunk Employee
Splunk Employee

batch inputs work much the same way as monitor inputs, so the troubleshooting instructions here will enable you to see what Splunk is doing when it reads these files. Set up the DEBUG logging as instructed, and add it in for the archiveProcessor too, that's the one that will open up your files. You'll see the resulting logs in splunkd.log

Also, be aware that due to the unknown 'real' size of a compressed file, Splunk will only process one file at a time, so it may actually be working, just a lot slower than you would expect.

View solution in original post

mattmorning
Explorer

Here's whats in the splunkd.log:

11-04-2010 18:46:23.346 INFO ArchiveProcessor - handling file=C:\logs\test\2010-10-01\pwdf4619.zip 11-04-2010 18:46:23.346 INFO ArchiveProcessor - reading path=C:\logs\test\2010-10-01\pwdf4619.zip (seek=0 len=2479775) 11-04-2010 18:46:23.346 DEBUG ArchiveProcessor - fishstate=key=0xb6863de5131a6b9f sptr=2479775 scrc=0x11837cc8fbac8c86 fnamecrc=0x619903da57a2fbc6 modtime=0 11-04-2010 18:46:23.346 INFO ArchiveProcessor - Finished processing file 'C:\logs\test\2010-10-01\pwdf4619.zip', removing from stats 11-04-2010 18:46:23.346 DEBUG ArchiveProcessor - 'C:\logs\test\2010-10-01\pwdf4619.zip' is located in a sinkhole, deleting archive

0 Karma

mattmorning
Explorer

OK. I fixed this damn issue regarding zip files.

zip files were currupted. Now splunk eats what I drop in that folder.

Mission part 1 accomplished.

0 Karma

Mick
Splunk Employee
Splunk Employee

batch inputs work much the same way as monitor inputs, so the troubleshooting instructions here will enable you to see what Splunk is doing when it reads these files. Set up the DEBUG logging as instructed, and add it in for the archiveProcessor too, that's the one that will open up your files. You'll see the resulting logs in splunkd.log

Also, be aware that due to the unknown 'real' size of a compressed file, Splunk will only process one file at a time, so it may actually be working, just a lot slower than you would expect.

mattmorning
Explorer

Thank you Mick.

Splunk eats up my files, but there still must be some problems regarding my files.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...