Getting Data In

load compressed files

dmlee
Communicator

Hi,

as we know , before splunk eat a compressed file, splunk will decompress it first then index it.

but, if we have many compressed files under the same directory (ex: ap_20110301.zip, ap_20110302.zip ...) and their original file name are the same (ex:ap.log), what will happen ?

will splunk decompress all those files then index them ? or decompress and index one by one ?

because their original file name are the same , if splunk decompress all of the files at first , it will overwrite existing files (actually, this is what we observed, but we want to make sure).

thanks.

Tags (1)
1 Solution

Stephen_Sorkin
Splunk Employee
Splunk Employee

Splunk never actually decompresses the files within archives to a temporary location on disk. Instead we use a library called "libarchive" that allows us to stream through the contents of archives. These streamed contents are then indexed.

View solution in original post

Stephen_Sorkin
Splunk Employee
Splunk Employee

Splunk never actually decompresses the files within archives to a temporary location on disk. Instead we use a library called "libarchive" that allows us to stream through the contents of archives. These streamed contents are then indexed.

dmlee
Communicator

lessons learned, thanks

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...