Getting Data In

Will Splunk re-index a log file if I compress it after it's been indexed?

Branden
Builder

I have an environment set-up such that syslog-ng redirects syslog to my central log host in this format: /server/host/year/month/day/syslog.log

Splunk monitors that directory tree and indexes it. No big deal.

I'd now like to compress older syslog files and leave the compressed file in that directory. So /server/foo/2010/05/04/syslog.log would become /server/foo/2010/05/04/syslog.log.gz.

I know Splunk can index compressed files. If I compress the syslog file, will the monitor detect the 'new' compressed file and attempt to index it? Or do I have to move the compressed logs somewhere out of the way? I don't want Splunk index data it's already indexed.

Thanks!

Tags (1)
2 Solutions

David
Splunk Employee
Splunk Employee

You can add a blacklist, telling splunk to ignore the compressed log files, e.g.:

blacklist = \.(gz|bz2|z|zip)$ 

(From http://www.splunk.com/base/Documentation/4.2/Data/HowLogFileRotationIsHandled )

Edit: Based on Steven Sorkin's reply, this is only true for 4.1.7 and below. 4.2 will not require a blacklist.

View solution in original post

Stephen_Sorkin
Splunk Employee
Splunk Employee

In Splunk 4.2, we will not re-index a log file it's been compressed as we compare the decompressed payload of the compressed file to our internal file tracking database. In Splunk 4.1 and before, the best practice is to blacklist the compressed files.

View solution in original post

Stephen_Sorkin
Splunk Employee
Splunk Employee

In Splunk 4.2, we will not re-index a log file it's been compressed as we compare the decompressed payload of the compressed file to our internal file tracking database. In Splunk 4.1 and before, the best practice is to blacklist the compressed files.

Branden
Builder

Thank you for the info!

0 Karma

David
Splunk Employee
Splunk Employee

That's good to know. Note that the documentation is out of date for that. ( http://www.splunk.com/base/Documentation/4.2/Data/HowLogFileRotationIsHandled )

0 Karma

David
Splunk Employee
Splunk Employee

You can add a blacklist, telling splunk to ignore the compressed log files, e.g.:

blacklist = \.(gz|bz2|z|zip)$ 

(From http://www.splunk.com/base/Documentation/4.2/Data/HowLogFileRotationIsHandled )

Edit: Based on Steven Sorkin's reply, this is only true for 4.1.7 and below. 4.2 will not require a blacklist.

Branden
Builder

I'm running 4.1.7 (for now). Looks like I'll need a blacklist.

Thank you for your help.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...