Getting Data In

Will Splunk re-index a log file if I compress it after it's been indexed?

Branden
Builder

I have an environment set-up such that syslog-ng redirects syslog to my central log host in this format: /server/host/year/month/day/syslog.log

Splunk monitors that directory tree and indexes it. No big deal.

I'd now like to compress older syslog files and leave the compressed file in that directory. So /server/foo/2010/05/04/syslog.log would become /server/foo/2010/05/04/syslog.log.gz.

I know Splunk can index compressed files. If I compress the syslog file, will the monitor detect the 'new' compressed file and attempt to index it? Or do I have to move the compressed logs somewhere out of the way? I don't want Splunk index data it's already indexed.

Thanks!

Tags (1)
2 Solutions

David
Splunk Employee
Splunk Employee

You can add a blacklist, telling splunk to ignore the compressed log files, e.g.:

blacklist = \.(gz|bz2|z|zip)$ 

(From http://www.splunk.com/base/Documentation/4.2/Data/HowLogFileRotationIsHandled )

Edit: Based on Steven Sorkin's reply, this is only true for 4.1.7 and below. 4.2 will not require a blacklist.

View solution in original post

Stephen_Sorkin
Splunk Employee
Splunk Employee

In Splunk 4.2, we will not re-index a log file it's been compressed as we compare the decompressed payload of the compressed file to our internal file tracking database. In Splunk 4.1 and before, the best practice is to blacklist the compressed files.

View solution in original post

Stephen_Sorkin
Splunk Employee
Splunk Employee

In Splunk 4.2, we will not re-index a log file it's been compressed as we compare the decompressed payload of the compressed file to our internal file tracking database. In Splunk 4.1 and before, the best practice is to blacklist the compressed files.

Branden
Builder

Thank you for the info!

0 Karma

David
Splunk Employee
Splunk Employee

That's good to know. Note that the documentation is out of date for that. ( http://www.splunk.com/base/Documentation/4.2/Data/HowLogFileRotationIsHandled )

0 Karma

David
Splunk Employee
Splunk Employee

You can add a blacklist, telling splunk to ignore the compressed log files, e.g.:

blacklist = \.(gz|bz2|z|zip)$ 

(From http://www.splunk.com/base/Documentation/4.2/Data/HowLogFileRotationIsHandled )

Edit: Based on Steven Sorkin's reply, this is only true for 4.1.7 and below. 4.2 will not require a blacklist.

Branden
Builder

I'm running 4.1.7 (for now). Looks like I'll need a blacklist.

Thank you for your help.

0 Karma
Get Updates on the Splunk Community!

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Wednesday, May 29, 2024  |  11AM PST / 2PM ESTRegister now and join us to learn more about how you can ...

Get Your Exclusive Splunk Certified Cybersecurity Defense Engineer Certification at ...

We’re excited to announce a new Splunk certification exam being released at .conf24! If you’re headed to Vegas ...