Whitelist a certain gz file extension from a monitoring directory

[mmohiuddin1512]

Hi All:

I want to monitor certain files on April 26th 2017. There was an outage in our environment, and these logs come through a syslog server that has a UF installed. The syslog server retains logs on its system for 1 hour, then the logs are rotated and stored as backup with a .gz extension file residing in the same directory. I would like to blacklist all the .gz extension files as they represent backup of logs, except the ones on the 26th and 27th of April when we had an outage and we did not receive logs due to them being rotated and residing as .gz file extension logs. Also the fact that monitoring is disabled (blacklisted) for .gz files.

The actual monitoring file stanza looks as follows:

[monitor:///var/log/hosts///]
sourcetype = syslog
host_segment=4
index = network
blacklist = .(gz|bz2|z|zip)$
ignoreOlderThan = 7d
crcSalt =

I would like to monitor files something that looks like :

/var/log/hosts/56./Mon/56.0.0.0-2017042604.gz
/var/log/hosts/56./Mon/56.0.0.0-2017042704.gz

Is there a way to create a whitelist only for these files and blacklist other gz file extensions.

Your help will be highly appreciated.

Obaid

[richgalloway]

You might be able to whitelist those files, but then you'd be blocked by the ignoreOlderThan setting.
I suggest moving/copying the desired files to a new directory then setting up a new monitor stanza for that directory. Something like this:

[monitor:///var/log/hosts///backfill]
sourcetype = syslog
host_segment=4
index = network
crcSalt =

[mmohiuddin1512]

If we remove ignore older than stanza, then is there a way to whitelist only the gz extension files? The problem is that there are more than 500 files with the .gz extension as we are getting data from 430 directories on this server and it becomes a cumbersome process to move these files to a new directory.

[richgalloway]

Don't touch your existing stanza. Create a NEW stanza to monitor a NEW directory. Unzip only the two files you want to ingest into that directory.