Getting Data In

inputs.conf whitelist blacklist question

ebailey
Communicator

Greetings

I have trying to gather logs by sifting through three levels of the file system with a white list and blacklist. It is not working and outside of creating a very long list of monitors I am not sure what to do anymore.

Any help is much appreciated.

Thanks!

My config

[monitor:///opt/transient/prod/data/xxxxxx/xxx/xx_ci/log/]
recursive = true
sourcetype=prd_xxx_xx_log
disabled = false
crcSalt =
whitelist = .dat$|.log$|.err$|.txt$
blacklist = .gz$

[monitor:///opt/transient/prod/data/xxxxxx/xxx/xxx_ci/error/]
recursive = true
sourcetype=prd_xxx_xx_error
disabled = false
crcSalt =
whitelist = .dat$|.log$|.err$|.txt$
blacklist = .gz$

Any ideas?

0 Karma
1 Solution

lguinn2
Legend

Yes! Ditch the blacklist. You only need the whitelist. You also don't need the recursive - that's the default.

Finally - why do you have the crcSalt?

View solution in original post

0 Karma

neelamssantosh
Contributor

why to give 'blacklist' of Specific extensions of compressed files to exclude, where splunk already ignores..

packed_extensions_list:
bz, bz2, tbz, tbz2, Z, gz, tgz, tar, zip

0 Karma

MuS
SplunkTrust
SplunkTrust

hmmm, this packed_extensions_list is an option to crawl.conf only http://docs.splunk.com/Documentation/Splunk/6.1.3/admin/Crawlconf

and here would be the correct statement regarding compressed files from http://docs.splunk.com/Documentation/Splunk/6.1.3/Data/MonitorFilesandDirectories

  • Splunk Enterprise decompresses archive files before it indexes them. It can handle these common archive file types: tar, gz, bz2, tar.gz, tgz, tbz, bz2, zip, and z.

lguinn2
Legend

Yes! Ditch the blacklist. You only need the whitelist. You also don't need the recursive - that's the default.

Finally - why do you have the crcSalt?

0 Karma

ebailey
Communicator

Thanks!

I use crcSalt with the expectation it is going to be needed. I pulled it out.

Question - I made the changes you recommended and the results look good except now I am picking up files such as

test.dat.lock also I am picking up all . files such as

.work

Any ideas?

Thanks

Ed

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...