Can you add a NOT condition in monitor path (inputs.conf)?

[ionicabalaurul]

I have 3 types of log file names, ones that simply end with .log.2018 (eg: dc1-sms.log.2018), others end with -error.log.2018 (eg: dc1-sms-error.log.2018), -access.log.2018(dc1-sms-access.log.2018).

I am trying to match files that end with:

.log.2018

or

-error.log.2018

and not

-access.log.2018

Basically a dc1-sms(!access)*.log.2018 would suffice, but obviously this does not work.

I tried several combinations ofr wildcards and regex, but without success.
I want to be able to do this from the [monitor://<path>] part of the config because the way our app is built, I only have access to that part of Splunk config.

[cpetterborg]

Try:

*(?<!-access).log.*

This is a regex form that "should" work. I have not tried it, but the syntax in pure PCRE would be:

.*(?<!access)\.log\..*

But Splunk does some manipulation of the string before processing it. That is why something like *.log works even though it is not a valid regular expression normally. I don't have the same environment that you have to be able to test it out, but it should at least get you looking in the right direction.

[ionicabalaurul]

Doesn't seem to be working

[cpetterborg]

So you are using only the [monitor:....] line designation of the files to be monitored, not even the whitelist (which is available through the "Add Data" link on the main Splunk page)? What I provided above works with the whitelist. If you have access to the whitelist, then you should have access to the blacklist, but if you don't have access to either, then you cannot do what you want because the [monitor:...] doesn't do regular expressions like the whitelist and blacklist. If that is your restriction, then you will have to designate each file individually, or accept that the -access files will also be indexed.

I did create an environment like you have described and the only way I got it to work was with the whitelist (I didn't try the blacklist, but it should work as well).

[somesoni2]

The inputs.conf monitoring stanza has blacklist attribute using which you can blacklist a file pattern from being ingested. You can try something like this

inputs.conf on the forwarder

[monitor://<Your log file folder>/*.log.*]
index= yourIndexerHere
sourcetype=yourSourcetypeHere
blacklist = -access\.log\.\d+$

Basically, monitor every file which is in form *.log.* but exclude any file with it's full path ending in *-access.log.<somenumber>. See this for more details:
http://docs.splunk.com/Documentation/Splunk/7.0.2/Data/Whitelistorblacklistspecificincomingdata

[ionicabalaurul]

As mentioned in the OP, I cannot use blacklist, only the monitor path

[gcusello]

Hi ionicabalaurul,
did you tried with whitelists and blacklists?
see http://docs.splunk.com/Documentation/Splunk/7.0.2/admin/Inputsconf

Anyway, try something like this

[monitor:///<your_path/*.log.2018]
index = my_index
sourcetype = my_sourcetype
blacklist = .*-access\.log\.2018

Bye.
Giuseppe

[ionicabalaurul]

As mentioned in the OP, I cannot use blacklist, only the monitor path

[gcusello]

Hi ionicabalaurul,
if you cannot use blacklists, the only way is to create more stanzas finding some rules for your files with the extension *.log.2018:
extension *-error.log.2018 hasn't any problem so you can create a monitor stanza with

[monitor:///<your_path>/*-error.log.2018]

instead for the other files you have to find more rules, e,g,:

[monitor:///<your_path>/dc*-*.log.2018]

and

[monitor:///<your_path>/dhcp*-*.log.2018]

and so on

Otherwise you could take all files

[monitor:///<your_path>/*.log.2018]

and then filter them in the Indexers or Heavy Forwarders
see http://docs.splunk.com/Documentation/Splunk/7.0.2/Forwarding/Routeandfilterdatad

Bye.
Giuseppe