Does monitoring similar files within a directory log require a separate props.conf configuration?

[centrafraserk]

Hello, I am struggling with a directory monitoring problem. I have a directory with a ton of different incremental log names. It's pretty easy to get all of these with the following in inputs.conf:

[monitor://path_to_directory\*.*]
sourcetype = source:one
index = index01

Now the problem becomes when there is one log that is very similar to all of the other logs that is not indexed correctly without a custom props.conf stanza to correctly format the log. For instance if my logs looks like:

server_log_trace.01.log
server_log_trace.02.log
server_log_sdk_trace.02.log
server_log_request_trace.02.log
app_log_trace.01.log
app_log_trace.02.log
...

It is the sdk log that needs to formatted differently. Imagine there are 100 logs all with different variations, however only a very specific few that need the custom formatting, each which contains sdk. I was thinking this should be simple by adding another monitor stanza in inputs.conf:

[monitor://path_to_directory\*.*]
sourcetype = source:one
index = index01

[monitor://path_to_directory\*sdk*.*]
sourcetype = source:two
index = index01

And then creating the proper [source:two] stanza in props.conf to achieve the formatting. However it appears that this does not work as I was expecting. It does not incorrectly format the sdk logs, it just simply makes splunk not monitor the sdk log at all. It doesn't matter the order of the monitor stanzas in inputs.conf....either way it only monitors all files NOT containing sdk. This doesn't make sense to me. Is there a simple way of monitoring this file that I am not thinking of? There are simply too many variations and different logs that creating 30-40 different monitoring stanzas to specifically target each type individually (imagine 100 more logs with variations on the names).

Finally, if I specify only to monitor the sdk logs...it does it correctly. So the input and props stanzas are correct and would work just fine without the star(dot)star monitor stanza. Any thoughts? Ideally there would be a way to create a stanza that said monitor where files != sdk and a stanza that says monitor files = sdk, but I don't think that syntax is available in a [monitor://] format.

[JohnRiddoch]

We have our httpd logs in /var/log/httpd, but some are of a different format. However, they do have consistent naming options, so we have:
in /opt/splunkforwarder/etc/apps/search/local/inputs.conf:

[monitor:///var/log/httpd/*_log]
disabled = false

so all files named "*_log" in the folder are monitored, but they might not get the right sourcetype. Within props.conf, we have:

[source::/var/log/httpd/http...error_log]
sourcetype=apache_error

[source::/var/log/httpd/http...access_log]
sourcetype=access_common

[source::/var/log/httpd/...track_log]
sourcetype=track_log

That sets the appropriate sourcetype on each. Note that props.conf uses "..." as a wildcard, NOT "*" which might be your problem.

[teunlaan]

We have the same problem, but only a hand full of variations. So we made multiple monitor stanzas.

Just wondering what would happen if you blacklist the sdk in the first one

[dineshraj9]

Yes, you have to blacklist the sdk files in the first stanza and it should work.

blacklist = ((.+sdk_trace.+|\.(tar|gz|bz2|tar.gz|tgz|tbz|tbz2|zip|z)$)