Getting Data In

Unexpected precedence of monitor stanzas in inputs.conf

remeika
Explorer

I have two monitor stanzas to watch nginx access logs: a specific stanza to route a team's error logs to their specific index, and another fallback stanza to catch any error logs not routed to a specific index:

$ splunk cmd btool inputs list
...
[monitor:///var/log/nginx/*batman*error.log]
_rcvbuf = 1572864
host = p2....00a
index = prod-batman
sourcetype = nginx-error
...
[monitor:///var/log/nginx/*error.log]
_rcvbuf = 1572864
host = p2....00a
index = prod-fallback
sourcetype = nginx-error

My intention is that the file /var/log/nginx/batman-service-a-error.log is routed to index prod-batman, while the file /var/log/nginx/other-team-service-a-error.log is routed to prod-fallback. But this is not happening. I see:

$ splunk list monitor
Monitored Directories:
    ...
    /var/log/nginx/*error.log
        /var/log/nginx/batman-service-a-error.log
        /var/log/nginx/batman-service-b-error.log
        /var/log/nginx/batman-service-c-error.log

Indeed, there is no entry for /var/log/nginx/*batman*error.log in the output of splunk list monitor. Is there any way to force the stanza [monitor:///var/log/nginx/*batman*error.log] to take precedence over [monitor:///var/log/nginx/*error.log]?

0 Karma
1 Solution

micahkemp
Champion

Put *batman*error.log in the blacklist for your *error.log stanza. From inputs.conf spec:

blacklist = <regular expression>
* If set, files from this input are NOT monitored if their path matches the
  specified regex.

View solution in original post

christianvalin
Explorer

In network router configurations, the most specific match usually wins. As for inputs.conf "monitor" stanzas, its usually the least specific match that wins / take precendence if two or more match the wildcard. This makes huge difference in hostname matching when processing syslog directories, for example.

0 Karma

remeika
Explorer

for inputs.conf "monitor" stanzas, its usually the least specific match that
wins / take precendence if two or more match the wildcard

Could you be more specific about your use of the word "usually"? Is there any documentation to explain this?

0 Karma

micahkemp
Champion

Put *batman*error.log in the blacklist for your *error.log stanza. From inputs.conf spec:

blacklist = <regular expression>
* If set, files from this input are NOT monitored if their path matches the
  specified regex.

remeika
Explorer

Hopefully there is a cleaner way to do this out there, but this does work.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...