Getting Data In

Inputs.conf with wildcards

peter_gianusso
Communicator
[monitor://\\njros1bva0597\d$\LogFiles\W3SVC1\*.log]
disabled = false
host = NJROS1BVA0621
alwaysOpenFile = 1
sourcetype = Image Importer Logs

With the stanza above, splunk reports the number of files is 3. There are only 2 files in the directory!! Also we can get it to actual index any of the files in the directory. There are only 2 files in the directory and they both end in .log

Please help

Tags (2)
0 Karma
1 Solution

lguinn2
Legend

From the manual: "Warning: In Windows, you cannot currently use a wildcard at the root level." Which is what you are doing. And to answer cmonig, I believe that the double-backslash \\ is only required in whitelists and blacklists, not the monitor stanza itself. (And you don't necessarily need a backslash in a whitelist, as it is a regular expression.)

The following should fix the problem:

[monitor://\\njros1bva0597\d$\LogFiles\W3SVC1]
disabled = false
host = NJROS1BVA0621
alwaysOpenFile = 1
sourcetype = Image Importer Logs
whitelist=.log$

I don't understand why you can't use whitelists. If you can't, then you are stuck with monitoring everything in the directory or finding another way around this AFAIK.

I would personally not use a sourcetype with spaces, but that's up to you. Also, the number of files that Splunk is monitoring may also include directories in the path, I think. To see what is actually being monitored, try this

$SPLUNK_HOME\bin\splunk list monitor

And Splunk will give you a list of the files it is monitoring - it may be a longer list than you expect!

Good move on checking the internal logs for the tailing status.

Updated: you said in the comments: "I want the ROUTEDB log files to get one source type and UPDATEDB to get another source type. I don't want or need the remaining 10 files." You can override the sourcetype setting as needed in props.conf - more info here in the middle of the page at Specify sourcetype for source. Perhaps this would help you so that you can use a whitelist and not violate the rule about multiple stanzas with the same path.

If the ultimate problem is the $ in your path name, can you create a link and use it instead? For example, create a link to the d$ directory named d.dollar, and then make the monitor stanza look like this:

[monitor://\\njros1bva0597\d.dollar\LogFiles\W3SVC1]

View solution in original post

lguinn2
Legend

From the manual: "Warning: In Windows, you cannot currently use a wildcard at the root level." Which is what you are doing. And to answer cmonig, I believe that the double-backslash \\ is only required in whitelists and blacklists, not the monitor stanza itself. (And you don't necessarily need a backslash in a whitelist, as it is a regular expression.)

The following should fix the problem:

[monitor://\\njros1bva0597\d$\LogFiles\W3SVC1]
disabled = false
host = NJROS1BVA0621
alwaysOpenFile = 1
sourcetype = Image Importer Logs
whitelist=.log$

I don't understand why you can't use whitelists. If you can't, then you are stuck with monitoring everything in the directory or finding another way around this AFAIK.

I would personally not use a sourcetype with spaces, but that's up to you. Also, the number of files that Splunk is monitoring may also include directories in the path, I think. To see what is actually being monitored, try this

$SPLUNK_HOME\bin\splunk list monitor

And Splunk will give you a list of the files it is monitoring - it may be a longer list than you expect!

Good move on checking the internal logs for the tailing status.

Updated: you said in the comments: "I want the ROUTEDB log files to get one source type and UPDATEDB to get another source type. I don't want or need the remaining 10 files." You can override the sourcetype setting as needed in props.conf - more info here in the middle of the page at Specify sourcetype for source. Perhaps this would help you so that you can use a whitelist and not violate the rule about multiple stanzas with the same path.

If the ultimate problem is the $ in your path name, can you create a link and use it instead? For example, create a link to the d$ directory named d.dollar, and then make the monitor stanza look like this:

[monitor://\\njros1bva0597\d.dollar\LogFiles\W3SVC1]

peter_gianusso
Communicator

Thanks. I spoke with them again and we are going with the props.conf file. They acknowledged that splunk struggles with the $ in the path when you have a wildcard as well.

0 Karma

lguinn2
Legend

You are correct - you can't have more than one monitor on the same path. But if there are only text files and binaries in the directory, along with the log files - why would you have more than one monitor stanza with the same path?

0 Karma

peter_gianusso
Communicator

First, they told me to use whitelists. But then, based on my experience, you can't have more than 1 monitor with the same path. Then they told me to use wildcards. I have been trying to get this simple example going before I tackle the real problem.

I have confirmed that the $ is the issue. I just don't know how to fix it.

monitor://\njros1bva0597\dshare\LogFiles\W3SVC1*.log
or
monitor://\njros1bva0597\dshare\LogFiles\W3SVC1\UPDATEDB*.log

All work just great.

It's our standard of creating shares like D$ that seems to be the issue.

0 Karma

peter_gianusso
Communicator

I don't think I can use whitelists because the Splunk SE guys told me that I couldn't.

In the end, I have 20 files in the directory. 5 that are named UPDATEDB-MM-DD-YY.log and 5 that are named ROUTEDB-MM-DD-YY.log. The remaining 10 are text files and binaries.

I want the ROUTEDB log files to get one source type and UPDATEDB to get another source type. I don't want or need the remaining 10 files.

0 Karma

cmonig
Explorer

Hi,

I'm not sure if this is really necessary, but don't you have to escape the backslashes in the monitor stanza?

[monitor://\\njros1bva0597\\d$\\LogFiles\\W3SVC1\\*.log]

Also, you could try to remove the dot after the wildcard, and see if that helps:

[monitor://\\njros1bva0597\\d$\\LogFiles\\W3SVC1\\*log]

Good luck!
:-)

Cheers,

Christoph

0 Karma

aholzer
Motivator

Have you read these two similar questions:
http://splunk-base.splunk.com/answers/2775/regexs-and-windows-paths-in-inputsconf-and-propsconf
&
http://splunk-base.splunk.com/answers/26094/whats-the-syntax-for-monitoring-a-local-windows-director...

It probably gets really tricky when you are trying to use the administrative pathing to a different box. Have you considered installing a universal forwarder directly on the box you are trying to monitor files from?

peter_gianusso
Communicator

escaping the $ did not work...didn't even recognize the input

went back to my original config that was in the question. Here's some more info from the tailing status

parent \njros1bva0597\d$\LogFiles\W3SVC1*.log
type File did not match whitelist '^\\njros1bva0597\d$\LogFiles\W3SVC1\[^\]*.log$'.

0 Karma

aholzer
Motivator

try escaping the dollar sign: \$

0 Karma

peter_gianusso
Communicator

tried this
monitor://\njros1bva0597\d$\LogFiles\W3SVC1\*log

I think the problem is our share d$...i think when everything get converted to regex the dollar sign messes things up but we don't know to fix

0 Karma

peter_gianusso
Communicator

we cannot use whitelisting due to another splunk limitation

0 Karma

peter_gianusso
Communicator

for some reason the splunkbase is removing the backslashes in the path!!! Bottom line we want all files ending in .log in the W3SVC1 directory

0 Karma

peter_gianusso
Communicator

This is the stanza

[monitor://\njros1bva0597\d$\LogFiles\W3SVC1*.log]
disabled = false
host = NJROS1BVA0621
alwaysOpenFile = 1
sourcetype = Image Importer Logs

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...