Getting Data In

Why are so many files being tracked because of this one monitor?

Ricapar
Communicator

Copying everything exactly how it appears...

I have this in my inputs.conf:

[monitor:///opt/firewalker/data/*/*/make_me_compliant.log]
disabled = 0
followTail = 0
host_segment = 5
index = firewalker
sourcetype = make_me_compliant
crcSalt = <SOURCE>

I took a look at $SPLUNK_HOME/var/log/splunk/splunkd.log, and I see this flying all over the place:

11-21-2012 15:21:06.422 -0500 INFO  WatchedFile - Will use tracking rule=modtime for file='/opt/firewalker/data/AIX/myhostname/pam.conf'.
11-21-2012 15:21:06.424 -0500 INFO  WatchedFile - Will use tracking rule=modtime for file='/opt/firewalker/data/AIX/myhostname/resolv.conf'.
11-21-2012 15:21:06.426 -0500 INFO  WatchedFile - Will use tracking rule=modtime for file='/opt/firewalker/data/AIX/myhostname/ssh.config'.
11-21-2012 15:21:06.427 -0500 INFO  WatchedFile - Will use tracking rule=modtime for file='/opt/firewalker/data/AIX/myhostname/sshd.config'.
11-21-2012 15:21:06.428 -0500 INFO  WatchedFile - Will use tracking rule=modtime for file='/opt/firewalker/data/AIX/myhostname/syslog.conf'.
11-21-2012 15:21:06.429 -0500 INFO  WatchedFile - Will use tracking rule=modtime for file='/opt/firewalker/data/AIX/myhostname/vintella.vas.conf'.
11-21-2012 15:21:06.429 -0500 INFO  WatchedFile - Will use tracking rule=modtime for file='/opt/firewalker/data/AIX/myhostname/vintella.vgp.conf'.

Is this normal operation? There are at least a thousand files for each hostname folder in that directory structure.
I only want to monitor one of them. It seems extremely wasteful that Splunk watches every file in there.

I verified that it was that stanza that is causing Splunk to do this. I disabled it (from the Web GUI), and the logs stopped getting those lines. Waited a while.. nothing. I enabled that input, and the entires came back pretty much right away.

That being said, I don't see data from any of those files showing up in any of my indexes, so Splunk doesn't seem to be doing much with it.

0 Karma
1 Solution

Drainy
Champion

Well, its not tailing them so I wouldn't worry but the fact you've used wildcards means that Splunk does have to scan through all files to locate the one you've specified, you may as well give the full location to it if you want to specify a single file.

View solution in original post

Drainy
Champion

Well, its not tailing them so I wouldn't worry but the fact you've used wildcards means that Splunk does have to scan through all files to locate the one you've specified, you may as well give the full location to it if you want to specify a single file.

lguinn2
Legend

You should use a whitelist to specify the specific file name you want to monitor. This will make Splunk more efficient. Otherwise, as Drainy says, you will be scanning through the directories unnecessarily.

0 Karma

Drainy
Champion

Since you haven't given it the full filename it will need to locate the files first. If it isn't tailing or saying that it has started to read at offset etc then it isn't reading the file.

0 Karma

Ricapar
Communicator

Ah, sorry, maybe I was a little vague..

There's one file name I want to watch, and that file name happens once per directory. That's what the wildcards are matching there.

Does the use of wildcards there make it scan every file in every one of those directories though?

I have a few other wildcard monitors.. and none of them go scanning every single other file in the tree.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...