splunk monitor file at the bottom of big directory structure

[mic1024]

Hi,
So I have following example directory structure:

/mnt/name/Logs/Grid/SITE1/version/20140409/QA/_Log.20140410080009.log

the part with SITE1 has multiple dirs called SITE2 SITE3 etc.

in each of the SITE folders there is number of version subfolders.

daily in each of these new date folder is created.

in each of dates folders daily LOADS of .gz files are being written ( I dont want to index them)

I'm interested in the contents of each QA subfolder that contains the log files.

so my stanza at the moment is (I use heavy forwarder) :

[monitor:///mnt/name/Logs/Grid/*/*/*/*/*.log]
blacklist = \.gz$
disabled = false
followTail = 0 
index = gridlog
sourcetype = gridlog
whitelist = _Log\.\d+\.log$
recursive=false

That doesnt seem to work. I'm getting no errors, but no files are getting indexed to.

splunk list monitor gives:

Monitored Directories:
monitor:///mnt/name/Logs/Grid/*/*/*/*/*.log

When I set recursive = true that is starting to scan all folders, which is not something I want to happen (there are around 2 mln .gz files within the structure).

When I set up direct path to a random file in inputs.conf, such as:

[monitor:///mnt/name/Logs/Grid/SITE1/version/20140409/QA/_Log.20140410080009.log]

this is going in fine and gets picked up and indexed.

Any ideas/suggestions?

thanks,
Mic

[mic1024]

good point. I should provide some feedback.
So the TailingProcessor:FileStatus helped me to figure out that no matter what all the files on the path to the place I was interested in monitored need to be looked at. To fix it I've changed the files structure, and logs are written to outside of 'data' files.

[piebob]

if somesoni2's response helped you figure out your problem, please provide it as an Answer below so others can benefit. thank you!

[somesoni2]

You can see what files being monitored (and their status) from below url.

https://your-splunk-server:8089/services/admin/inputstatus/TailingProcessor:FileStatus

[mic1024]

Hi somesoni2,
Yeah, that was my next logical step actually.
So nothing really happens until I go recursive=true, which also starts scanning all the other files that I'm not interested in.
Is there actually a way to display files that are being monitored?
splunk list monitor only shows directories.

[somesoni2]

Try this

[monitor:///mnt/name/Logs/Grid/*/*/*/QA/*.log]