I have setup a monitor for '/opt/vimana/tenant/*/log/vimana.log'. When I do 'splunk list monitor' I get this:
Monitored Directories:
/opt/vimana/tenant/*/log/vimana.log
/opt/vimana/tenant/x1111
/opt/vimana/tenant/x1111/log
/opt/vimana/tenant/x2222
/opt/vimana/tenant/x2222/log
/opt/vimana/tenant/x3333
/opt/vimana/tenant/x3333/log
/opt/vimana/tenant/x4444
/opt/vimana/tenant/x4444/log
/opt/vimana/tenant/x5555/log/vimana.log
/opt/vimana/tenant/x6666
/opt/vimana/tenant/x6666/log
/opt/vimana/tenant/x7777
/opt/vimana/tenant/x7777/log
I don't understand this at all. Every one of these log directories contains a file called 'vimana.log', but only one is listed above and for that tenant it does not list the directories. All of the files are being appended to by apps, but nothing gets forwarded to storm.
How can I debug this? Does splunk have any local log files that I can check to see what's really doing?
I think I have an answer. For all of the above I had not yet opened the firewall to storm. Once I did that using the original wildcard every tenant showed a log file being monitored (albeit under the directories section).
As to my other question, how to monitor what the forwarder was doing:
$SPLUNK/var/log/splunk/splunkd.log
Hmm, so I executed the following
$ splunk add monitor /opt/vimana/tenant/x7777/log/vimana.log
And now the list monitor looks like:
Monitored Directories:
/opt/vimana/tenant/x7777/log/vimana.log
Monitored Files:
/opt/vimana/tenant/*/log/vimana.log
And I thought was confused before...