Trouble Indexing Multiple sourcetypes from a Single monitor

[jheilman]

I have a set of logs that no longer appear to be being indexed. I had originally configured the monitor as follows...

[monitor://D:\jboss-4.0.2\server\appname\log]
disabled = false
host = ServerName
index = default
sourcetype = log4j
whitelist = (boot|stderr|stdout|server|appname|appname-web-audit)\.log

This configuration seemed to work fine. I was getting data from all of the logs as expected.

The problem was that I have log4j, access_combined and a custom log type in this same folder. I've tried a couple of different solutions and neither seemed to work. I'm not sure if my syntax is right or where to get feedback from splunk as to if they are any good or not.

First attempt:

inputs.conf

[monitor://D:\jboss-4.0.2\server\appname\log\(boot|stderr|stdout|server|appname|appname-web-audit).log]
disabled = false
followTail = 0
host = ServerName
index = default
sourcetype = log4j

[monitor://D:\jboss-4.0.2\server\appname\log\appname-virtualhost_(\d\d\d\d-\d\d-\d\d).log]
disabled = false
followTail = 0
host = ServerName
index = default
sourcetype = access_combined

Second attempt:

inputs.conf

[monitor://D:\jboss-4.0.2\server\appname\log]
disabled = false
followTail = 0
host = ServerName
index = default

props.conf

[source::D:\\jboss-4.0.2\\server\\appname\\log\\(boot|stderr|stdout|server|appname|appname-web-audit).log]
sourcetype = log4j

[source::D:\\jboss-4.0.2\\server\\appname\\log\\appname-virtualhost_(\d\d\d\d-\d\d-\d\d).log]
sourcetype = access_combined

Neither of these approaches seems to work as I would expect it to. Am I not configuring this correctly? Is there a way to get feedback from splunk on problems with the configuration? If I switch back to the original configuration it seems to start indexing again.

This configuration is being used with splunk v4.1 as a full forwarder running on Windows.

[jrodman · Accepted solution]

In Splunk 4.1.x, both 'attempt' patterns are intended to work. There may be some outstanding issues with followTail, so you may want to evaluate what results you get without that.

You may want to try turning on some debug settings to get more insight, or work with Splunk Support.

In etc/log-local.cfg, you could turn these on.

• category.TailingProcessor=DEBUG
• category.WatchedFile=DEBUG
• category.BatchReader=DEBUG

You can also turn them on/off interactively from Manager.

This might be of use: http://www.splunk.com/wiki/Community:Troubleshooting_Monitor_Inputs#File_inputs

[gkanapathy]

inputs.conf:

[monitor://D:\jboss-4.0.2\server\appname\log]
host = ServerName
_whitelist = (boot|stderr|stdout|server|appname|appname-web-audit)\.log$

props.conf:

[source::(?i)D:\\jboss-4.0.2\\server\\appname\\log\\(boot|stderr|stdout|server|appname|appname-web-audit)\.log]
sourcetype = log4j

[source::(?i)D:\\jboss-4.0.2\\server\\appname\\log\\appname-virtualhost_(\d\d\d\d-\d\d-\d\d)\.log]
sourcetype = access_combined

Note that I corrected the name vitualhost that you had to virtualhost.