Getting Data In

Trouble Indexing Multiple sourcetypes from a Single monitor

jheilman
Explorer

I have a set of logs that no longer appear to be being indexed. I had originally configured the monitor as follows...

[monitor://D:\jboss-4.0.2\server\appname\log]
disabled = false
host = ServerName
index = default
sourcetype = log4j
whitelist = (boot|stderr|stdout|server|appname|appname-web-audit)\.log

This configuration seemed to work fine. I was getting data from all of the logs as expected.

The problem was that I have log4j, access_combined and a custom log type in this same folder. I've tried a couple of different solutions and neither seemed to work. I'm not sure if my syntax is right or where to get feedback from splunk as to if they are any good or not.

First attempt:

inputs.conf

[monitor://D:\jboss-4.0.2\server\appname\log\(boot|stderr|stdout|server|appname|appname-web-audit).log]
disabled = false
followTail = 0
host = ServerName
index = default
sourcetype = log4j

[monitor://D:\jboss-4.0.2\server\appname\log\appname-virtualhost_(\d\d\d\d-\d\d-\d\d).log]
disabled = false
followTail = 0
host = ServerName
index = default
sourcetype = access_combined

Second attempt:

inputs.conf

[monitor://D:\jboss-4.0.2\server\appname\log]
disabled = false
followTail = 0
host = ServerName
index = default

props.conf

[source::D:\\jboss-4.0.2\\server\\appname\\log\\(boot|stderr|stdout|server|appname|appname-web-audit).log]
sourcetype = log4j

[source::D:\\jboss-4.0.2\\server\\appname\\log\\appname-virtualhost_(\d\d\d\d-\d\d-\d\d).log]
sourcetype = access_combined

Neither of these approaches seems to work as I would expect it to. Am I not configuring this correctly? Is there a way to get feedback from splunk on problems with the configuration? If I switch back to the original configuration it seems to start indexing again.

This configuration is being used with splunk v4.1 as a full forwarder running on Windows.

Tags (2)
0 Karma
1 Solution

jrodman
Splunk Employee
Splunk Employee

In Splunk 4.1.x, both 'attempt' patterns are intended to work. There may be some outstanding issues with followTail, so you may want to evaluate what results you get without that.

You may want to try turning on some debug settings to get more insight, or work with Splunk Support.

In etc/log-local.cfg, you could turn these on.

  • category.TailingProcessor=DEBUG
  • category.WatchedFile=DEBUG
  • category.BatchReader=DEBUG

You can also turn them on/off interactively from Manager.

This might be of use: http://www.splunk.com/wiki/Community:Troubleshooting_Monitor_Inputs#File_inputs

View solution in original post

0 Karma

jrodman
Splunk Employee
Splunk Employee

In Splunk 4.1.x, both 'attempt' patterns are intended to work. There may be some outstanding issues with followTail, so you may want to evaluate what results you get without that.

You may want to try turning on some debug settings to get more insight, or work with Splunk Support.

In etc/log-local.cfg, you could turn these on.

  • category.TailingProcessor=DEBUG
  • category.WatchedFile=DEBUG
  • category.BatchReader=DEBUG

You can also turn them on/off interactively from Manager.

This might be of use: http://www.splunk.com/wiki/Community:Troubleshooting_Monitor_Inputs#File_inputs

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

inputs.conf:

[monitor://D:\jboss-4.0.2\server\appname\log]
host = ServerName
_whitelist = (boot|stderr|stdout|server|appname|appname-web-audit)\.log$

props.conf:

[source::(?i)D:\\jboss-4.0.2\\server\\appname\\log\\(boot|stderr|stdout|server|appname|appname-web-audit)\.log]
sourcetype = log4j

[source::(?i)D:\\jboss-4.0.2\\server\\appname\\log\\appname-virtualhost_(\d\d\d\d-\d\d-\d\d)\.log]
sourcetype = access_combined

Note that I corrected the name vitualhost that you had to virtualhost.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...