Monitoring Splunk

3 monitor stanzas of the same folder but only one sourcetype getting indexed. Why?

Nicholas_Key
Splunk Employee
Splunk Employee

It seems that Splunk is only indexing the sourcetype defined last (in this example, it's WebSphere:SystemOutErrLog) when I'm monitoring the same folder but with different whitelists. I'm only seeing WebSphere:SystemOutErrLog being indexed. Is there a workaround to this issue?

[monitor://C:\Program Files (x86)\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\server1\]
sourcetype = WebSphere:NativeStdOutErrLog
whitelist = native(.*)\.log$
crcSalt = <SOURCE>
disabled = 0

[monitor://C:\Program Files (x86)\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\server1\]
sourcetype = WebSphere:StartStopServerLog
whitelist = startServer\.log$|stopServer\.log$
crcSalt = <SOURCE>
disabled = 0

[monitor://C:\Program Files (x86)\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\server1\]
sourcetype = WebSphere:SystemOutErrLog
whitelist = SystemOut\.log$|SystemErr\.log$
crcSalt = <SOURCE>
disabled = 0
1 Solution

Lowell
Super Champion

Out of curiosity, any reason why you aren't monitoring these with a single [monitor://] and then setting up sourcetype rules in props.conf. It seems that Splunk 4.1 is more flexibal than previous versions, but I'm still wondering which approach is better.


To be clear, here is what I'm suggesting:

inputs.conf:

[monitor://C:\Program Files (x86)\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\server1]
whitelist = (native.*|(start|stop)Server|System(Out|Err))\.log$

Side Note: BTW, I saw your question where someone recommended using the "crcSalt = <SOURCE>" entry, but you really don't want this setting when you are monitoring regular log files because using this setting will disable splunk's automatic detection and proper handling of rotated log files. Of course, if you change it now, your files may be indexed again, it's tricky. Which is also why it should be generally avoided.)

props.conf:

[source::...[/\\]WebSphere[/\\]...[/\\]native*.log]
sourcetype = WebSphere:NativeStdOutErrLog

[source::...[/\\]WebSphere[/\\]...[/\\](start|stop)Server.log]
sourcetype = WebSphere:StartStopServerLog

[source::...[/\\]WebSphere[/\\]...[/\\]System(Out|Error).log]
sourcetype = WebSphere:SystemOutErrLog

Note: I use the ackward looking expression [/\\] to match both windows and unix style paths (that way I can use the same rule on either systems). You may want to match more than just WebSpehere and the log name in your path, but this is just to give you a general idea. You probably don't want to match just the file name (without any other portion of the path) then could could have collisions with non-WebSphere logs files.


Of course, if you are not running 4.1 yet, then this would probably be your only option. I would be curious if anyone has any pros/cons to the multiple monitor approach vs the traditional props-based (source matching) approach?

BTW: I've used this an approach like this to get certain log files to get the correct sourcetypes for quite a number of configurations. When I first started using splunk I really wanted to use the multiple monitor approach (like what is now available in 4.1), however I now see the props-based approach as much more flexible. Think about it this way, when you add a new set of log files, do you want to have to remember the exact name of every sourcetype (especially if your doing this a few months down the road), or would you rather have existing rules setup that do the work for you.

Another advantage I see to the props-based approach is that you can interactively probe log files to determine there properties. This is great for new setups and for double-checking yourself after a config change. This feature has saved me TONS of time (and it works against the current config so you can test your props settings without restarting splunk.) To run the tool, open a command window, and cd to splunk's bin directory. Then run the following command:

splunk test sourcetype "<path to logfile>"

Ok, that was way too long. Sorry. Hopefully this was helpful.

View solution in original post

Lowell
Super Champion

Out of curiosity, any reason why you aren't monitoring these with a single [monitor://] and then setting up sourcetype rules in props.conf. It seems that Splunk 4.1 is more flexibal than previous versions, but I'm still wondering which approach is better.


To be clear, here is what I'm suggesting:

inputs.conf:

[monitor://C:\Program Files (x86)\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\server1]
whitelist = (native.*|(start|stop)Server|System(Out|Err))\.log$

Side Note: BTW, I saw your question where someone recommended using the "crcSalt = <SOURCE>" entry, but you really don't want this setting when you are monitoring regular log files because using this setting will disable splunk's automatic detection and proper handling of rotated log files. Of course, if you change it now, your files may be indexed again, it's tricky. Which is also why it should be generally avoided.)

props.conf:

[source::...[/\\]WebSphere[/\\]...[/\\]native*.log]
sourcetype = WebSphere:NativeStdOutErrLog

[source::...[/\\]WebSphere[/\\]...[/\\](start|stop)Server.log]
sourcetype = WebSphere:StartStopServerLog

[source::...[/\\]WebSphere[/\\]...[/\\]System(Out|Error).log]
sourcetype = WebSphere:SystemOutErrLog

Note: I use the ackward looking expression [/\\] to match both windows and unix style paths (that way I can use the same rule on either systems). You may want to match more than just WebSpehere and the log name in your path, but this is just to give you a general idea. You probably don't want to match just the file name (without any other portion of the path) then could could have collisions with non-WebSphere logs files.


Of course, if you are not running 4.1 yet, then this would probably be your only option. I would be curious if anyone has any pros/cons to the multiple monitor approach vs the traditional props-based (source matching) approach?

BTW: I've used this an approach like this to get certain log files to get the correct sourcetypes for quite a number of configurations. When I first started using splunk I really wanted to use the multiple monitor approach (like what is now available in 4.1), however I now see the props-based approach as much more flexible. Think about it this way, when you add a new set of log files, do you want to have to remember the exact name of every sourcetype (especially if your doing this a few months down the road), or would you rather have existing rules setup that do the work for you.

Another advantage I see to the props-based approach is that you can interactively probe log files to determine there properties. This is great for new setups and for double-checking yourself after a config change. This feature has saved me TONS of time (and it works against the current config so you can test your props settings without restarting splunk.) To run the tool, open a command window, and cd to splunk's bin directory. Then run the following command:

splunk test sourcetype "<path to logfile>"

Ok, that was way too long. Sorry. Hopefully this was helpful.

Nicholas_Key
Splunk Employee
Splunk Employee

lengthy answer --> yes
helpful answer --> yes

I am impressed that you suggested the regex [/\] to match Windows and *NIX path delimiters.

I'll give this a try.

0 Karma

Chris_R_
Splunk Employee
Splunk Employee

Try setting your inputs to debug mode, we will need to figure out why splunk's complaining on those files in your etc/log.cfg set these or input them if not already there.

category.TailingProcessor=DEBUG
category.WatchedFile=DEBUG
category.BatchReader=DEBUG
category.FileTracker=DEBUG

restart splunk, let it run a few minutes and check log/splunkd.log for any of those files like native*.log, splunk should be pretty verbose about what's wrong

Nicholas_Key
Splunk Employee
Splunk Employee

it seems that i have to configure props.conf to monitor the log files

0 Karma

jrodman
Splunk Employee
Splunk Employee

To splunk, you're defining the same input three times. We fold together stanzas with the same names, intentionally, to allow redefinitions from default to apps, to local settings.

Create independent stanzas by using wildcards as part of the stanza name (note, this is specific to 4.1).

cphair
Builder

What is the recommended solution on 4.3.2? Wildcards don't seem to work anymore.

0 Karma

jrodman
Splunk Employee
Splunk Employee

Pretty unreadable, but yes that should work fine, on 4.1.

Lowell's approach will work on 4.0 also.

0 Karma

Nicholas_Key
Splunk Employee
Splunk Employee

Are you suggesting:

[monitor://C:\Program Files (x86)\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\server1\native*.log]
sourcetype = WebSphere:NativeStdOutErrLog
crcSalt =
disabled = 0

[monitor://C:\Program Files (x86)\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\server1\st*Server.log]
sourcetype = WebSphere:StartStopServerLog
crcSalt =
disabled = 0

[monitor://C:\Program Files (x86)\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\server1\System*.log]
sourcetype = WebSphere:SystemOutErrLog
crcSalt =
disabled = 0

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...