Multiple sourcetypes for different kind of logs

[asarolkar]

We are using the Universal lightforwarder on a linux box and pushing the monitored output for the several log files onto an indexer that is a part of a full fledged splunk instance on a dedicated splunk Windows box.

[ Unix box -> Application Server Logs->Splunk LF ] --- talks to --- [ Splunk instance]

We have figured out a way to channel all of the extracts to a single sourcetype.

Now we are looking to create and segregate into sourcetypes based on the type of log

The linux configuration file (inputs.conf) located at usr/local/splunkforwarder/etc/apps/Splunk_TA_nix/local/inputs.conf has the following current content and this works (this is setup to read the Standard JBoss Application server files - server.log and boot.log and the apache access log file) :

[monitor:///usr/local/jboss/server/default/log]

whitelist=(server*.log.$|boot.log$| localhost_access_log..log$)

disabled=0

index=os

sourcetype = onelog

But I when I segregate like this - it fails :

[monitor:///usr/local/jboss/server/default/log]

whitelist=(server*.log.*$)

disabled=0 index=os

sourcetype = serverlog

[monitor:///usr/local/jboss/server/default/log]

whitelist=(boot*.log$)

disabled=0 index=os

sourcetype = bootlog

[monitor:///usr/local/jboss/server/default/log]

whitelist=(localhost_access_log.*.log$)

disabled=0

index=os

sourcetype = accesslog

What this does is that it will ignore the first two - and ONLY capture the third (access log) logs.

Analysis:
The reason why it does THAT is because it will extract the last whitelist of the last declaration of this monitor and then ignore the first two - because it only cares for - and matches against the regular expression in the LAST entry.

Any input would be appreciated on how to get this up and running.

Another alternative would be to be able to use transforms.conf to extract these on the INDEXER side (?)

[asarolkar]

Many thanks !

[asarolkar]

ha.. i cleaned it up and refreshed.. and lo and behold its working again !

[asarolkar]

It goes to a certain JBoss Profile - location /server/cal_pt

boot log
[monitor:///usr/local/jboss/server/cal_pt/log/boot*]
disabled = 0
index = os
sourcetype = bootlog

server log

[monitor:///usr/local/jboss/server/cal_pt/log/server*]
disabled = 0
index = os
sourcetype = serverlog

access log
[monitor:///usr/local/jboss/server/cal_pt/log/localhost_access_log*]
disabled = 0
index = os
sourcetype = accesslog

[asarolkar]

Unfortunately that did not work out.

This time around Splunk did not capture anything actually.

[Lamar]

That's bizarre, it works for me.

Can you share you inputs?

[Lamar]

As opposed to doing it with whitelists, just use something like this:

[monitor:///usr/local/jboss/server/default/log/server*]
disabled=0 
index=os
sourcetype = serverlog

[monitor:///usr/local/jboss/server/default/log/boot*]
disabled=0 
index=os
sourcetype = bootlog

[monitor:///usr/local/jboss/server/default/log/localhost_access_log*]
disabled=0
index=os
sourcetype = accesslog

NOTE: Splunk doesn't necessarily work well when it comes to using whitelist/blacklists on a single directory.