Route data to index based on host

[Yancy]

Hi folks,

I'd like to route WMI logs to different indexes based off the host name (I have a few environments)

Going off the directions here: http://www.splunk.com/support/forum:SplunkAdministration/3468 I'm not sure why this isn't working. My data from my remote collected WinEventLog keeps going into 'main'. Ideas?

props.conf

[host::foo*]
TRANSFORMS-foo = route_to_foo

transforms.conf

[route_to_foo]
SOURCE_KEY = _raw
REGEX = (?m)^wmi_type=(WinEventLog:System|WinEventLog:Application)
DEST_KEY = _MetaData:Index
FORMAT = foo

EDIT: Per suggestions this is now working as below

props.conf

[wmi]
TRANSFORMS-FIELDS = wmi-host, wmi-source, wmi-sourcetype, wmi-color

transforms.conf

default transform for wmi-host

[wmi-host]
REGEX = (?m)ComputerName=(.+)
DEST_KEY = MetaData:Host
FORMAT = $1

Environment host names are unique in the three characters after "FOO" ie for the BAR & BAD environment, host names would be "FOOBAR1", "FOOBAR2", "FOOBAD1", "FOOBAD2"

[wmi-color]
SOURCE_KEY = MetaData:Host
REGEX = (?m)FOO(.{3})(.+)
DEST_KEY = _MetaData:Index
FORMAT = $1

[jblaine]

I'm having a hell of a time with this, and it seems right to continue the thread here instead of posting a nearly identical question.

"All" I am trying to do is route ANY data (syslog, WMI, whatever ... anything) from host::ats-* to the index 'ats'

I have tried:

# props.conf
[host::ats-*]
TRANSFORMS-whatever = ats

with

# transforms.conf
[ats]
REGEX = .+
DEST_KEY = _MetaData:Index
FORMAT = ats

and the only thing I see in index=ats is some notifications from the Splunk forwarder(s) on ats-* hosts due to my restarting our main Splunk server. There is no WMI data showing up in index=ats, as it is all still going to index=main. This WMI data that I can see in index=main clearly shows fields of host=ats-17.our.org, etc.

Any advice would greatly be appreciated.

[Yancy]

Hi, it's probably a good idea to open a new question and link back to this one, as new questions are much more visible. This question has been answered, and while your scenario is similar, I'm not quite sure what the next steps would be.

[Ledio_Ago]

Another example redirecting Windows event logs using the log channel name:

http://answers.splunk.com/questions/970/how-do-i-configure-splunk-to-index-windows-event-log-data-in...

[gkanapathy]

FORMAT = index::$1 should be FORMAT = $1 in stanza [wmi-color]

[kbains · Accepted solution]

since the hostname is extract via a transform stanza itself, you cannot call the routing transform based on host. here are the configuration changes you should make:

props.conf:

[wmi]
TRANSFORMS-FIELDS = wmi-host, wmi-source, wmi-sourcetype, wmi-foo1, wmi-foo2

#notice the wmi-foo1 and wmi-foo2 transforms are called AFTER the wmi-host transform   

transforms.conf:

#make everything from this host go to a different index 
[wmi-foo1]
SOURCE_KEY = MetaData:Host
REGEX = <the host you want to route>
DEST_KEY = _MetaData:Index
FORMAT = foo

#now revert the index for the events you DON'T want to in the foo index for the host
[wmi-foo2]
SOURCE_KEY = _raw
REGEX = <some regex here based on raw text event>
DEST_KEY = _MetaData:Index
FORMAT = <non foo index>

[gkanapathy]

Oh, yeah, I just thought of another thing while commenting on another answer to this. By default, [wmi] sourcetype also does a TRANSFORM that sets the host, set in etc/system/default/props.conf and etc/system/default/transforms.conf. It's likely that it's running after your host one and therefore overwriting your change.

You might need to override that by setting:

[wmi]
TRANSFORMS-FIELDS = wmi-source, wmi-sourcetype
# the original was TRANSFORMS-FIELDS = wmi-host, wmi-source, wmi-sourcetype

Update:

oops sorry, got confused, thought you were trying to update "host" not "index". Ignore the override I said and look at kbains answer.

[Yancy]

Thanks, I think this is on the right track.. just not quite there yet.

Overriding the props.conf [wmi] per your suggestion, it now simply leaves the host as the Splunk Server. Index is still set to 'main'.

This is on 4.0.10

[Dan]

I would test with a simpler REGEX. It looks like yours might have unmatched parentheses. I'd also try matching on MetaData:Source as your SOURCE_KEY rather than _raw. Seems simpler and would have better performance.

[gkanapathy]

In the case of wmi inputs, unfortunately the source is simply set to "wmi", and is set during the same phase as this TRANSFORM.

[Simon]

Check the host field of the events in your 'main' index. Make sure it matches with your wildcard in props.conf (foo*).