I use several SplunkLightFirwarders on Suslog-ng servers to have a "buffer" to relatively large amounts of syslog that are then forwarded to the main indexer.
Several different corporate units need separate acces restrictions and different retention time. so i would like to put data from different host:: in separate indexes.
as far as I found in the Docunentation the Light Forwarder do not take any routing desicions. and i therefore put following config on the main indexer:
[host::192.168.9*] TRANSFORMS-routing = route_to_groupA [host::192.168.13.*] TRANSFORMS-routing = route_to_groupB
[route_to_groupA] DEST_KEY = _MetaData:Index FORMAT = index_groupA [route_to_groupB] DEST_KEY = _MetaData:Index FORMAT = index_groupB
Put the same config on the SplunkLightForwarder which made no change to my result.
Not working ...
vidar
Thanks for all the help here. There are obviously several ways to solve this problem . I wanted to separate log into separate indexes based on which host generated the event.
Since it was indicated to me that the host:: tag was already put in there by the LWF , i used some time to try to make that work. to no avail. ( i really miss some "sniffing" tools to see what are actually see what king of data and tags are communicated between Indexer, FW and LWF )
Since the source tag are available from the LWF My solution became this.:
props.conf:
[source::/var/log/syslog]
TRANSFORMS-routing = route_to_a,route_to_b,route_to_c
transforms.conf:
[route_to_a]
DEST_KEY = _MetaData:Index
REGEX = ^\S+\s+\d+\s+\S+\s+192\.168\.111\.19
FORMAT = index_a
[route_to_b]
DEST_KEY = _MetaData:Index
REGEX = ^\S+\s+\d+\s+\S+\s+192\.168\.111\.86
FORMAT = index_b
[route_to_c]
DEST_KEY = _MetaData:Index
REGEX = ^\S+\s+\d+\s+\S+\s+192\.168\.111\.85
FORMAT = index_c
`
The example I gave in a related question works for directing sourcetypes to a specific index for data coming from a LWF. I assume it would work equally well for subnets.
You mentioned syslog-ng. Assuming your host is the path for monitor input, you should be able to do this by setting the following on your LWF:
[monitor:///syslog-ng/path/.../logs] host_segment = 3
So if you have a path like /var/log/192.168.9.1/ events will be set to host=192.168.9.1 Without this setting, you can't use your current props configuration.
This will set the host field for your syslog-ng events and allow the props/transforms on the indexer to do their job.
If this isn't working for you still, please feel free to file a support ticket. I'm sure its just some type of misconfiguration.
Seems like a nice suggestion. Tested and failed as well . I suppose I am doing something stupid in this case. Host:: seems to be set correctly but still the indexer does not route the events of interrest into the any other index.
Are there any debugging tools that gives access to what is actually sent from the LWF ( or recieved on the indexer would also work i suppose)
v
Light Forwarders do not perform transforms. That's part of what makes them light forwarders.
Consequently, sophisticated routing cannot be accomplished on a light forwarder.
(Internal details: transforms are performed on events. Events are identified by timestamp extraction and so on. Timestamp extraction and so on is avoided on light forwarders. Light forwarders only deal in chunks of uninspected event data, not split into discrete events.)
Yep, I am quite comfortable with the LightForwarder just handling the raw dataflow. I will get several different flows coming in to the indexer so I would like to do most of my routing as centrally as possible.
Try removing the transforms stanza and modify your props with the below settings, restart afterwards:
props.conf [host::192.168.9*] index = index_GroupA [host::192.168.13.*] index = index_GroupB
I made these changes on the main indexer, but still the events from host::192.168.9* and host::192.168.13.* end up in index = main.
I thought this should be a relatively straight forward scenario...