Deployment Architecture

How to route to separate indexes based on host:: when forwarding to a indexer from SplunkLightForwarders

ostmovid
New Member

I use several SplunkLightFirwarders on Suslog-ng servers to have a "buffer" to relatively large amounts of syslog that are then forwarded to the main indexer.

Several different corporate units need separate acces restrictions and different retention time. so i would like to put data from different host:: in separate indexes.

as far as I found in the Docunentation the Light Forwarder do not take any routing desicions. and i therefore put following config on the main indexer:

  • props.conf *
[host::192.168.9*]
TRANSFORMS-routing = route_to_groupA

[host::192.168.13.*]
TRANSFORMS-routing = route_to_groupB
  • transforms.conf *
[route_to_groupA]
DEST_KEY = _MetaData:Index
FORMAT = index_groupA

[route_to_groupB]
DEST_KEY = _MetaData:Index
FORMAT = index_groupB

Put the same config on the SplunkLightForwarder which made no change to my result.

Not working ...

vidar

Tags (2)
0 Karma

ostmovid
New Member

Thanks for all the help here. There are obviously several ways to solve this problem . I wanted to separate log into separate indexes based on which host generated the event.

Since it was indicated to me that the host:: tag was already put in there by the LWF , i used some time to try to make that work. to no avail. ( i really miss some "sniffing" tools to see what are actually see what king of data and tags are communicated between Indexer, FW and LWF )

Since the source tag are available from the LWF My solution became this.:

props.conf:

[source::/var/log/syslog]
TRANSFORMS-routing = route_to_a,route_to_b,route_to_c

transforms.conf:

[route_to_a]
DEST_KEY = _MetaData:Index
REGEX = ^\S+\s+\d+\s+\S+\s+192\.168\.111\.19
FORMAT = index_a

[route_to_b]
DEST_KEY = _MetaData:Index
REGEX = ^\S+\s+\d+\s+\S+\s+192\.168\.111\.86
FORMAT = index_b

[route_to_c]
DEST_KEY = _MetaData:Index
REGEX = ^\S+\s+\d+\s+\S+\s+192\.168\.111\.85
FORMAT = index_c
`
0 Karma

dwaddle
SplunkTrust
SplunkTrust

The example I gave in a related question works for directing sourcetypes to a specific index for data coming from a LWF. I assume it would work equally well for subnets.

0 Karma

the_wolverine
Champion

You mentioned syslog-ng. Assuming your host is the path for monitor input, you should be able to do this by setting the following on your LWF:

[monitor:///syslog-ng/path/.../logs] host_segment = 3

So if you have a path like /var/log/192.168.9.1/ events will be set to host=192.168.9.1 Without this setting, you can't use your current props configuration.

This will set the host field for your syslog-ng events and allow the props/transforms on the indexer to do their job.

the_wolverine
Champion

If this isn't working for you still, please feel free to file a support ticket. I'm sure its just some type of misconfiguration.

0 Karma

ostmovid
New Member

Seems like a nice suggestion. Tested and failed as well . I suppose I am doing something stupid in this case. Host:: seems to be set correctly but still the indexer does not route the events of interrest into the any other index.

Are there any debugging tools that gives access to what is actually sent from the LWF ( or recieved on the indexer would also work i suppose)

v

0 Karma

jrodman
Splunk Employee
Splunk Employee

Light Forwarders do not perform transforms. That's part of what makes them light forwarders.

Consequently, sophisticated routing cannot be accomplished on a light forwarder.
(Internal details: transforms are performed on events. Events are identified by timestamp extraction and so on. Timestamp extraction and so on is avoided on light forwarders. Light forwarders only deal in chunks of uninspected event data, not split into discrete events.)

ostmovid
New Member

Yep, I am quite comfortable with the LightForwarder just handling the raw dataflow. I will get several different flows coming in to the indexer so I would like to do most of my routing as centrally as possible.

0 Karma

BunnyHop
Contributor

Try removing the transforms stanza and modify your props with the below settings, restart afterwards:

props.conf

[host::192.168.9*]
index = index_GroupA

[host::192.168.13.*]
index = index_GroupB
0 Karma

ostmovid
New Member

I made these changes on the main indexer, but still the events from host::192.168.9* and host::192.168.13.* end up in index = main.

I thought this should be a relatively straight forward scenario...

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...