How to use sourcetype to route data from a heavy forwarder to a specific index?

[ebailey]

I am trying to route data from a heavy forwarder to a specific index. I would prefer the rule be run on the indexers for a more scale-able solution. I using the sourcetype to route the data but this is not working. Any ideas are much appreciated.

props.conf

[cisco:asa]
TRANSFORMS-force-sourcetype_for_cisco_devices = route_to_firewall_index

transforms.conf

[route_to_firewall_index]
SOURCE_KEY = MetaData:Sourcetype
REGEX = (%ASA)
DEST_KEY = _MetaData:Index
FORMAT = net_firewall

Thanks!

[sowings · Accepted solution]

The Splunk for Cisco ASA TA (http://apps.splunk.com/app/1620/) app expects the data to come in over the wire, either on TCP or UDP, and uses transform rules in props.conf to alter the sourcetype. This means that your rule to change the destination index has to fire on the udp:514 (or tcp:514) sourcetype of the original ingest. To change the destination index, you'll have to add another transform, after the sourcetype change (see below) and key that from the origin sourcetype.

I do this traditionally by first setting my transform to detect the base sourcetype string like this:

TRANSFORMS-0_sourcetype = force_sourcetype_for_cisco_asa

Then, choosing the name of the RHS of my TRANSFORMS keyword to sort lexically after the sourcetype rule, I do something like this:

TRANSFORMS-1_index = my_transform_force_index_for_asa

transforms.conf:

[my_transform_force_index_for_asa]

SOURCE_KEY = MetaData:Sourcetype

REGEX = cisco:asa

DEST_KEY = _MetaData:Index

FORMAT = firewall

Takeaway: You get one pass, top to bottom, left to right, of the indexing rules for your initial sourcetype upon contact with Splunk. Having a rule to first set sourcetype from udp:514 -> cisco:asa does not fire index-time rules for the new sourcetype. Search-time parsing properties would in fact be used, however.

[ebailey]

I need to route the ASA events to a specific index for all of the above reasons and to make searching faster on the indexers. This rule worked fine in the HF until we reached enough volume that the HF has become a bottleneck. Long term we are going to move the firewall data to a F5 and then distribute data across all of the indexers. In the short term I am going to use your idea of a light weight forwarder and then move the rule down to the indexer level. Thanks for the advice.

[ebailey]

Wouldn't I still run into the bottleneck where all events have to evaluated to be routed into the right index? I am doing this because the heavy forwarder has more events than it can handle and it is backing up.

[sowings]

Yes, you'd still have to have the available queue depth on the heavy to apply the transform.

What's your rationale for changing the index? Simply routing the data to a different index won't lessen the load on the heavy, so I'm curious about the motivation for your original question:

Are you trying to change the destination index for (retention OR access control), or to try to alleviate load on the heavy?

[ebailey]

yes indeed - but I can make some adjustments if I do not have any other options.

[sowings]

One thing: "queue blocking on the rule" you could simply make the heavy a light forwarder so that it's not parsing, and simply acting as a passthrough. But I'm guessing that you have a heavy for other reasons, so let's proceed.

[ebailey]

I have the Splunk ASA TA installed on the HF so the data from the ASA is correctly assigning the cisco:asa sourcetype to the events. So I cannot route cooked data once it hits the indexer? The reason why I want to move the routing rule is that the queue is blocking on the rule so I would prefer to move the rule sot the indexing tier where we have more indexers to spread the load instead of just one queue in the HF. Am I out of luck?

[sowings]

How is your ASA sending to your heavy? If it's coming in over the wire, it may be assigned an initial sourcetype of udp:514 (or whatever is tied to the network inputs.conf defnition). If it's a transform that's invoked on [udp:514] to change the sourcetype to cisco:asa, then you've already missed your change to (also) set the target index. You get one pass through TRANSFORMS at index time, keyed from the initial sourcetype of the data.

More information about your setup is required to be able to tell precisely what's going on.

[MuS]

Hi ebailey,

you got it almost, try something like this:

props.conf

 [cisco:asa]
 TRANSFORMS-routing_for_cisco_sourcetype = route_to_cisco_asa_index

transforms.conf

[route_to_cisco_asa_index]
 REGEX = .
 DEST_KEY = _MetaData:Index
 FORMAT = <your new index name>

You don't need to regex on the sourcetype, because you assigned this special transforms already to the sourcetype.

hope this helps ...

cheers, MuS

[ebailey]

I did some testing and this did not work. From the comment above it seems I cannot route cooked data.

[MuS]

This should be done on the heavy forwarder and it seems I cannot route cooked data is not true. You cannot do this by default, but if you change your splunktcp stanza in inputs.conf on the indexer to something like this:

[splunktcp] route=has_key:_utf8:parsingQueue;has_key:_linebreaker:parsingQueue;absent_key:_utf8:parsingQueue;absent_key:_linebreaker:parsingQueue

it will re-parse cooked data. By doing this, you somehow make the heavy forwarder obsolete 😉
So, either but it on the heavy ( props/transforms) or use as mentioned the light weight forwarder instead.

[jrodman]

PLEASE do not redirect already parsed events back into the parsing queue. This ability will be removed in the future and you may be left stuck, and there is no way that we can guarantee it will work as we continue to fix bugs and handle the expected usecases.
This should be considered similar to reconfiguring the pipelines in splunk which technically is still possible but is totally unsupportable.

[MuS]

As I wrote it makes no sense to do something like that, even when it is possible. Thanks for your input, I'll keep that in mind!