How to route a subset of data to syslog?

[maciep]

I was hoping to get some help with routing a subset of data to a syslog server.

The goal is to send a handful of windows eventlog id's to a third party syslog server so they can use them for correlation. We want them indexed in our cluster as well. We know we have to use a hf to route to syslog which isn't a problem. But we also don't want to send those events through the hf to get to our indexer cluster, because it isn't as robust.

So the plan is to send the all Win Security events to both our indexer cluster and the heavy forwarder (that does work). Then on the heavy forwarder, send all events except the ones we want to the nullQueue. And finally route the desired events to the syslog server. We would also like for our heavy forwarder itself to log its data to the indexer cluster.

I've tried many different configs, but there seems to be a general theme. If I try to send any of the security events to the null queue, they all go to the null queue and nothing goes to syslog. If I try to just route the specific events to syslog, they do go as expected. But the other events still get forwarded to our indexer cluster - so we have duplicate events there - one from the uf and one from the hf.

Any idea what I'm doing wrong? Here are conf files on the heavy forwarder.

outputs.conf:

[tcpout]
# Ensure indexer received data
# Use 7MB buffer
maxQueueSize = 7MB
useACK = true

# Switch indexers every 30s
forceTimebasedAutoLB = true

# TURN OFF INDEXING ON SEARCH HEAD
# Deploy org_all_forwarder_outputs to search heads as well, to dictate
# where the data should be sent.
[indexAndForward]
index = false

[tcpout:indexers]
disabled = false
server = <bunch of servers here>

[syslog:palo_alto]
server = <the syslog server here>:514
type=udp
timestampformat = %b %e %H:%M:%S

props.conf

[WinEventLog:Security]
#TRANSFORMS-2_null_routing = route_to_null_queue
#TRANSFORMS-1_syslog_routing = route_to_palo_alto
TRANSFORMS-syslog_routing = route_to_null_queue,route_to_palo_alto

[source::/tools/splunk/*]
TRANSFORMS-indexer_routing = route_to_indexers

transforms.conf

[route_to_palo_alto]
REGEX = EventCode=(4624|5447|4957)
DEST_KEY = _SYSLOG_ROUTING
FORMAT = palo_alto

[route_to_null_queue]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[route_to_indexers]
REGEX = .
DEST_KEY = _TCP_ROUTING
FORMAT = indexers

If I get rid of the tcpout stuff and don't try to send anything to the null queue, it works well enough. Just the events I want go to syslog, and the others are just lost I guess. But then I lose the local logs for the heavy forwarder as well.

This is for a POC right now, so not a huge deal, but just really would like to know what I'm missing.

[alacercogitatus]

Transforms are performed in order listed. So, you have null queue first. So everything goes to null queue. Try this:

 TRANSFORMS-syslog_routing = route_to_palo_alto,route_to_null_queue

Once you try that, and if it doesn't work, comment below, and we will dive deeper.

[maciep]

That was one of the configs I tried, but I can give it another shot. My understanding, though, is that we should set everything to go to the null queue, then right after that change those few events to go to syslog instead. If we do the null queue transformation last and are using a "." regex, then everything would end up going.

Meaning, all of the transformations get applied in order and then whatever the final state of a message is gets processed.

[alacercogitatus]

You can also find me on IRC (#splunk on efnet.org) for additional help without all the emails.