Out of Band Forwarder

[antlefebvre]

We have an out of band (OOB) management network that does not route to our production network. It is on physically different switches. I need to forward syslog data from the OOB network to my splunk indexers on the production network.

Is there a best way to do this?

I was thinking universal forwarder open on the syslog port on the OOB and then forward the data to the splunk indexer on the production. I have not tried this with a universal forwarder. Will this work? Or will I need to set up a syslog server on that machine and then forward the syslog data with the UF to the splunk indexer?

[rturk · Accepted solution]

Hi Antlefebvre,

The Splunk Universal Forwarder can definitely be set up to collect inbound Syslog data, but seeing as Syslog is typically sent via UDP, any time the forwarder applicaiton is restarted (required by some application installs), you will not collect any UDP messages while this happens.

So yes... the Splunk Forwarder can collect & forward Syslog... but the forwarder will need to run as a user with Admin/root privs in order to bind to a privileged port (<1024). Best practice would be to use a dedicated Syslog collection method such as Syslog-NG, Rsyslog, or Kiwi Syslog to collect data, and have Splunk pick it up and forward it on from there.

BUT...

In the scenario you've outlined, you've stated that the OOB network does not have a route to the production network. Regardless of whether you use TCP/UDP 514 (Syslog) or TCP 9997 (Default Splunk forwarding port), you're going to need network connectivity to your Indexer to get the data in.

Here's a dramatic re-enactment...

                   Out-of-Band Network      |  A  |  Production Network
            --------------------------------|  I  |--------------------------------
Current:                    OOB Switch  --->|  R  |  Indexer (Feeling unloved) :(
                                            |  G  |
Proposed:   OOB Switch ---> Splunk Fwdr --->|  A  |  Indexer (Why won't anyone talk to me) :(
                                            |  P  |

My guess is that there's no route for a reason (typical for OOB due to the nature of it's use), or it may be a security consideration. So this kinda stops being a Splunk issue, and is more of a networking connectivity problem best solved internally.

TL;DR: Yes Splunk Forwarders can collect Syslog data and forward them on... but the data still needs to be able to get to the destination. Have a chat to your network admins to suss out a solution for your environment.

Hope this helps 🙂

[antlefebvre]

I forgot to mention that this server has 2 nics. One on each network to bridge the routing gap. I opened the syslog port in the inputs.conf and forwarded the data to the production network splunk server. Thanks for the answer.