Getting Data In

Forward specific inputs from indexer to intermediate forwarder

mccartneyc
Path Finder

Hi guys, here is the current setup I have.

UF uses data cloning to send to both an indexer cluster and an intermediate forwarder.
The intermediate forwarder sends local audit, secure, and messages and the data it receives from universal forwarders to an external splunk environment using props and transforms.

An issue I am trying to resolve is I also need the local audit.log, secure, and messages inputs on our indexers to forward to the intermediate server so they can be forwarded to the external system. I want the indexer to locally index those 3 inputs from it's local logs while also sending ONLY those 3 logs to the intermediate and locally indexing everything else it receives.

I've tried creating an outputs.conf file on the indexer with the target group [tcpout:intermediate] and in the inputs.conf, for the messages, secure, and audit.log monitors, adding in _TCP_ROUTING = intermediate to see if they would at least be forwarded to the intermediate forwarder, but it seems to break indexing.

Not sure of the process for this, but the reason is we need all servers to send those 3 logs over to the intermediate forwarder (which is basically a heavy forwarder that doesn't index), and those will be sent to an external splunk environment for monitoring all of our servers.

Thanks in advance!

0 Karma
1 Solution

mccartneyc
Path Finder

The solution I found for this issue in our environment was to install UFs on each Splunk server alongside the enterprise install. Installed UFs in /opt/splunkforwarder and created a custom service for the UF to use in /etc/init.d by copying the /etc/init.d/splunk to /etc/init.d/splunkfwd and changing the paths from /opt/splunk to /opt/splunkforwarder and the cache path from splunk to splunkfwd. When splunk is installed and you install splunkforwarder on the same machine, the UF chooses the next available port to run on.

Did this all through a puppet module and currently have it deployed and running successfully to the following servers:
Master
Indexers
Intermediate Forwarder - This receives cloned data from all UFs and forwards externally. The UF sends data to our local indexers.
Search Heads

For whatever reason, I was unable to get the indexers to receive all data and then forward to the intermediate forwarder. When setting up forwarding on the indexers, they would either stop indexing and not forward anything or they would index and still not forward anything.

For the time being, this solution works and has not had an impact on performance. The UFs on the splunk servers are configured to forward only local data to our indexers and the intermediate, while the enterprise installation only indexes internal logs.

View solution in original post

0 Karma

mccartneyc
Path Finder

The solution I found for this issue in our environment was to install UFs on each Splunk server alongside the enterprise install. Installed UFs in /opt/splunkforwarder and created a custom service for the UF to use in /etc/init.d by copying the /etc/init.d/splunk to /etc/init.d/splunkfwd and changing the paths from /opt/splunk to /opt/splunkforwarder and the cache path from splunk to splunkfwd. When splunk is installed and you install splunkforwarder on the same machine, the UF chooses the next available port to run on.

Did this all through a puppet module and currently have it deployed and running successfully to the following servers:
Master
Indexers
Intermediate Forwarder - This receives cloned data from all UFs and forwards externally. The UF sends data to our local indexers.
Search Heads

For whatever reason, I was unable to get the indexers to receive all data and then forward to the intermediate forwarder. When setting up forwarding on the indexers, they would either stop indexing and not forward anything or they would index and still not forward anything.

For the time being, this solution works and has not had an impact on performance. The UFs on the splunk servers are configured to forward only local data to our indexers and the intermediate, while the enterprise installation only indexes internal logs.

0 Karma

jimodonald
Contributor

You can use syslog to forward those Linux OS logs to an external syslog server (which could be hosted on your HF or elsewhere). Once that is done, set up a UF/HF to foward those logs to your intermediate forwarding layer.

0 Karma

mccartneyc
Path Finder

Update:
Haven't received any responses yet, but have been trying some things out. When adding an outputs file and inputs file to the indexers to only send /var/log/messages, secure, and audit.log, indexing stops on the server.

Tried using indexandForward, but the indexer attempts to forward everything.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...