Splunk Search

Transfer logs between different network segments - which forwarders to use where ?

dlugasny
New Member

Hi,

our network count ~9000 Servers. Most of them running in the separate network IP segments. I would like to kindly ask You about log forwarding from that machines. Between indexer and some servers we have to build several hops (forwarders). How to build it properly ?

Please take a look on that example:

Example:

Linux Server GROUP LAN1 (splunk forwarder which one ?) ----- > Splunk Forwarder (which one ?) LAN1 ----> Splunk Indexer
Linux Server GROUP LAN4 ----> splunk forwarder LAN3 ---....---> Splunk Forwarder LAN1 --- > Splunk Indexer ?

If I good understand Splunk architecture, on the each machine I have to install Splunk Universal Forwards(or lightforwarder ?) to transfer logs from the local running applications. Each Universal Forwarder installed on the app servers will push the logs to the
heave forwarder which will be connected with the next hop (also Heave Forwarder or in the final step with the indexer). Is this the proper solution ?

lightforwarder ---> heave forwarder ---> heavy forwarder ----> Indexer ?

What about loadbalancing, we need it. If we would like to push logs from ~500 heavy loaded systems, we need minimum two machines I suspect. Is it possible to loadbalance such a traffic ?

Thanks in advance for any hints.

With kind regards

Mike

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

Your forwarding architecture, while complex and difficult to troubleshoot, is technically workable. Sometimes, you have to do what you have to do in a highly segmented network with overly(?) restrictive firewall policies.
Unless you have advanced routing/filtering requirements, you don't need a heavy forwarder anywhere; universal forwarders can do it, and do it more efficiently.
You need to take care not to introduce any bottleneck that will affect your event distribution across the indexer tier. The last forwarding tier needs to have at least 2x forwarders (or ingestionPipelines) than you have indexers to ensure indexers are served concurrently and as evenly as possible.
Your first intermediary tier needs to be properly sized as well to handle the 9000 endpoints (2 TCP connections each, one heartbeat, one data). Make sure you properly set your ulimits such that your intermediary forwarders can handle the large connection counts.

If your intermediary forwarder servers are equipped with enough cores, you can safely increase parallelIngestionPipelines to have each forwarder process multiple inbound/outbound connections in parallel. See the documentation for details. If you have eight-core servers, you can safely configure four parallel pipelines.
Also, ensure you have sufficient network bandwidth on ingress and egress to handle the data volume going through your intermediary forwarding tier. And don't introduce single points of failure into your intermediary forwarding architecture, i.e. have at least 2 servers at each tier.

Hope that helps.

0 Karma

tmarlette
Motivator

This would be an extremely complex environment, and very difficult to maintain as an administrator.

The only reason I've ever seen this done, is not because of VLAN's or segments, but data centers and cost. Basically, the data center charged the company I was working with per segment it opened, and per host, due to NAT'ing all traffic at the edge. opening a hole in the firewall for the ~1200 machines behind it was cost prohibitive.

I would agree with everything below, in that simple is better. you have 9000+ servers. I can say that I have administered 8300+ servers, and used a single heavy forwarder in each data center (in that case there were 5 datacenters) to do any excess pre-index parsing necessary. Mind you that includes running 5000+ nodes of hadoop parsing through them.

making this too complicated will hurt your efforts in the future, depending on the competency, and number of your staff that is familiar with Splunk. if it's just you... this would be more pain than it's worth.

for your load balancing efforts, simply add forceTimeBasedLoadBalancing = true to your outputs.conf on any forwarder, and then autoLB = true. This should take care of equally load balancing your data across your indexers.

look them up in the docs for outputs.conf

0 Karma

ddrillic
Ultra Champion

Mike, why do you need the heavy forwarder(s) in the middle?

0 Karma

dlugasny
New Member

I do not know if I need this 😉 this was only question if its necessary.

We will transfer logs from different machines, devices and I thought that it would be better to build Heavy Forwarders (with a lot of options to filter logs) between.

What You prupose ?

0 Karma

ddrillic
Ultra Champion

My advice is to keep things simple. Universal forwards are designed to communicate perfectly fine with the indexers. So, if you don't have at the moment any special parsing needs I would go with the simplest approach.

0 Karma

dlugasny
New Member

At the moment we have no any special parsing requirements but if its possible we would like to parse logs on our hops/forwarders. Which forwarder will be the best one for it ?

Our target is to keep application server configuration as simple as possible so we will forward everything to the first forwarder, where we would like to setup filtering for an each incoming log stream. In that case Heavy Forwarder make sense, isnt it ?

0 Karma

ddrillic
Ultra Champion

Look, you can filter at the heavy forwarder level or at the indexer level. Normally, we start with the simplest architecture and then amplify it based on the needs, but obviously, you can start with more evolved architecture. When we do have heavy forwarders, it's normally in order to alleviate pressure from the indexers, so if you know about a high velocity stream of data that needs to be parsed, then the heavy forwarders layer(s) might make sense.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...