Getting Data In

How to configure outputs.conf to forward data in a fail-over method

ekcsoc
Path Finder

We have HF 1 and HF2 that are located in DC1 and DC2 respectively. How can we configure outputs.conf in below method.

  • All servers in DC 1 should forward data to HF 1 primarily and only send data to HF 2 in case of a fail over (ie HF1 goes down.)
  • All servers in DC 2 should forward data to HF 2 primarily and only send data to HF 1 in case of a fail over (ie HF2 goes down.)

Please note that we are not looking for a autolb feature and just looking for a failover kind of configuration in outputs.conf

woodcock
Esteemed Legend

Splunk does not work that way but what you can do is do something like this:

DC1 outputs.conf:

server = DC1HF1:9997, DC2HF1:9999

DC2 outputs.conf:

server = DC2HF1:9997, DC1HF1:9999

Then always run inputs.conf with port 9997 on but 9999 off.
Then have cron job on both HFs that is checking on the health of his partner in the other DC and when it is down, enable the local 9999 port in inputs.conf and restart local splunk. Do the opposite when the other side comes back up.

arjunpkishore5
Motivator

I don't think Splunk explicitly can do this.

However, if you can introduce a load balancer in between and forward to the load-balanced URL, this is possible. Unfortunately, from your question, it looks like this is not one of the available options.

0 Karma

skalliger
Motivator

This is not how Splunk works, sorry.

Why are you using Heavy Forwarders? Any need to do so?

I don't really like to give you a workaround here, but.. If you have separate DNS in those data centres, you could manage a failover with DNS. So, in case server 1 goes down, assign the other IP address to its hostname.

Skalli

0 Karma
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...