Getting Data In

What are suggested approaches/steps for Syslog to switch to another heavy forwarder when primary goes down?

neelamssantosh
Contributor

Hi Splunkers,

Kindly suggest any approach/steps for

Syslog must Switch to another Splunk Heavy Forwarder automatically when One Splunk Heavy Forwarder goes down.

If Primary FWD goes then it must switch to Secondary FWD.

Thanks in Advance.

2 Solutions

hortonew
Builder

Depends on your infrastructure, but one way which we use is to set up a virtual IP on a network device that you send all log data to, which can then map to multiple physical addresses. If one of the physical addresses in the pool goes down, it'll still send to the ones remaining.

View solution in original post

musskopf
Builder

Here my 2cents:

HeavyForwarders don't support it. As alternative solution I suggest one of below:

  1. Use the syslog destination as a FQN A/CNAME in the DNS and update it if you need to failover - Might take some time til the DNS changes to replicate

  2. Use a Virtual IP and a Cluster software, something like Linux heartbeat

  3. If you have a device like F5 in your network, configure the virtual IP and fail-over rules there

  4. Send syslog to both Splunk and perform a Dedup before index the data - waste of bandwidth/load?

  5. Send syslog to both Splunk but one of the destinations will be off-line or blocked via firewall - you might have duplicated data if you end-up with both up at same time. Also similar bandwidth issue as above.

I'm probably going to have a F5 device in front of my heavy forwarders. The syslogs messages will be send via UDP from the devices but the F5 check rule will be checking the TCP/514 to confirm the service is up. I believe F5 will also allow me to create affinity rules....

Let me know if you have any other idea... I have similar problem with DBX App, that I still finding a better solution instead o manually migrate the configuration in case of failure.

View solution in original post

musskopf
Builder

Here my 2cents:

HeavyForwarders don't support it. As alternative solution I suggest one of below:

  1. Use the syslog destination as a FQN A/CNAME in the DNS and update it if you need to failover - Might take some time til the DNS changes to replicate

  2. Use a Virtual IP and a Cluster software, something like Linux heartbeat

  3. If you have a device like F5 in your network, configure the virtual IP and fail-over rules there

  4. Send syslog to both Splunk and perform a Dedup before index the data - waste of bandwidth/load?

  5. Send syslog to both Splunk but one of the destinations will be off-line or blocked via firewall - you might have duplicated data if you end-up with both up at same time. Also similar bandwidth issue as above.

I'm probably going to have a F5 device in front of my heavy forwarders. The syslogs messages will be send via UDP from the devices but the F5 check rule will be checking the TCP/514 to confirm the service is up. I believe F5 will also allow me to create affinity rules....

Let me know if you have any other idea... I have similar problem with DBX App, that I still finding a better solution instead o manually migrate the configuration in case of failure.

kylerose
Explorer

With regards to #1, I changed the A record in DNS and splunk never resolves it again. It requires a reboot to updates resolution. With tcpout, you can set dnsresolutioninterval , but not with syslog:

Invalid key in stanza [syslog:mysyslog] in /opt/splunk/etc/system/local/outputs.conf, line 13: dnsresolutioninterval (value: 300)

0 Karma

neelamssantosh
Contributor

Kindly correct me if i am wrong,
can i achieve it with check sum redundancy in inputs.conf
crcSalt = <-source>

As perimeter device(Syslog) can send syslog logs to up-to 4IP's/HFW's.

it avoids the duplicate data while Re-indexing.

0 Karma

neelamssantosh
Contributor

Thanks MuS,
For quick response.

Cheers..!!

0 Karma

MuS
SplunkTrust
SplunkTrust

Nice list of options 😉

Just one more to add: Linux Heartbeat can do this too

But remember, this Question is not related to Splunk. This feature must be configured outside Splunk.

cheers, MuS

hortonew
Builder

Depends on your infrastructure, but one way which we use is to set up a virtual IP on a network device that you send all log data to, which can then map to multiple physical addresses. If one of the physical addresses in the pool goes down, it'll still send to the ones remaining.

grijhwani
Motivator

You're missing what he said. You have an intervening device presenting a static virtual IP, which (as with the F5 load-balancer suggested in the previous answer) monitors the health of the various available forwarders, and routes the traffic accordingly.

Remember, by default syslog is UDP - i.e. stateless. As remarked, something needs to be present on the indexer to indicate the health of the syslog inbound on each host.

neelamssantosh
Contributor

Thanks hortonew,
Syslog device server can send upto 4IP Address in parallel for High Availability of Forwarders but thats not the case we are looking for.
If Primary FWD goes then it must switch to Secondary FWD.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...