Deployment Architecture

Handling a large number of forwarders

JovanMilosevic
Path Finder

Hi,

The set-up is Splunk 5.0, and the requirement is to monitor the Windows Security Event Logs on 10,000 desktops for specific Event Codes. We would be using the Heavy Forwarder on the desktops, so that only the specific events would be forwarded. The actual number of events expected is very low (probably less than double figures per host per day). So, the actual indexing and searching workload would be minimal, but how would an indexer handle that many connections from the forwarders ? I have read a previous post on "How many Forwarders can an indexer handle ?", and the reply only concerned itself with the amount indexed.

I have also read various documents on Deployment Server with a large number of forwarders being pointed to a single Deployment Server, and came across a suggestion that the polling interval could be extended to 10 minutes. There was also a comment that up to 1000 forwarders had been successfully connected to a single Deployment Server. There wasn't any indication on what polling interval was used. It is not likely that much maintenance would be carried out, so changes would be minimal. We're not that bothered if changes weren't propagated for an hour or 2. Is there any randomness in the polling interval ? I'm thinking that if we set the polling interval to 2 hours, it still may not help, as if 80% of users start up at around 09:00, then all hell would break loose, then a lull, then same thing at 11:00.

Any comments on what might be needed to support this would be gratefully received. Thanks.

0 Karma

glancaster
Path Finder

You could split up your 10,000 assets to poll at different intervals with breaking up your assets into different server-classes, say of 1,000 assets each, or however many you feel confortable with. Then with the assets split up you could set the phoneHomeIntervalInSecs attribute at a different value for each server-class and that would help with congestion a bit. This manual helped me a ton

http://docs.splunk.com/Documentation/Splunk/5.0.3/Deploy/Configuredeploymentclients

As for the randomness of the polling, I don't believe there is, looking at my deployment, polling is very consistant. There may be a way to specify if you want an asset to poll every minute but you want it to poll on the :30 sec marks instead of :00 marks, maybe set phoneHomeIntervalInSecs = 90.

Anyways there is probable more than one way to do what you are talking about but this should help get some ideas going.

0 Karma

JovanMilosevic
Path Finder

Hadn't considered splitting them up like that. Thanks. Might still get rather busy at the beginning of the day, but it would certainly help smooth things out during the rest of the day.

0 Karma

krugger
Communicator

My main concern with that many connections is the SSL overhead. I would advise that you stress test the server with thousand of SSL connections to see how it performs.

SIEGE tool:
http://www.joedog.org/siege-home/

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

By default forwarders should send a heartbeat every 30 seconds to the indexer(s) they'd be sending to if they had data, so I don't think there is any auto-closing of idle connections - see heartbeatFrequency in outputs.conf.

0 Karma

JovanMilosevic
Path Finder

Thanks for the suggestion. In order to test this out, I would need to know how many connections would be active, and that would depend on whether the forwarder closed the connection if there was nothing to send, and if it did, how long would it wait after the last event before it did.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...