Deployment Architecture

Is there a way to limit the number of clients a Splunk deployment server pushes a new/updated app to at the same time?

jbresciani
New Member

Is there a way to limit the number of clients a Splunk deployment server pushes a new/updated app to at the same time? We've run into issues where to the Splunk universal forwarder is restarted on hundreds of virtual machines simultaneously causing massive IO delays on shared storage.

I'm looking for a way to have a Splunk deployment server stage out the deployment of app changes so that only X number of nodes are deployed to at a time.

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

There is no way to do this with Splunk native functionality aside from groups in serverclasses [ group by subnet / server location / server role / etc.] The best option is to adjust the phonehomeinterval to a level that reduces your i/o to acceptable levels. The caveat to this is that it takes longer for your clients to update.

Outside of Splunk, there are some other things you could potentially do, although not recommended. One option Ive seen is to run with multiple dns cnames for your deployment server. Push your updates in a staggered manny by dropping the cnames out while pushing updates. The UF will still phone home, but wont be able to connect until you put the cnames back in. You need to be aware of the ttl values in dns in this case though and have it set really low.

Another option would be firewall rules, block inbound to the DS for portions of your UF's subnets while updates are being pushed...

sobrien
Splunk Employee
Splunk Employee

Hi Jacob,

A good way to achieve this would be to break up your fleet into smaller serverclasses:

https://docs.splunk.com/Splexicon:Serverclass

You can then confirm the successful install of your apps and updates per serverclass prior to rolling out to the rest of the fleet.

Hope this helps.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...