Splunk App for Stream: Has anyone come up with an elegant solution to manage thousands of stream forwarders?

[dstaulcu]

Has anyone come up with an elegant solution to manage many thousands of stream forwarders?

I am excited about the prospect of utilizing stream forwarders throughout our environment but I am also concerned about issues of scale with regard to configuration update checking.

It appears that stream forwarders check in with their stream management server for configuration updates every 5 seconds. I realize that update checking frequency for stream forwarders can be reduced with the PingInterval configuration option in streamfwd.xml, but before I plan to turn that dial far in the other direction to achieve scale, I figured I'd ask if others have had success in distribution of stream management functions in a manner that allows them to maintain responsiveness of forwarders to changes in configuration directives.

Inputs appreciated!

[cching_splunk · Accepted solution]

Managing thousands of Stream Forwarders: Additional management of the Stream Forwarders is advancing in the next release of Stream targeting mid-year 2015. The release specifically addresses issues regarding more precise control over Stream Forwarder protocol collection as well as a grouping capability and this is currently being field tested. Moving forward the addition for more "command and control" capabilities from distribution to collection into App for Stream continues to be priority.

[dstaulcu]

Outstanding! -Thank you.

[muebel]

I haven't implemented the Stream app, but my guess is that you could use the deployment server to manage the forwarder config like any other app.

[dstaulcu]

I am using deployment servers to deploy and update versions of the stream forwarder app (splunk_ta_stream). However, the input stanza for splunk_ta_stream only seems to influence the state of the app (enabled|disabled), metadata regarding the source name, and finally the location of the splunk web server hosting the splunk app for stream (splunk_app_stream).

Stream content forwarding directives seem to be synchronized over a communication path between stream forwarder and stream server. The stream forwarder checks in with stream server every 5 seconds for changes in content forwarding directives. It does not appear that these directives are written to disk on the stream forwarder in any way.. which would seem to place a high availability dependency on the stream server in addition to high capacity for concurrent processing of synchronization messaging with forwarders.