It would be nice to just click a button in a dashboard, or use a custom search command to be talk to the universal forwarders and enable/disable individual stanzas in inputs.conf (or any conf file really)
Example
"We are having trouble with radius
authentication this morning, it seems
flaky for some users. Oh, well lets
turn up the monitoring on that server
and see whats going on." Then the user
could browse to a Splunk app, select a
few things she thinks would be helpful
and a minute later data is flowing in.
More data then she would want indexed
regularly, but just for this ticket
she wants to see it. Then, when she is
done, she just turns it off again.
Of course the forwarder management features gets the job done, but it is more construction than surgery. I am also aware of the deployment manager app which, despite its name, does very little in the way of managing. S.o.S is nice too, and with a little extra effort, you can get it watching UFs as well. But what about managing the actual configuration files on the forwarders themselves?
I am also aware many splunkers use CM tools to manage Splunk's configuration, but it would be nice if Splunk was less codependent with other systems.
I have looked and looked but, to my surprise, did not find any convenient ways to interact with the UFs configurations without editing the files directly. Can someone point me to some remote management mechanisms for the Splunk Universal Forwarders?
I personally use ansible to manage forwarders. It's great as long as you have ssh keys.
How about REST API from Splunk any instances to UF, event though you will set user/password for connections in UF ?
http://docs.splunk.com/Documentation/Splunk/6.1.5/RESTAPI/RESTusing
Note about this strategy: You may only access a UF on its REST interface if you've first changed the default admin password.
Did you tried using "Forwarder management" dashboard available in splunk web UI
distributed environment -> Forwarder management
Using "deployment Server" - Use separate serverclass stanzas to push configs to splunk UF. - easy to manage and deploy apps.
Yep, we are already doing that. I am looking for something more surgical, on an input-by-input basis.