Confused about having the deployment server on a separate system.

[jwinderDDS]

I am in the process of revamping our Splunk installation. This time around we are attempting to implement a more distributed system.

In the past we have had a single server running everything, indexing, searching, deployment services, etc. Currently I have a system setup running the Indexer, KV Store, License Master, and Search Head and another running the Deployment Server and Distributed Management Console. Indexing will be off loaded to other indexers as the buildout progresses.

With the deployment server on a separate system, I am confused trying to following scenario.

I have created a deployment application directly in the filesystem to configure the Universal Forwarders to forward data to the indexer. This is working as expected.

Then to setup data imports, I login to the system running the deployment server, click 'Settings', 'Data Inputs', under Forwarded Inputs 'Files & Directories', 'Add new'. I progress through the wizard until asked about the index. I can not see the indexes that were created on the other system.

If I use the same process on the Search Head/Indexer, the first page of the wizard gives me a "There are currently no forwarders configured as deployment clients to this instance."

This all leads me to believe I am using the system incorrectly. While the documentation goes thru how to setup everything, it doesn't really cover what instance you should do what functions under. It seems to me the %SPLUNK_HOME/etc/deployment-apps should be shared between systems, so the deployment server has them to deploy and the other systems have it to be configured via the web interface.

Any help in clearing this up will be greatly appreciated.

Thank you in advance,

Jeremy

[lguinn2]

Okay, here is the deal.

The deployment server is the only machine that has the /etc/deployment-apps directory. This is the repository from which apps will be deployed. There is a configuration file, serverclass.conf, that tells the deployment server which apps go to which deployment clients. (The deployment clients are forwarders, and possibly indexers, search heads etc.) When a deployment client receives an app, it stores it in /etc/apps - never in etc/deployment-apps.

When you are creating new inputs using the Splunk GUI, it can both create an entry in inputs.conf AND create a deployment app in /etc/deployment-apps AND create the appropriate entry in serverclass.conf. From the GUI, you can pick and choose indexes - if they are configured on the deployment server. There is no way for the deployment server to know what indexes actually exist on the indexers. Note that nothing is actually indexed on the deployment server (unless you are running an all-in-one Splunk that contains the deployment server and a single indexer all on one box.) Choosing the index is a "feature" of the GUI interface of the deployment server.

Finally, if you are creating this stuff "live" on your deployment server, then it isn't actually being tested first - and that is a bad idea, especially in a distributed production environment. So here is what I would do:

1) Create a server somewhere (could be a VM, could even be on your laptop) where you can test the inputs and create the apps.
2) On that testing server, create all the indexes that you will have in the production environment. Size them as small as you like, since they are just for testing.
3) Design your deployment apps and serverclasses. Yes, I know that the GUI will do that for you, but in the long run I think it can become hard to manage. At least, read this article and think about it: Deployment Server Goodies. The article is pretty old, but the remarks about app design are still completely on target.
4) Set up a test forwarder somewhere - a VM is fine. It doesn't need to even have any data, although it would be better if it did. Make the forwarder a deployment client of your test server.
5) Set up the "remote inputs" on your test deployment server and assign them to the right app. Confirm that the app is part of the serverclasses that you want, and that the forwarder will be restarted after the app is installed.
6) Make sure the app gets deployed to the client and that it works as it should.

7) Deploy into production: Copy the app from the test server's /etc/deployment-apps to the actual deployment server
8) On the production deployment server, run splunk reload deploy-server to have it scan for new apps
9) On the production deployment server, add the app to the appropriate existing serverclass or create the new serverclass, making sure that it has the right clients.

One of the saddest posts I've ever seen on this forum started with "Help! I accidentally loaded the wrong app onto my deployment server - what do I do now?" My immediate thought was "You are going to be up all night and it isn't going to be pretty..." You don't want to be that person.

[jwinderDDS]

Avoiding the whole "I copied over the wrong app" is the whole reason we prefer to use the web interface. The only thing we are using the deployment-app for is pushing inputs configuration to forwarders. All of the apps on the Search Heads are handled via the Manage Applications.

But this brings to mind a good point. Having an application to configure the forwarders and a separate one for saved searches, alerts, and dashboards, won't work anymore because we'll need to also deploy that to the indexes.

There is a lot of reading in my future.

[lguinn2]

Well, it will work to have separate apps - you just need to put the app for the indexers on the deployment server, and configure the indexers as deployment clients.

As you are not clustering the indexers, you can manage their configurations with the deployment server. And if you ever move to indexer clustering, you will have a cluster master to manage the indexer configurations.

Same for the search heads. Do a "search head" version of the app and deploy it via the deployment server as well... all you really need to remove is the definition of the indexes.

[somesoni2]

If I'm not wrong, you're doing it wrong. The data import configuration should be configured/setup in Universal forwarders. So, you'd create a Splunk data input app in etc/deployment-apps folder, create a serverClass for Forwarders and have this data input app to the forwarders.

See how to create inputs.conf for setting up data inputs here
http://docs.splunk.com/Documentation/Splunk/6.2.0/Data/Configureyourinputs#Edit_inputs.conf

How to setup serverclass.conf (which I believe you already are doing)
http://docs.splunk.com/Documentation/Splunk/6.2.0/Updating/Useserverclass.conf

[jwinderDDS]

If you use the Forwarded Inputs under 'Settings', 'Data Inputs', it will create an directory under etc/deployment-apps and create the needed inputs.conf file. It will also allow you to select an existing server class or create a new one.

Even with a Linux background, I do not like editing configuration files by hand if there is an interface. This way I don't have to look up the specifications every time and it helps to eliminate typos that will break the entire system.