How do I configure default apps with deployment server?

[Joffer]

I've got my custom apps (mostly custom inputs.conf files) working, and now I'm looking into configure some of the default apps, at this point SplunkLightForwarder (enable it) and sample_app (disable it).

I've got the forwarding functioning with my own "fwd-to-splunk01" app (outputs.conf file), but this doesn't kill the webserver on the server, neither does it run in forward only mode, all which the SplunkLightForwarder does.

I did try this simple config, hoping it would work, but it didn't:

[serverClass:LightForwarder:app:SplunkLightForwarder]
stateOnClient = enable
restartSplunkd = true

I know the reason. Reading the splunkd.log I see this:

08-16-2010 16:57:46.452 WARN  Application - Application: SplunkLightForwarder cannot be loaded as path: /opt/splunk/etc/deployment-apps/SplunkLightForwarder does not exist.
08-16-2010 16:57:46.452 WARN  ServerClass - Unable to load application: SplunkLightForwarder

Creating a /opt/splunk/etc/deployment-apps/SplunkLightForwarder/local/app.conf which could enable the app replaces the entire /opt/splunk/etc/apps/SplunkLightForwarder on my client. I've read other places here on Splunk Answers that I should link or copy the whole app from /opt/splunk/etc/apps/SplunkLightForwarder to /opt/splunk/etc/deployment-apps/SplunkLightForwarder on my server, but I was hoping there were other means to get this configured (and the only config I want at this point is to activate LightForward and disable sample_app).

Why I'm asking, is that if I link from apps to deployment-apps on my server and then upgrade my splunk indexer/deployment server (they are on the same server), my clients that happen to not be upgraded at that time could potentially get a config that is not working with the version installed on the client. Or the other way around.. (newer client doesn't like old defaults for LightForwarder). Is this a valid problem?

Comments?

[bbingham · Accepted solution]

You would be correct that I missed seeing you already had it up and working. Yes I do keep my deployable apps as copies, not links for the reason you specified, 1 change will push out the entire app onto the remote device.

Really, the splunk lightweight forwarder is 5 conf files, and all those conf files should withstand an upgrade path regardless of version. So if it's the only thing you plan on using the deployment server for, you'd probably be safe with a link. Other applications that use API's are the ones you should worry about an upgrade on (unix app, windows app, search app).

If you decide you'd like to make your own app (as you were doing above), web.conf shuts off the web service, limits.conf sets a max on bandwith and default-mode.conf is used to turn off some advanced features on the forwarder to lower cpu usage. Copying those files (along with your outputs you created earlier) into any app you maintain would have the same effect.

[bbingham]

Under your lightweight forwarder, check the web.conf under deployment-apps/SplunkLightForwarder/default and make sure it looks like this:

 [settings]

 # disable the webserver
 startwebserver = 0

and then make sure there is not a web.conf under the local directory.

The disable should override the etc/system/local/web.conf that says enabled.

[Joffer]

Isn't this the defaults? I copied it from /apps/ and did not touch any files after that. If I create a local/app.conf with state = enabled it works, but I didn't think I had to do that when I use deployment server - I enable the app in serverclass.conf

[Joffer]

I've got a followup question...

I've now copied the SplunkLightForwarder into my deployment folder. I did not add any new files/config to that app. My serverclass.conf for that app is as follows:

[serverClass:LightForwarder:app:SplunkLightForwarder]
stateOnClient = enable
restartSplunkd = true

The app is distributed to the clients and enabled, but the webserver is still running. Why? Do I need to add a local/app.conf (or should I enable default/app.conf?) file to the folder and 'enable' it there? I thought the stateOnClient should take care of that and enable the app and hence disable the webserver?

[Joffer]

There is a 'typo' in my config. Is should be 'stateOnClient = enabled'. Missing a 'd'. After fixing this it worked fine 🙂

[gkanapathy · Accepted solution]

Basically I ignore whatever apps might happen to come with the forwarder and set everything on the Deployment Server. That might be a copy or it might be a link. It depends on how you want to manage your Deployment Server, but a copy is usually the way to go, as most people would want upgrades of the forwarders separated from upgrades of the DS.

[bbingham · Accepted solution]

You would be correct that I missed seeing you already had it up and working. Yes I do keep my deployable apps as copies, not links for the reason you specified, 1 change will push out the entire app onto the remote device.

Really, the splunk lightweight forwarder is 5 conf files, and all those conf files should withstand an upgrade path regardless of version. So if it's the only thing you plan on using the deployment server for, you'd probably be safe with a link. Other applications that use API's are the ones you should worry about an upgrade on (unix app, windows app, search app).

If you decide you'd like to make your own app (as you were doing above), web.conf shuts off the web service, limits.conf sets a max on bandwith and default-mode.conf is used to turn off some advanced features on the forwarder to lower cpu usage. Copying those files (along with your outputs you created earlier) into any app you maintain would have the same effect.

[bbingham]

Did you bind the forwarded machine to this deployment server? On each remote machine you wish to pull apps from you need to add a conf file to etc/system/local named deploymentclient.conf

This file should look as simple as this:

[deployment-client]
clientName=AIXClient

[target-broker:deploymentServer]
targetUri= splunk.servers.healthtrans.com:8089

Once you've added a deployment machine, immediately on startup you should see lines like the following in the splunkd.log on the forwarder:

07-29-2010 17:37:45.921 WARN  DeploymentClient - DeploymentClient has been asked to redo-handshake. Resetting to inital state.

On the main splunk instance you can run the following command to list all connected deployment clients:

./splunk list deploy-clients

Make sure you see the remote machine inside that list before attempting to push any applications.

Once you see your client you can start pushing out remote apps, below is the lightweight forwarder from our environment (etc/system/local/serverclass.conf on the main splunk server):

[global]
repositoryLocation = /Applications/splunk/etc/deployable-apps
whitelist.0=*

[serverClass:forwarders]

[serverClass:forwarders:app:SplunkLightForwarder]
whitelist.0=*
stateOnClient=enabled
restartSplunkd=true

The first global tag specifies where all our deployable apps are located, and allows all machines to connect to the deployment server, after that, a simple forwarders block allows all machines to download the light weight forwarding app. Once you make a change to the serverclass.conf, you can run:

./splunk reload deploy-server

to have it pick up any changes without a restart. The splunkd.log on the main splunk deployment server should list lines like the following:

splunkd_stderr.log:09-22-2009 21:04:53.482 DEBUG DeploymentClient - Setting : disabled=false
splunkd_stderr.log:09-22-2009 21:04:53.483 DEBUG DeploymentClient - Setting : workingDir=/Applications/splunk/var/run
splunkd_stderr.log:09-22-2009 21:04:53.483 DEBUG DeploymentClient - Setting : clientName=deploymentClient
splunkd_stderr.log:09-22-2009 21:04:53.483 DEBUG DeploymentClient - Setting : repositoryLocation=/Applications/splunk/etc/apps
splunkd_stderr.log:09-22-2009 21:04:53.483 DEBUG DeploymentClient - Setting : serverRepositoryLocationPolicy=acceptSplunkHome
splunkd_stderr.log:09-22-2009 21:04:53.484 DEBUG DeploymentClient - Setting : serverEndpointPolicy=acceptAlways
splunkd_stderr.log:09-22-2009 21:04:53.484 DEBUG DeploymentClient - Setting : maxRetries=3
splunkd_stderr.log:09-22-2009 21:04:53.484 DEBUG DeploymentClient - Setting : waitInSecsBetweenRetries=60
splunkd_stderr.log:09-22-2009 21:04:53.484 DEBUG DeploymentClient - Setting : phoneHomeIntervalInSecs=60

If everything is going correctly, the remote client should now download the lightweightforwarder and restart once it's been installed.

Hope this helps!

[Joffer]

Thanks for the info and the effort, and not to be rude, but I don't think you read my post very thorough... I have the deployment up and running and working just fine 🙂

I did have a question/discussion (with my self?!!?) about SplunkLightForwarder.

What are your contents of the /Applications/splunk/etc/deployable-apps/SplunkLightForwarder? Did u copy or link the $SPLUNK_HOME/etc/apps/SplunkLightForwarder to /Applications/splunk/etc/deployable-apps/SplunkLightForwarder?

Please read on my post again 🙂