Deploying updated searches, report, etc. to Production

[terryloar]

I would like to know if there is a standard deployment mechanism in Splunk. I want to update my reports, searches, dashboards, macros, users, etc. from a development computer to a staging (QA) environment then to a production environment.

For example, a non-private dashboard that I create is stored in:

$SPLUNK_HOME/etc/apps/search/local/data/ui/views/

as an xml file. But I do not know if there are any files that depend upon it or to which it is related in any way.

To summarize, how do you folks go from a development environment to a production environment?

Thanks for your help.

[gkanapathy]

First, you should understand the concept of "apps". In Splunk, "app" is not really very much like an application on a smartphone or PC. Instead, it's just a bunch of config files that are stored together in the same folder. It might be better to think of them as a "config bundle".

All Splunk configuration is in files within apps (or folders, or bundles). It's up to you to determine the dependencies, however, based on your app sharing and other items. For example, your dashboard may or may not reference a saved search. This saved search may be in the same app, or it may be in a different globally shared app. Depending on that, you will need to move a different savedsearches.conf file. In turn, the saved search may depend on macros or field extractions. Those may be defined globally, or they may be specific to the app.

One problem you'll run into is that you may have unrelated objects in the same file, if they're in the same app.

In practice, with some planning on keeping related objects in the same app, this is not really that complicated. It can be very complicated and hard to manage if you either create apps all over the place and make them all global, or if alternatively, you stick everything into a single app. Create a sensible folder structure, and for the most part, you move items that you have designed and related together.

[terryloar]

Thanks. The plot thickens!

It's a great program, but it would be very helpful if Splunk could put some resources into simplifying the development to production migration process; or, minimally, publish some guidelines.

[jbsplunk · Accepted solution]

You can copy the entire app, presuming its on the same version of the product, from $SPLUNK_HOME/etc/apps/ on the development system to the same location on the production system, which will ensure all of your searches, inputs, props, etc, will be preserved. You'll probably also want to copy $SPLUNK_HOME/etc/system/local/. This really should be all you need to do to go from dev to prod.

[terryloar]

Thanks, that a good starting point, but as 'gkanapathy' mentions below there are directories other than $SPLUNK_HOME/etc/apps which would be needed for updating a production server. $SPLUNK_HOME/var/ has many *.csv files which I created using the Splunk UI. Also" $SPLUNK_HOME/etc/system/metadata and $SPLUNK_HOME/etc/systemlookups.

I also isolated the paths to any files that had changed since Splunk was installed on my development machine. There were 1931 of them! That's a lot files to manage everytime I want to make a modification to the production server, even adding a simple search.