Best practices for migrating an app we've created from one Splunk instance to another

[sideview]

<posted on behalf of a customer>

I see there are app-packaging instructions on http://docs.splunk.com/Documentation/Splunk/6.1/AdvancedDev/PackageApp .

We already have everything in its own folder in etc/apps, and everything that's a core part of the app is in default and we have an app.conf and all that stuff. We're not moving it to Splunkbase or anything so we don't need the screenshot and all that stuff.

but what are the best practices and strategies (or worst practices and pitfalls)?

[cmeo]

Can only agree with Nick and add this:

Splunk should do more to help developers 'unspaghetti' the results of a 'rich and organic editing history'. I've believed for a while that Splunk web should have a Developer Mode or something like that which explicitly puts all generated content (saved searches, props, transforms, UI etc.) into default in a specific App context which you should be able to select when firing this mode up.

That way, if your exact objective is to create content for an App which is intended for publication either to customers or Splunkbase, you know where the bits are going.

A tool to merge default and local would also be really nice to have, and a few Apps have more or less had a swing at it, but as the problem stands it's very difficult to solve. I'd also like to see built in hooks to the source code system of your choice, but I won't hold my breath for any of this.

The difficulty being that as many Splunk instances mature, they become really, really messy.

cmeo

[sideview]

Most of that advice is for if you're packaging an app to be downloaded from Splunkbase, or distributed to new users or customers some other way. Plus it sounds you've already done most of it anyway.

At the very very simplest you'll literally copy etc/apps/YOUR_APP from one server to the other.

If you've done everything right, like for instance you never allowed anyone to develop or modify core app functionality by editing things in the Admin UI, then that's it. Copy the whole thing and then edit the various local-overrides in the app's local directory. Wheee

But of course nobody does everything right!

The default/local split is familiar - In an app the "local" directory should have only overrides for that one instance. Conversely, if there's something in APPNAME/local and there's no way that thing would ever be different from instance to instance, it shouldn't be there - move it to the default directory in the app.

The trick is that the same problem happens at lots of layers. App development on a live instance is messy and there will have been config orphaned off into other weird places. You'll have to find as many of these as you can first.

-- stuff in "system/local" when it's really part of the app and should be in APPNAME/local
-- stuff in "etc/apps/launcher/local" (when you click "Settings" from Home, anything you go on to create ends up in "launcher" =/ )
-- stuff in "etc/apps/search/local"

-- stuff that is in users/admin/APPNAME/local, or even the search or launcher app, that may have been "fixed" by being later exported to system level (they no doubt meant well!)
-- stuff got knows where.... If it's effectively being used by lots of users from other weird apps, one shortcut is to go the reverse way and audit all the *.meta files on the system.

Be pedantic and be paranoid. Leave some time to search on the filesystem following dependencies trying to find orphaned config.

Config that is inherently not namespaced, like indexes.conf - can easily hide in other apps. but Namespaced stuff like searches and eventtypes can hide in crazy places as long as someone exported it to system later, and it'll appear to work great in your app. If you find this has happened, dont forget to clean up or delete all the local.meta and default.meta as well.

Also promote any local.meta entries you find to default.meta... (You shouldn't ship with local.meta)

For both larger apps with lots of conf and also for apps with "rich and organic editing histories", I recommend being super methodical and working from the outside in:

• etc/users/admin/search/local delete local.meta stuff or move to default, clean or move default.meta. If other admin accounts might have done something evil assume they did.
• etc/users/admin/launcher/local (same stuff wrt *.meta)
• etc/system/local
  
• etc/apps/search/local
  
• etc/apps/launcher/local

you get the hang of it.

• FINALLY, etc/apps/APPNAME/local looking for the stuff that should be in default.

There may also be stuff in here that has absolutely nothing to do with this app at all! move it to another app or to system. Be pedantic. delete local.meta entries or move to default, clean up or move or delete default.meta entries. Ideally leave no broken windows but at least leave it in a better state and finish what you start.

If you actually do this, you'll find like 20 other things that are broken in your app. Sorry. Deadlines are fun!

if you are shipping this app to third parties ever, you would delete the local directory entirely and delete local.meta before packaging the app. The reason being of course that subsequent upgrades of a shipped local file would clobber the local local file.... and people dont like their own config to be horribly and unexpectedly deleted later on.

(Splunkbase wont let you upload with local foo in an app I don't think but if you're distributing in some other way you have to remember this rule yourself. )

Lookups can be tricky. Sometimes they're core to the app and sometimes they're really artifacts specific to a particular deployment. If they're the latter kind then you shouldn't ship them, and in fact you shouldn't even ship any file at all there cause of course it'll be clobbered on upgrades or redeployments later. What I do is have the app's homepage do a weird little FTR search to generate an empty lookup if there's ever none there, but ymmv.

And last but not least.... regarding default.meta and exporting things to system. Remember that if this instance has other apps in it, be hesitant to export things cause they'll clutter up and possibly disrupt the other apps and it'll ruin someone's day. imo exporting to system is an extraordinary thing. Remember that the "search" view is itself exported so there's no need for you to export your stuff so it works in the "search" app - go the other way and use the search view in your own app instead.

[dwaddle]

lots of good advice from someone who has obviously been there. We keep $SPLUNK_HOME/etc in a mercurial repository and can see based on the changelog in there what folks have done via the admin UI that might need to be fixed in the 'master copy' of apps