How to copy saved searches, dashboards, alerts, etc. from a single instance to a search head and indexer clustering environment?

[sim_tcr]

Hello,

We have a need to copy saved searches, dashboard, alerts etc from one Splunk instance (not clustered) to another Splunk clustered setup. What is the best way to do this?

The single instance acts as the search head and indexer.
The clustered setup has 4 search heads with search head clustering enabled which has its own search head deployer and 6 indexers with index clustering enabled with a master server.

The single instance has several saved searches, dashboard, alerts under search app context (public) and some under user folders (private). Below is where the items are located,

SPLUNK_HOME/etc/users/<id>/search/local/savedsearches.conf 
SPLUNK_HOME/etc/users/<id>/search/local/data/ui/views/xmls
SPLUNK_HOME/etc/apps/search/local/savedsearches.conf
SPLUNK_HOME/etc/apps/search/local/data/ui/views/xmls

Thanks,
Simon Mandy

[sloshburch]

Hey Simon,

Apps: You'll want to copy any config from the $SPLUNK_HOME/etc/apps directory (and anything that slipped into the system/local folder). For the apps that come WITH splunk, you'll just want their local directories.

Users: Copy all of the $SPLUNK_HOME/etc/user directories

Place all of them in the respective app or user directory on the deployer for pushing out to the SHC members. From then on, NEVER manually edit conf on any SHC member as it will cause the entire cluster to go out of sync.

Here's a link that covers this is stronger detail: http://docs.splunk.com/Documentation/Splunk/6.5.1/DistSearch/Migratefromsearchheadpooling

[nls7010]

Will this work even though the users have not logged in and created their folders under /etc/users?

[nls7010]

I'm probably not clear, so let me try to clarify. The system I'd like get the objects onto is a Splunk 6.6.3 Linux installation from a Splunk 5.0.4 environment. The Splunk 5 is just two servers (1 search head, 1 indexer). I can see the objects under the user names in the old environment. But I noted in the savedsearches files under the users that the document contains the server the search was created on. Will that cause a problem in the 6.6.3 environment which will have different servers the search should appear on? Also, the users haven't logged into the new system yet, so their folders don't exist in the new system. Is it possible to create the folder before a user has logged in and place the old objects in it? In the saved searches document would the single splunk server name need to be changed to the new Linux servers (search heads).

[sloshburch]

Oh wow! So let me make sure I understand this correctly. You are trying to migrate knowledge objects from one install to another AND things are made complicated by the fact that your changing versions.

Assuming that is accurate then I would STRONGLY recommend upgrading the 5.0.4 version and only then, migrating the content.
As a classic IT change management principle, you'd always want to avoid conflating changes. By doing the upgrade, you can be sure the upgrade process will update any conf to the version you need in case changes were made. Plus you can test before changing what instances it lives in. If you do it all at once, you will struggle to identify if any issue is related to the move or the version change.

"savedsearches files under the users that the document contains the server the search was created on" - this doesn't sound familiar with me. Maybe provide a sample of the config so we can see exactly what attribute you are seeing? I'm hoping it's something that is no longer relevant in 6+.

FYI: If you do an upgrade, you should check the docs on the upgrade path. They may not allow jumping straight from 5 to 6.6. You may need to go from 5.0 to 6.0 and increments from there.

[sloshburch]

(Changing this to a comment, not another potential answer)

If the user never logged in then they likely won't have a folder under the $SPLUNK_HOME/etc/user directory. So, you still want to exhaustively identify what content DOES exist in that directory but it sounds like you may not have many items.

I'm not sure if that answers your question cause it actually feels like you answered your own question. What is it that your worried about not capturing assuming the user never logged in?

[jkamdar]

Thanks @sloshburch. We are in the process of doing the same thing from the stand-alone SH to cluster using a deployer. I am concerned about all the Knowledge objects.
So, copying $SPLUNK_HOME/etc/users $SPLUNK_HOME/etc/apps at once to deployer and finally to all the cluster member, will it migrate all the knowledge objects? I am afraid, I will mess it up 😞

[sloshburch]

I get that. This is a great time to justify getting a lab environment to test such activities.

Your understanding sounds accurate. In fact, here's a more effective page that I think you'll find is just like what you described: http://docs.splunk.com/Documentation/Splunk/latest/DistSearch/Migratefromstandalonesearchheads

[jkamdar]

@SloshBurch, thanks for your response.

Thanks for the link to the document. That's the document I started with, the section - Migrate settings to a search head cluster. As I was going through the steps, I was little confused.

Following those steps, we did migrated with a couple of apps and it looked like, the permission didn't migrated properly. Here is the observation by the owner app:

Everything came over, except a private search owned by the user. My guess is that even though the user exists, he never logged on, so his home directory didn't exist.
Private search, tags, and eventtype belonging to another user now has No Owner permissions change to be shared to the app: Everyone read/write.
The app was owned by no-one and Maybe it was because, we didn't copy user(s).

So, as I read your response, you are confirming that, copying those 2 directories (/etc/apps and /etc/users) will take care of migrating everything, please confirm? Sorry, if I am being overly concerned.

[sloshburch]

It's hard to follow exactly what they are referring to there but I appreciate you sharing the message from them.

Make sure the metadata folder is brought over. Also, if the config lived in the users directory before, then putting it in the shcluster/users directory should keep permissions the same.

Finally, yes, the etc/apps and the etc/users but please follow the instructions. There are some apps (like search) that come with Splunk and you would't want the deployer trying to overwrite them.

Everything you said is logically accurate. Since you have expressed concern to get it right, I would use that same motivation to set up a lab where you can copy production and test this change OR hire PS to do this migration for you.

Otherwise, good luck and take backups!