What are the best practices for S.o.S - Splunk on Splunk to monitor a Search Head Cluster tied with a Multisite Indexer Cluster?

[singhbc]

I need to install S.o.S - Splunk on Splunk to monitor a Multisite Indexer Cluster and Searh Head Cluster:
2 Sites with 2 Peers Each
3 SH on the SH Cluster
Master and Deployer reside on the same VM.
Multiple Forwarders.

The documentation that's available on Splunk Docs for Planning the SOS installation does not cover SH Cluster.

[bmacias84]

I would advise against use sos be instead configure Distributed Management Console. The DMC has a lot more information and is more responsive. Works well with clustered search heads and indexers. It really easy to setup

[hexx]

A fine point indeed, @bmacias84! If you are running Splunk 6.1 or above, you should be setting up a Distributed Management Console to monitor your Splunk deployment.

[sk314]

I recently installed S.o.S on our clustered Splunk(6.2) instance. As per @bmacias84, I went ahead and configured DMC. It's really cool! Thanks for the tip.

[hexx]

In addition to the Splunk Answer that @sk314 points out, I will indicate that as of right now S.o.S has not been certified to work without a hitch on a search-head cluster.

I would expect that the app will work fine, with one exception: It's dynamically generated lookups which are used as asset tables and are maintained on a per-instance basis which may not be entirely compatible with an SHC.

In order to circumvent that, you'll probably need to do something like this:

• Push the latest version of Sideview Utils to your SHC via its App Deployer
• Stage the S.o.S app before pushing it to your SHC via its App Deployer by renaming every *.xml.bak file to *.xml in the app's default/data/ui/views. This can be accomplished with the disable_FTR.sh script located in that directory.
• Push the S.o.S app to your SHC via its App Deployer
• Disable the "sos_refresh_splunk_servers_cache" scheduled search in the S.o.S app
• Populate the $SPLUNK_HOME/etc/apps/sos/lookups/splunk_servers_cache.csv table manually instead, following the specifications recorded in splunk_servers_cache.spec.

[sk314]

check out this post: http://answers.splunk.com/answers/81351/what-are-the-best-practices-for-installing-sos-on-cluster.ht...