S.o.S and search-head pooling with job scheduler disabled

[rmorlen]

We just upgraded to 5.0.2 and I had a question about SOS.

We have 4 searchheads plus a jobs server. The jobs servers runs all scheduled searches. The 4 SH's have the scheduler pipeline disabled.

What will we be missing in SOS by doing this since no scheduled searches will be running on the SH's?

[hexx · Accepted solution]

The main issue is that S.o.S runs two lightweight scheduled searches to populate two lookups, one of which is heavily used in the app:

• sos_splunk_instances_info -> splunk_instances_info.csv -> used to populate the "A glimpse of your Splunk instance" info panel in the Home view
• sos_refresh_splunk_servers_cache -> splunk_servers_cache.csv -> used to populate the "Server to query" pulldown which exists in most views

At a minimum, you'll need to manually run those searches once to ensure that S.o.S discovers your topology. Given that you are in a search-head pool, you'll need to run this once per search-head in the pool.

Here's how you would run these searches manually:

• Go to the flashtimeline within the context of the S.o.S app - http[s]:[splunkweb_hostname]:[splunkweb_port]/en-US/app/sos/flashtimeline
• Run the following two searches:
  | savedsearch sos_refresh_splunk_servers_cache
| savedsearch sos_splunk_instances_info | outputlookup splunk_instances_info

Note that you should also run these searches whenever a new indexer or pooled search-head is added, to help S.o.S discover it.