We just upgraded to 5.0.2 and I had a question about SOS.
We have 4 searchheads plus a jobs server. The jobs servers runs all scheduled searches. The 4 SH's have the scheduler pipeline disabled.
What will we be missing in SOS by doing this since no scheduled searches will be running on the SH's?
The main issue is that S.o.S runs two lightweight scheduled searches to populate two lookups, one of which is heavily used in the app:
At a minimum, you'll need to manually run those searches once to ensure that S.o.S discovers your topology. Given that you are in a search-head pool, you'll need to run this once per search-head in the pool.
Here's how you would run these searches manually:
http[s]:[splunkweb_hostname]:[splunkweb_port]/en-US/app/sos/flashtimeline
| savedsearch sos_refresh_splunk_servers_cache
| savedsearch sos_splunk_instances_info | outputlookup splunk_instances_info
Note that you should also run these searches whenever a new indexer or pooled search-head is added, to help S.o.S discover it.
The main issue is that S.o.S runs two lightweight scheduled searches to populate two lookups, one of which is heavily used in the app:
At a minimum, you'll need to manually run those searches once to ensure that S.o.S discovers your topology. Given that you are in a search-head pool, you'll need to run this once per search-head in the pool.
Here's how you would run these searches manually:
http[s]:[splunkweb_hostname]:[splunkweb_port]/en-US/app/sos/flashtimeline
| savedsearch sos_refresh_splunk_servers_cache
| savedsearch sos_splunk_instances_info | outputlookup splunk_instances_info
Note that you should also run these searches whenever a new indexer or pooled search-head is added, to help S.o.S discover it.