At each splunk start a script scans the glasstable_icon_library.conf and imports anything new to the kvstore.
The UI then works off the kvstore.
In this case it looks like the KVstore is having issues.
Here is the error message:
2018-06-08 18:46:48,091 INFO import_icons_SA-ITOA: Glasstable icon importer started
2018-06-08 18:46:49,015 INFO Starting new HTTPS connection (1): 127.0.0.1
2018-06-08 18:46:49,023 INFO Starting new HTTPS connection (1): 127.0.0.1
2018-06-08 18:46:49,029 INFO Starting new HTTPS connection (1): 127.0.0.1
2018-06-08 18:46:49,034 ERROR HTTP 503 Service Unavailable -- KV Store is initializing. Please try again later.
Solution:
To manually restart the script go to https://[your_ITSI_SH]/en-US/manager/itsi/data/inputs/script and disable and then enable the “/opt/splunk/etc/apps/SA-ITOA/bin/import_icons_SA_ITOA.py” script.
If the script is successful you will see the following entry in the log /opt/splunk/var/log/splunk/gt_icon_collection.log
2018-06-08 19:26:12,931 INFO import_icons_SA-ITOA: Successfully imported 37 icons to kvstore
