When using SSO with clustered search heads, users who lose SSO access leave behind knowledge objects and directories on the file system. I'm doing some work to clean these up. In order to be able to query the Splunk API for the full set of information, it's necessary to re-create the user so that Splunk will see and return information about their private knowledge objects. While doing this, I noticed the following problem:
I can ask the API about the knowledge objects owned by a user via /servicesNS/-/-/admin/directory
However, if I ask about saved searches from /servicesNS/USERNAME/search/saved/searches/SEARCH_NAME, I get a 404 for searches that are present in (1)
After 5-10 seconds, (2) begins to work
I suspected that maybe the search head cluster needs to sync some configuration, so I hit /services/shcluster/status while (2) is failing repeatedly to get some info about the search head cluster state. None of the search heads specify they are out of sync, and the last conf replication time reset (indicating a configuration replication had happened), but the API was still returning 404s on saved searches for a few seconds.
Is there any way to know when it's "safe" to request information pertaining to a user? Is the /directory endpoint potentially affected by this? Are there other endpoints that may be affected in the same way?
One other thing I tested was querying /servicesNS/-/search/saved/searches/SEARCH_NAME, however it exhibited the same behavior. Not all users seem to behave this way, but the particular user in question had a couple knowledge objects of type "props-extract". It seems likely that re-adding those to the system is taking longer, and this added delay somehow affects their saved searches showing up.
... View more