The model listing answers from a cache and refreshes behind it - #342
Merged
Merged
Conversation
Every load of the model list fetched the gateway catalog and pinged each provider family, about a second here and several on a deployment whose gateway and providers are further away, and the client asked three times per mount. The computed listing is now kept per viewer and credential: a load inside the fresh window is answered from it, an older one is answered from it too and refreshed in the background so the next load is current, and only the first load per credential waits. ?refresh=1 is the user saying "look again"; it drops the probes and recomputes before answering, and a key test in Settings clears every listing. A model whose last call failed is decorated after the cache, so that mark stays live. On the client one request serves the picker, the banner, the credential notice, and the fleet page, with a fifteen-second reuse so a remount does not refetch. Status stays fresh without a reload: the list is asked again when the tab comes back into view and every five minutes while it stays visible, and the picker is remounted only when the set of marked models actually changes, since a remount mid-reply would drop the visible stream. Measured on the dev server: first load 1.2 s, the next three under ten milliseconds, a forced refresh 0.4 s.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every load of the model list fetched the gateway catalog and pinged each provider family (about a second on dev, several seconds where the gateway and providers are further away and each user's credential probes on its own), and the client asked three times per mount.
Server: the computed listing is cached per viewer and credential (the listing is shaped by the viewer, since shared providers are labeled with their owner). A load inside the 60 s fresh window is answered from cache; an older one is answered from cache and refreshed in the background (deduplicated), so the next load is current; only the first load per credential waits.
?refresh=1drops the probes and recomputes before answering, and the key test in Settings clears every listing. The last-call-failed decoration is applied after the cache so it stays live.Client: one deduplicated request (
fetchModelsinapi.ts) serves the picker, the banner, the credential notice, and the fleet page, with a 15 s reuse. Freshness without a reload: refetch when the tab becomes visible and every five minutes while visible; the picker is remounted only when the set of marked models changes, never mid-reply.Measured on dev: first load 1.2 s, next three under 10 ms, forced refresh 0.4 s. Tests cover cached loads, refresh seeing a changed catalog, and the live decoration. 187 server and 12 web tests pass.