ADFA-5094: catalog freshness service (weekly publish + on-device refresh) - #404
Merged
Conversation
…n-device refresh Amend ADR-4954's "staleness rides the APK OTA" sub-decision with the evidence it lacked (Books catalog ~2 months stale; ~83 KB catalog pull vs ~30 MB APK OTA). Move catalog generation off Gradle onto a weekly scheduled GitHub Actions workflow that uploads versioned catalogs + a fixed-key manifest to Cloudflare R2, reusing the update.json pipeline and the k2go-download.appdevforall.org base; the app pulls when online (bundled snapshot as the offline floor, superseded by an overlay), with TTL + ETag + hash for freshness. Content stays on the Nginx mirror; only the catalog metadata moves to Cloudflare. One catalog-agnostic mechanism; Kolibri first.
…oudflare R2
New scheduled workflow (weekly + workflow_dispatch): run the existing generators once,
build a fixed-key manifest per catalog (version, generated, sha256 hash, url, and
per-channel {id,version} for jsonl), and upload catalog + manifest to R2 under
catalogs/, reusing the update.json R2 pattern (aws s3 cp, CLOUDFLARE_* vars). Served
from k2go-download.appdevforall.org/catalogs/. Never fails on a blocked source fetch;
skips that catalog's upload and keeps the last published one.
Pure, catalog-agnostic domain for the on-device refresh (ADR-5094): CatalogManifest value type (catalog/version/generated/hash/url + per-item versions) and CatalogFreshness rules — dueForCheck (TTL, with a backwards-clock guard) and changed (hash is the truth, version only a label). Unit-tested. No Android; the data layer parses the manifest JSON into these and the worker consumes them.
…store) Data plumbing for the on-device catalog refresh (ADR-5094), catalog-agnostic: - CatalogManifestParser: manifest JSON -> CatalogManifest (null on malformed). - CatalogManifestClient: conditional GET with ETag/If-None-Match (304), plus a temp-file catalog download, mirroring the OTA update client's HTTP. Never throws. - CatalogRefreshStore: per-catalog prefs (last check, active hash/generated, ETag).
Catalog-agnostic on-device refresh (ADR-5094): CatalogRefreshWorker (TTL gate ->
conditional manifest fetch -> if the hash changed, download + sha256-verify ->
write the overlay), CatalogRefreshScheduler (weekly periodic + opportunistic
one-shot, network-constrained, unique per catalog), CatalogOverlay (the overlay
file path both the worker and the source use). Catalog to refresh is passed as
WorkManager input data {name, manifestUrl, basename}. Mirrors OutboundSyncWorker.
BundledCatalogSource now prefers a pulled overlay (filesDir/catalogs/kolibri_catalog.jsonl) over the APK asset when present, and self-invalidates when the overlay's mtime changes — no explicit callback needed. CatalogRepositoryImpl(Context) schedules the weekly + opportunistic refresh for the Kolibri catalog (the test constructor does not). With no overlay, behaviour is unchanged; the bundled asset stays the offline floor.
Show "Catalog updated on <date>" on the Kolibri browse screen, from the catalog's generated date (already carried by KolibriCatalogUiState). With the overlay-aware source, it reflects the pulled catalog's date once refreshed, else the bundled one; hidden when empty. New string k2go_kolibri_catalog_updated + 33 locales.
Review follow-up: BundledCatalogSource.ASSET and CatalogRepositoryImpl.BASENAME both held "kolibri_catalog.jsonl" independently, so the source (reads the overlay) and the worker (writes it) could silently drift. BASENAME now references ASSET.
luisguzman-adfa
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
First half of ADFA-5094: keep the Kolibri catalog current without an APK release. A weekly GitHub Actions workflow publishes a versioned catalog + manifest to Cloudflare; the app pulls it when online and overlays the APK-bundled asset, with a freshness label on the picker. The topic-tree depth (the ticket's other half) is a separate follow-up PR — nothing here depends on it.
Why
The catalog refreshes only when a new APK ships (the Books catalog is ~2 months stale) and generation hit Studio from the build. This centralises the source fetch to one weekly run and lets the app stay current independently, at ~83 KB vs a ~30 MB OTA. Amends ADR-4954's "staleness rides the OTA" sub-decision (see
ADR-5094).Notes
Independent of the depth half: with only this merged, topics keep working exactly as today (live from Studio); the catalog just gains freshness. 5094 stays open for the depth PR.
Content stays on the Nginx mirror; only the catalog metadata moves to Cloudflare.
New strings translated across all 33 locales.
Verification
javalangparse on the changed Java; XML well-formedness on the layout + 34 strings files;CatalogFreshnessunit assertions. Not yet device-tested: run the workflow once(
workflow_dispatch) to publish the URLs, then confirm the app pulls/overlays on a device.