K2GO-390 fix(kiwix): self-healing catalog to end the stale-download loop - #554
Merged
Merged
Conversation
…irst) Design authority for the fix to the "infinite loop attempting to download content" bug: the Kiwix catalog ships as a fixed asset with no freshness, so a pruned dated ZIM URL 404s and the provisioning drain re-attempts it forever. Decision: a self-healing FLAT catalog. Keep the asset as the offline baseline; layer a light, connectivity-guarded freshness on top, reusing Kolibri's catalog-agnostic core (manifest + ETag + overlay + store + scheduler, ADFA-5094) -- and NOT its N-level tree machinery, which Kiwix does not need. Hybrid trigger: a TTL-gated conditional (ETag/304) check at catalog open, plus a forced refresh on a 404. The wishlist key is date-free (project|lang|flavour), so the same selection re-resolves to the current dated file after a refresh; the refresh itself is the classifier (changed file = was stale, heal + retry; unchanged = real failure, stop bounded). No UI, no size re-confirm. The infinite loop is fixed as a consequence. Requires publishing catalogs/kiwix.manifest.json + the refreshed CSV (build_kiwix_catalog.py already exists).
… loop A stale baked kiwix_catalog.csv points a ZIM at a dated URL that Kiwix has pruned (only the YYYY-MM date rolls over; the identity is stable). It 404s (aria2 code 3), and because a FAILED item was never removed from the wishlist, the provisioning drain re-attempted it every ~2s forever (device-proven). ADR-390. Integral fix, reusing Kolibri's catalog-agnostic freshness core (no tree machinery, Kiwix is flat): - KiwixCatalog is overlay-aware (prefer the pulled CSV over the APK asset; the asset stays the offline baseline). It nudges the refresh once per process (weekly + opportunistic, TTL-gated) and loads under a lock. - On a download failure the guard forces a refresh (expedited, network-constrained so offline is a no-op). The date-free wishlist key re-resolves to the current dated file on the next drain -> heal. Failures are counted against the catalog version (overlay mtime); a refreshed catalog renews the budget, and after MAX_HEAL_ATTEMPTS against an unchanging catalog the item is dropped -> the loop is bounded, no more infinite re-drain. - CatalogRefreshWorker/Scheduler gain a force flag (bypass TTL) + an expedited forceRefresh; catalog-agnostic, so Kolibri benefits too. - ContentDownloadSession.Host gains onItemError (default no-op; books unaffected). - build_kiwix_catalog.py --manifest/--csv-url emit kiwix.manifest.json (sha256+url) for the hosting pipeline (mirrors kolibri.manifest.json). Offline-first: no network -> asset/overlay serve with no error. No UI, no size re-confirm (month-to-month delta is negligible). Compiles clean; two-pass review applied (expedited heal, synchronized load, once-per-process nudge).
…/K2GO-390-kiwix-catalog-freshness
Review second pass: ZimDownloadService.onItemError re-derived the catalog version by re-opening the overlay with a hardcoded basename, duplicating KiwixCatalog's CSV_ASSET and its overlay-mtime notion. Expose KiwixCatalog.catalogVersionTag(ctx) so the version identity has a single owner; the service calls it. No behavior change.
…e flavour
Kiwix re-releases within a month add a letter suffix (2026-07a, 2026-07f).
parse_name only treated \d{4}-\d{2} as a date, so the letter leaked into the
flavour: terraria.wiki.gg_en_all_2026-07 and _2026-07a produced different
entry keys (all vs all_2026-07a). The self-heal key is creator+flavour, so a
letter roll-over broke re-resolution -- the item could not heal and was dropped.
Match \d{4}-\d{2}[a-z]? in one shared DATE_RE so both date checks agree and the
key stays stable across a re-release. Needs a catalog regen to take effect.
The catalog map key joins creator and flavour with a U+0001 control char so "<creator><flavour>" keys cannot collide across rows. It was written as an invisible char inside a "" literal, so the source read as an empty separator and any key built from parts (a deep link, external tooling, a hand-staged wishlist) silently failed to match the catalog. Spell it out as KEY_SEP = "\u0001" -- the same char, so existing persisted keys still match (no behavior change, no migration); it is now visible and greppable. Drop the same stray char from the class-doc comment.
…tate - ADR-390: Status Proposed -> Accepted (implemented, device-verified). Section 7 hosting is now live (manifest + CSV published; a 404 self-healed end to end on device 2026-09-05), not a pending dependency. Record the re-release letter in the date (YYYY-MM[a-z]?) and the DATE_RE fix; correct the wishlist key to project|lang|<creator+flavour>; state the MAX_HEAL_ATTEMPTS (5) bound; mark the device verification done with results (unit coverage of the wiring is a follow-up). - KiwixCatalog: the shape comment now shows the entry key joins creator and flavour with KEY_SEP (was misleading after the delimiter was made explicit). - build_kiwix_catalog.py: document the --manifest ops path in the usage header.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The Kiwix ZIM catalog shipped as a fixed asset with no freshness. Kiwix prunes
old dated ZIMs, so a baked entry points at a URL that 404s (aria2 code 3). A
failed item was never removed from the wishlist, so the post-install drain
re-attempted it every ~2s forever.
This makes the catalog self-healing and bounds the retry:
asset stays the offline baseline), reusing the Kolibri freshness core (manifest
date-free wishlist key re-resolves to the current dated file on the next drain
-> heal. Failures are counted against the catalog version (overlay mtime); a
refreshed catalog renews the budget, and after 5 failures against an unchanging
catalog the item is dropped -> the loop is bounded.
forceRefresh; catalog-agnostic, so Kolibri benefits too.
hosting pipeline.
Two catalog-constructor bugs found and fixed while verifying on-device:
\d{4}-\d{2} as a date, so the letter leaked into the flavour and destabilized
the self-heal key across a roll-over. Fixed in build_kiwix_catalog.py (a shared
DATE_RE).
inside a "" literal. It works internally (all sites copy the key) but reads as
an empty separator and breaks any key built from parts. Now an explicit
KEY_SEP constant (same char, no migration).
Why
Offline-first: with no network the asset/overlay serve with no error. No UI
change and no size re-confirm (month-to-month delta is negligible). Authority:
controller/docs/ADR-390-kiwix-catalog-freshness.md.
Verification
Device (OnePlus, arm64, API 35), with a real stale entry
(bulbagarden_en_all_nopic 2026-07, 404 on the mirror):
-> downloads.
loop stops.
Coordination / follow-up
Ops: regenerate and publish the catalog (asset + hosted CSV/manifest) by
re-running the fixed generator, so the letter-date entries collapse and can heal.
This re-runs the fixed constructor -- no manual data patching.