diff --git a/controller/docs/ADR-K2GO-394-maps-download-via-dashnode.md b/controller/docs/ADR-K2GO-394-maps-download-via-dashnode.md new file mode 100644 index 000000000..f6783f2ab --- /dev/null +++ b/controller/docs/ADR-K2GO-394-maps-download-via-dashnode.md @@ -0,0 +1,112 @@ +# ADR-K2GO-394 -- Base-map downloads run through dash-node, not the maps role + +## Status + +Accepted (2026-09). + +## Context + +The maps SETUP downloads large whole-world base-map pmtiles (vector, satellite, +terrain -- gigabytes at high zoom). Upstream's maps role downloads them in-proot +through `roles/maps/tasks/download_large_file.yml` (an `aria2c` over a `.meta4` +metalink). + +On Android the role runs in a proot with a mobile radio. An in-proot `aria2c` +that the app drove over JSON-RPC wedged at zero connections on a full network +drop and never recovered (K2GO-394): it did not exit, so nothing could retry it, +and the SETUP hung. We need the base-map download to be resilient (survive a +radio drop) and controllable (pause / resume / live progress) -- the same +properties ZIM and Kolibri downloads already have. + +The box already has an engine for exactly this: **dash-node**, the in-server +durable job engine (ADR-4832, ADR-4893). It downloads with `aria2 --continue` +plus an outer reconnect loop, reports "Reconnecting n/5", and survives a client +disconnect. ZIM, Books and Kolibri downloads all run through it (LIVE-REST class, +ADR-5061); the app POSTs and polls, the box owns the download. + +## Decision + +**Route the base-map download through dash-node, ahead of the maps runrole, and +do NOT patch the maps role.** + +1. **dash-node** gains a `basemaps` job (see the dash-node CHANGELOG, 1.3.2 / + 1.3.3): given a bare pmtiles file name it composes the mirror URL and + downloads into the maps serve dir (`/library/www/maps`) with the proven kiwix + reconnect mechanism. +2. **The app** orchestrates it, the same way it drives ZIM (`RestContentClient`): + for the maps module it resolves the selected layers to file names from the + catalog, POSTs `{ids:[...]}` to dash-node, shows the live download bar, and + only when the download finishes does it run the maps runrole. The download and + the runrole are two sequential phases, not one. +3. **Ansible (the maps role)** stays STOCK. Because dash-node has already placed + each pmtiles at its `dest_path`, the role's native `creates: dest_path` (and + the `.meta4` size-probe's own existence check) SKIP those downloads. The role + only post-processes (symlinks, `maps-config.js`). Nothing in the role changes. + +So three parts share the work -- ansible post-processes, dash-node downloads, the +app orchestrates -- mirroring the live-download pattern the rest of the app uses. + +## Why the role is NOT patched + +An earlier version patched `download_large_file.yml`: it gated the download +`when: not is_proot` and asserted dash-node pre-placement `when: is_proot`. This +**broke the CI rootfs bake**. + +`is_proot` is `True` for the Android tiers (`vars/local_vars_android_*.yml`), and +the bake builds those tiers. But the bake has **no dash-node and no app** -- it +builds the rootfs image and is meant to download the base maps itself. With the +patch, the bake skipped the downloads and the `is_proot` assert failed: + +``` +TASK [maps : Fail if dash-node did not place maps.black-component.js on proot] +fatal: assertion 'proot_basemap.stat.exists' failed +``` + +`is_proot` does not distinguish "a live device with dash-node" from "a CI bake +without it". Both are proot. So no `is_proot` condition -- inline or in a separate +task file -- is correct here. + +The native `creates:` skip needs no condition and is right in both contexts: + +- **Device:** the app pre-places the selected pmtiles through dash-node, so + `creates:` skips them; the role post-processes. +- **Bake:** nothing is pre-placed, so the role downloads everything itself over + the CI runner's stable network, exactly as upstream intends. + +Patching an upstream role only to say "we do not run this step here" would also +be a carry with no upstream value -- the opposite of the upstream-first policy in +`tools/upstream-patches/README.md`. + +## The search tarball stays in-proot + +The maps role also downloads the static-search database through the same task +with `expand_archive=true` (a `.tar.gz` it extracts). dash-node downloads files, +not archives -- it does not extract -- so search is **not** delegated. It +downloads AND extracts in-proot as upstream does. It is small (~16 MB), so the +in-proot download's exposure to a radio drop is short; a drop there fails the +role and the install's existing Retry re-runs it. + +## Consequences + +- The maps role reads exactly as upstream ships it. This divergence -- that on + the K2Go device path the base maps arrive from dash-node, not from the role -- + is invisible in the role itself, so it is recorded here and pointed to from + `tools/upstream-patches/README.md` (which is where a maintainer looks and finds + no maps patch). +- Anything the app does NOT delegate (the search tarball, the small map JS + components, or a selected layer the catalog cannot resolve) downloads in-proot + with the stock role behavior. The big, selected pmtiles -- the ones worth many + gigabytes and the reason this ticket exists -- are the ones delegated, so the + wedge-prone case is covered. +- The three-part split (ansible + dash-node + app) is more moving parts than a + lone role, but it is the same pattern ZIM/Books/Kolibri already use, so it is + not new machinery -- just a new content type on it. + +## References + +- ADR-4832 (live content channel / single proot dash-node core) +- ADR-4893 (download execution and user control) +- ADR-5061 (LIVE-REST vs STOPPED-proot operation model) +- dash-node CHANGELOG: 1.3.2 (basemaps runner), 1.3.3 (file-id + URL composition) +- `tools/upstream-patches/README.md` (why there is no maps download patch) +- Jira: K2GO-394 diff --git a/tools/upstream-patches/README.md b/tools/upstream-patches/README.md index a91d6043c..648811f39 100755 --- a/tools/upstream-patches/README.md +++ b/tools/upstream-patches/README.md @@ -77,3 +77,22 @@ When an upstream PR merges and ships in the pinned `iiab/iiab` commit, its patch no-op (reverse-dry-run skip). At the next maintenance pass, read each patch's `Upstream-Status`, delete the ones marked merged, and bump the pinned commit. Keep the set small. + +## Notable non-patches + +Sometimes the right carry is **no patch at all** -- recorded here so a deliberate absence is +not mistaken for an oversight. + +- **Maps base-map download (K2GO-394) -- no patch, on purpose.** Upstream's maps role + downloads the base-map pmtiles in-proot through `roles/maps/tasks/download_large_file.yml`. + On the K2Go device path, dash-node (the in-server durable job engine) pre-downloads them -- + app-driven, resilient, resumable -- into the maps serve dir BEFORE the runrole, so the role's + native `creates: dest_path` skips those downloads and it only post-processes. The role reads + exactly as upstream ships it. We deliberately do NOT patch it: an earlier `is_proot` + gate + assert broke the CI rootfs bake, where `is_proot` is `True` for the Android tiers too + (`vars/local_vars_android_*.yml`) but no dash-node runs -- so the bake must download the base + maps itself, which the stock role does. `is_proot` cannot tell "device with dash-node" from + "CI bake without it"; `creates:` needs no such flag and is correct in both. Rationale: + `controller/docs/ADR-K2GO-394-maps-download-via-dashnode.md`. (The search tarball is the one + map file NOT delegated -- dash-node does not extract archives -- so it still downloads and + extracts in-proot as upstream does.) diff --git a/tools/upstream-patches/patches/0003-maps-download-is-proot-delegate.patch b/tools/upstream-patches/patches/0003-maps-download-is-proot-delegate.patch deleted file mode 100644 index 4cf83f92a..000000000 --- a/tools/upstream-patches/patches/0003-maps-download-is-proot-delegate.patch +++ /dev/null @@ -1,36 +0,0 @@ -Upstream-PR: not yet submitted -Upstream-Status: open -Applies-to: roles/maps/tasks/download_large_file.yml -Summary: On proot, delegate the base-map PMTILES download to dash-node (the dashboard's durable job engine) instead of running a blocking aria2c in-proot. The in-proot aria2 could not recover from a full mobile-radio drop -- it wedged with no exit (K2GO-394). dash-node downloads the pmtiles with its own resumable, reconnecting aria2 (--continue plus an outer reconnect loop -- the proven kiwix mechanism) into the maps serve dir BEFORE this role runs. So a pmtiles download is gated `when: not is_proot` (unchanged off proot), and a new step asserts the file is already at dest_path. Archives (expand_archive, e.g. the search tarball) are NOT delegated -- dash-node does not extract them -- so they still download AND extract in-proot (the download task also does the extract/mv, and the search tarball is small so the wedge risk is low). The role then only post-processes the delegated pmtiles. Carried for our build; WIP for upstream (K2GO-394). - -diff --git a/roles/maps/tasks/download_large_file.yml b/roles/maps/tasks/download_large_file.yml -index f8aa705..78a55cc 100644 ---- a/roles/maps/tasks/download_large_file.yml -+++ b/roles/maps/tasks/download_large_file.yml -@@ -140,6 +140,26 @@ - args: - executable: /bin/bash - creates: "{{ dest_path }}" -+ when: not is_proot or (expand_archive | default(false)) -+ -+ # proot / Android (K2GO-394): dash-node -- the durable job engine -- downloads the base-map -+ # PMTILES with its own resumable, reconnecting aria2 (the proven kiwix mechanism) BEFORE this -+ # role runs, because a blocking in-proot aria2 cannot recover a mobile-radio drop (it wedges with -+ # no exit). So a pmtiles download is skipped here and only asserted present. Archives -+ # (expand_archive, e.g. the search tarball) are NOT delegated -- dash-node does not extract them -- -+ # so the task above still downloads AND extracts them in-proot (small; the wedge risk is low). -+ - name: "Base map {{ file_name }} must be present on proot (downloaded by dash-node)" -+ stat: -+ path: "{{ dest_path }}" -+ register: proot_basemap -+ when: is_proot and not (expand_archive | default(false)) -+ -+ - name: "Fail if dash-node did not place {{ file_name }} on proot" -+ assert: -+ that: proot_basemap.stat.exists -+ fail_msg: "On proot, {{ dest_path }} must be downloaded by dash-node before the maps role runs (K2GO-394)." -+ quiet: yes -+ when: is_proot and not (expand_archive | default(false)) - - rescue: - # We output summaries to a log file for the user's benefit,