K2GO-394 feat(dashboard): base-map downloads via the durable job engine

static/dashboard/CHANGELOG.md

@@ -4,6 +4,7 @@ One line per version, newest first. Every REST-facing change bumps the version i
 (the app surfaces it via `/system/dashboard/update-check` and the "Update available" pill), so this
 file is the human record of what each bump enables. Keep entries short: `version - change (TICKET)`.
 
+- **1.3.2** - Base-map downloads via the durable job engine (K2GO-394). New `basemaps` job type + runner (`sockets/maps-base.exec.ts`), reached through the generic surface (`POST /api/basemaps/download {items:[<pmtiles url>]}`, `GET /api/basemaps/jobs/:id`, `POST /api/basemaps/jobs/:id/{cancel,pause,resume,retry}`). It downloads the selected global map pmtiles (vector / satellite / terrain) with aria2 straight into `/library/www/maps`, reusing the EXACT kiwix mechanism -- the canonical aria2 flag set plus the `withRetry` OUTER reconnect loop -- so a full Wi-Fi drop (aria2 exits on DNS, code 19) recovers by re-running aria2 which resumes via `--continue`, surfaced as "Reconnecting n/5". This replaces the aria2c the maps runrole ran IN-PROOT, which could not recover a mobile-radio drop (it wedged with no exit): the app runs this job (server up) and the runrole then only post-processes, its download tasks SKIPPING via `creates:` (see the maps role's `is_proot` delegate patch). Two things learned the hard way and encoded here: it passes the DIRECT pmtiles URL, NOT a `.meta4` metalink (metalink downloads were the exact case aria2 could not recover), and it does NOT diverge the flags (an aggressive `--max-tries=1 --timeout=10` cut wedged worse; the kiwix values are load-bearing). Kept SEPARATE from the FQR `maps` type (tile-extract). Device-verified: cut Wi-Fi at 27% of a pmtiles -> "Reconnecting 5/5", partial kept -> restore -> resumed to done, file complete at dest_path. (K2GO-394)
 - **1.3.1** - Live firehose signal for the app-side backstop (K2GO-386, ADR-386 §6). New read-only `GET /system/disk-guard/firehose` returns `{ recurring, maxStreak, paths, lastTruncatedAtMs, now }`. The in-box guard (1.3.0) truncates a runaway `.log` every tick, so the disk may never go low -- but a recurring firehose means an off-proot orphan the box CANNOT stop; only an app-side reap can. This endpoint exposes the guard's LIVE in-memory streak state (never a parsed log line, so a restart-resolved firehose reports clean) as the app's SECOND reap trigger. `recurring` is `maxStreak >= 2` (a single `.log` refilled past the cap on at least two consecutive ticks); `lastTruncatedAtMs` (wall-clock, 0 if never) lets the app judge freshness. It is an ALERT only: the app re-probes live log growth before it reaps (confirm before acting). Localhost-only. (K2GO-386)
 - **1.3.0** - Proot log rotation, dash-node-triggered (K2GO-386, ADR-386). proot has no systemd/cron, so `/etc/cron.daily/logrotate` never runs — logrotate was installed but never triggered, and a service log (php-fpm, dash-node) could grow until the device hit ENOSPC. dash-node now runs, every 10 min (no work at boot; `timer.unref`), a firehose guard THEN `logrotate /etc/logrotate.conf`: the guard truncates any log past ~1 GiB in place first (a runaway ~GB/min that logrotate would otherwise copy — doubling disk + pegging CPU on a weak phone), so L2 never meets a firehose; a recurring firehose is flagged for the future app-side reap (ADR-386 §6). The K2Go-owned config `/etc/logrotate.d/k2go` (copytruncate + `size 100M`, proot-correct — no reopen signal — overriding the RPi-oriented nginx/php-fpm snippets and adding calibre-web + dash-node; kiwix has no log, kolibri self-rotates) is installed at deploy by `tools/setup-proot-logging.sh` (rootfs build + rebuild/dev-push), not at boot. Not a REST-surface change; the version bump is the delivery mechanism for the new dash-node behavior (no ansible role yet). (K2GO-386)
 - **1.2.12** - Dashboard-update card back end (ADFA-5339, Phase 1 server half). New read-only `GET /system/dashboard/rebuild/log`: the last ~200 lines of `/var/log/dash-rebuild.log`, for the card's expandable "Details" (no file yet = empty log, not an error). `POST /system/dashboard/rebuild` now accepts `{ site: true }`: it refreshes the served landing page in the SAME run via `site-updater.sh`, from the same clone the rebuild's git fetch+reset refreshes, in finalize AFTER the core swap verifies live — so the site matches the new source. The site is a separate, versionless artifact: it never touches the reported version, and a site failure is logged, never a rollback of the (already-verified) core update. Both localhost-only. (ADFA-5339)

static/dashboard/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dashboard-console",
-  "version": "1.3.1",
+  "version": "1.3.2",
   "description": "",
   "main": "index.js",
   "scripts": {

static/dashboard/routes.ts

@@ -38,7 +38,7 @@ const ZIMS_DIR = '/library/zims/content/';
 const KIWIX_INDEXER = '/usr/bin/iiab-make-kiwix-lib';
 const ZIM_NAME_RE = /^[A-Za-z0-9._-]{1,150}\.zim$/;
 
-const VALID_TYPES: JobType[] = ['kiwix', 'maps', 'books', 'kolibri'];
+const VALID_TYPES: JobType[] = ['kiwix', 'maps', 'books', 'kolibri', 'basemaps'];
 function isType(t: string): t is JobType {
     return (VALID_TYPES as string[]).includes(t);
 }

static/dashboard/server.ts

@@ -7,6 +7,7 @@ import helmet from 'helmet';
 import { jobs } from './sockets/jobs';
 import './sockets/kiwix.exec';
 import './sockets/maps.exec';
+import './sockets/maps-base.exec';
 import './sockets/books.exec';
 import './sockets/kolibri.exec';
 import { apiRouter } from './routes';

static/dashboard/sockets/jobs.ts

@@ -13,7 +13,7 @@ import fs from 'fs';
 import path from 'path';
 import { RollingLog, LogSlice } from './rolling-log';
 
-export type JobType = 'kiwix' | 'maps' | 'books' | 'kolibri';
+export type JobType = 'kiwix' | 'maps' | 'books' | 'kolibri' | 'basemaps';
 export type JobPhase =
     | 'queued' | 'downloading' | 'indexing' | 'processing'
     // ADFA-4894 (control surface): 'paused' is a stopped-but-resumable state — like 'canceled' it

static/dashboard/sockets/maps-base.exec.ts

@@ -0,0 +1,169 @@
+// sockets/maps-base.exec.ts -- K2GO-394
+//
+// Base-map runner for the durable job engine: download the selected GLOBAL map pmtiles
+// (vector / satellite / terrain) with aria2, straight into the maps serve dir. This is a
+// drop-in replacement for the aria2c the maps runrole used to run in-proot, which wedged
+// at CN:0 on a full network drop and never recovered (K2GO-394): once the files are here,
+// the runrole's `creates: dest_path` and the .meta4 size-probe both SKIP the download, so
+// the role only post-processes (symlinks, maps-config.js).
+//
+// Kept SEPARATE from the FQR `maps` runner (maps.exec.ts / tile-extract.py) -- same engine,
+// different concern. Reuses the PROVEN kiwix mechanism (see kiwix.exec.ts): the canonical
+// aria2 flag set plus the withRetry OUTER loop that re-runs aria2 (which resumes via
+// --continue) on a full-interface-loss exit (19 = DNS cannot resolve), surfaced on the poll
+// as "Reconnecting n/5". No --lowest-speed-limit, on purpose (it turns a slow mobile link
+// into a hard abort).
+import { jobs, RunnerContext, CanceledError, PausedError, classifyStop } from './jobs';
+import { withRetry } from './net-retry';
+import fs from 'fs';
+import path from 'path';
+
+// = maps_serve_path = dest_base_path in roles/maps; the runrole moves each pmtiles here.
+const MAPS_DIR = '/library/www/maps';
+// The app passes one catalog_file_url per selected layer. Only plain https URLs (no shell is
+// used -- spawn takes an argv -- so this guards SSRF/log-shape, not shell injection).
+const SAFE_URL = /^https:\/\/[^\s'"`$<>|;()]+$/;
+
+// ADFA-4832 CANONICAL aria2 flag set -- an EXACT copy of kiwix.exec.ts (only -d differs). The kiwix
+// runner recovers from a Wi-Fi drop (aria2 --max-tries=5 absorbs in-flight blips; a FULL interface
+// loss makes aria2 exit and the withRetry OUTER loop re-runs it, resuming via --continue), so this
+// mirrors it exactly rather than inventing a retry model. Notably NO --lowest-speed-limit (it turns
+// a slow mobile link into a hard abort). If you change a flag here, change kiwix.exec.ts and
+// controller/app/.../Aria2Manager.java too.
+const ARIA2_ARGS: string[] = [
+    '-d', MAPS_DIR,
+    '--continue=true',
+    '--allow-overwrite=true',
+    '--auto-file-renaming=false',
+    '--max-connection-per-server=4',
+    '--split=16',
+    '--follow-metalink=mem',
+    '--check-integrity=true',
+    '--console-log-level=warn',
+    '--summary-interval=1',
+    '--download-result=hide',
+    '--async-dns=false',
+    '--max-tries=5',
+    '--retry-wait=5',
+    '--timeout=60',
+    '--connect-timeout=15',
+    '-Z',
+    '-j', '5',
+];
+
+// The transient aria2 exit codes the OUTER loop re-runs (aria2 resumes via --continue): 1 unknown,
+// 2 timeout, 6 network, 7 unfinished, 19 DNS, 29 HTTP 503. Terminal (not retried): 3/4 not found,
+// 9 no space, 13 file exists.
+const ARIA2_TRANSIENT_EXITS = new Set<number>([1, 2, 6, 7, 19, 29]);
+// 5 VISIBLE reconnect waits (3, 6, 9, 18, 36 s ~= 72 s total), surfaced as "Reconnecting n/5".
+const RETRY_DELAYS = [3_000, 6_000, 9_000, 18_000, 36_000];
+
+/** Convert an aria2 rate token ("34MiB", "512KiB", "1.2MB") to bytes/sec. */
+function parseRate(token: string): number {
+    const m = /^([\d.]+)\s*([KMGT]?i?B)?/i.exec(token);
+    if (!m) return 0;
+    const val = parseFloat(m[1]);
+    const unit = (m[2] || 'B').toUpperCase();
+    const mult: Record<string, number> = {
+        B: 1, KIB: 1024, MIB: 1024 ** 2, GIB: 1024 ** 3, TIB: 1024 ** 4,
+        KB: 1000, MB: 1e6, GB: 1e9, TB: 1e12,
+    };
+    return Math.round(val * (mult[unit] ?? 1));
+}
+
+/** Remove the download artifacts aria2 leaves next to a completed pmtiles. */
+function cleanupMetadata(): void {
+    try {
+        for (const f of fs.readdirSync(MAPS_DIR)) {
+            if (/\.(meta4|aria2)$/.test(f)) fs.unlinkSync(path.join(MAPS_DIR, f));
+        }
+    } catch { /* non-fatal */ }
+}
+
+// Clean-on-cancel (mirrors kiwix.exec.ts): a canceled download must not leave a partial behind. aria2
+// preallocates the FULL file size, so a canceled partial looks complete to the maps role's
+// `creates: dest_path` and to the is_proot presence assert -- the role would then serve a truncated
+// pmtiles. Prune each requested file's partial and its .aria2/.meta4 siblings. A file with no .aria2
+// is already complete, so leave it. An ERROR keeps the partial on purpose (resume via --continue).
+function cleanupPartials(files: string[]): void {
+    for (const f of files) {
+        try {
+            const control = path.join(MAPS_DIR, `${f}.aria2`);
+            if (!fs.existsSync(control)) continue;   // no control file -> complete, do not delete it
+            fs.unlinkSync(control);
+            for (const sibling of [f, `${f}.meta4`]) {
+                const p = path.join(MAPS_DIR, sibling);
+                if (fs.existsSync(p)) fs.unlinkSync(p);
+            }
+        } catch { /* best-effort */ }
+    }
+}
+
+const mapsBaseRunner: (ctx: RunnerContext) => Promise<void> = async (ctx) => {
+    const urls = ctx.ids.map(String).filter((u) => u.length > 0);
+    if (urls.length === 0) throw new Error('no base-map URLs requested');
+    for (const u of urls) if (!SAFE_URL.test(u)) throw new Error(`unsafe base-map URL: ${u}`);
+    // aria2 saves each URL under its basename in MAPS_DIR (no --out); the same basename the maps role
+    // expects at dest_path. Used to prune the right partial on cancel.
+    const files = urls.map((u) => path.basename(u));
+    fs.mkdirSync(MAPS_DIR, { recursive: true });
+
+    ctx.throwIfCanceled();
+    ctx.update({ phase: 'downloading', speed: 0, detail: files.join(', ') });
+
+    // Pass the DIRECT pmtiles URL (not <url>.meta4), the same way the kiwix runner passes the .zim
+    // URL directly. aria2 downloads it into MAPS_DIR under its own basename -- exactly the runrole's
+    // dest_path, so the role's `creates:` then skips the download. (An explicit .meta4 metalink is
+    // what the in-proot runrole used, and metalink downloads are what wedged aria2 on a network drop
+    // -- K2GO-394; --follow-metalink=mem still honors a metalink the mirror serves on its own.)
+    try {
+        await withRetry(() => new Promise<void>((resolve, reject) => {
+            const dl = ctx.spawn('/usr/bin/aria2c', [...ARIA2_ARGS, ...urls]);
+            const onData = (buf: Buffer) => {
+                const text = buf.toString();
+                const re = /\((\d+)%\).*?DL:([^\s]+)/g;
+                let m: RegExpExecArray | null;
+                let lastPct = -1;
+                let lastRate = '';
+                while ((m = re.exec(text)) !== null) { lastPct = parseInt(m[1], 10); lastRate = m[2]; }
+                if (lastPct >= 0) { ctx.reportRetry(0, 0); ctx.update({ phase: 'downloading', percent: lastPct, speed: parseRate(lastRate) }); }
+            };
+            dl.stdout?.on('data', onData);
+            dl.stderr?.on('data', onData);
+            dl.on('error', reject);
+            dl.on('exit', (code, signal) => {
+                if (signal === 'SIGKILL' || ctx.isCanceled()) return reject(new CanceledError());
+                if (code === 0) return resolve();
+                // Carry the exit code so the outer loop tells a transient network failure (retry,
+                // resuming via --continue) from a terminal one (not found / no space).
+                const err = new Error(`aria2 exited with code ${code}`);
+                (err as { code?: number }).code = code ?? -1;
+                reject(err);
+            });
+        }), {
+            delaysMs: RETRY_DELAYS,
+            tries: RETRY_DELAYS.length + 1,
+            signal: ctx.signal,
+            isCanceled: ctx.isCanceled,
+            isTransient: (e) => ARIA2_TRANSIENT_EXITS.has((e as { code?: number }).code ?? -1),
+            onRetry: ({ attempt, err }) => {
+                ctx.reportRetry(attempt, RETRY_DELAYS.length);
+                ctx.log(`[basemaps] reconnect ${attempt}/${RETRY_DELAYS.length} after: ${err instanceof Error ? err.message : String(err)}`);
+            },
+        });
+    } catch (e) {
+        // pause KEEPS the partial (+ .aria2) so resume continues via --continue; a real error also
+        // keeps it so a retry/reconcile resumes rather than restarting from zero.
+        const stop = classifyStop(ctx);
+        if (stop === 'paused') throw new PausedError();
+        if (stop === 'canceled') { cleanupPartials(files); throw new CanceledError(); }
+        throw e;
+    }
+
+    cleanupMetadata();
+    ctx.update({ phase: 'done', percent: 100, speed: 0 });
+};
+
+jobs.registerRunner('basemaps', mapsBaseRunner);
+
+export { mapsBaseRunner };

tools/upstream-patches/patches/0003-maps-download-is-proot-delegate.patch

@@ -0,0 +1,36 @@
+Upstream-PR:     not yet submitted
+Upstream-Status: open
+Applies-to:      roles/maps/tasks/download_large_file.yml
+Summary:         On proot, delegate the base-map pmtiles download to dash-node (the dashboard's durable job engine) instead of running a blocking aria2c in-proot. The in-proot aria2 could not recover from a full mobile-radio drop -- it wedged with no exit (K2GO-394). dash-node downloads the pmtiles with its own resumable, reconnecting aria2 (--continue plus an outer reconnect loop -- the proven kiwix mechanism) into the maps serve dir BEFORE this role runs. So the original download task is gated `when: not is_proot` (unchanged off proot), and a new `when: is_proot` step asserts the file is already at dest_path (a missing file means the host skipped the dash-node step); the role then only post-processes. Carried for our build; WIP for upstream (K2GO-394).
+
+diff --git a/roles/maps/tasks/download_large_file.yml b/roles/maps/tasks/download_large_file.yml
+index f8aa705..78a55cc 100644
+--- a/roles/maps/tasks/download_large_file.yml
++++ b/roles/maps/tasks/download_large_file.yml
+@@ -140,6 +140,26 @@
+       args:
+         executable: /bin/bash
+         creates: "{{ dest_path }}"
++      when: not is_proot
++
++    # proot / Android (K2GO-394): dash-node -- the dashboard's durable job engine -- downloads the
++    # base-map pmtiles with its own resumable, reconnecting aria2 (--continue plus an outer reconnect
++    # loop) BEFORE this role runs. On proot this role only post-processes, so it must NOT download
++    # in-proot, where a blocking aria2 cannot recover from a mobile-radio drop (it wedges with no
++    # exit). The file is expected at dest_path already; fail early and clearly if the host skipped the
++    # dash-node step, rather than leaving a broken symlink for maps-update.py to trip on.
++    - name: "Base map {{ file_name }} must be present on proot (downloaded by dash-node)"
++      stat:
++        path: "{{ dest_path }}"
++      register: proot_basemap
++      when: is_proot
++
++    - name: "Fail if dash-node did not place {{ file_name }} on proot"
++      assert:
++        that: proot_basemap.stat.exists
++        fail_msg: "On proot, {{ dest_path }} must be downloaded by dash-node before the maps role runs (K2GO-394)."
++        quiet: yes
++      when: is_proot
+
+   rescue:
+     # We output summaries to a log file for the user's benefit,