Fatal "Turn execution failed" under concurrent processes: shared session DB hits "database is locked" (SQLITE_BUSY) with no busy_timeout or retry

[mwotton]

Summary

Under concurrent zcode processes (a fleet of headless agents plus interactive TUI sessions on one host), turns die fatally with Error: Turn execution failed (traceId: …). The CLI log always shows the same cause under that wrapper — the shared session database reporting SQLITE_BUSY:

{"timestamp":"2026-09-18T06:43:14.165Z","level":"error","event":"turn.failed","module":"core.runtime",
 "error":{"message":"Turn execution failed","code":"UNKNOWN_ERROR","type":"unknown_error",
          "context":{"errorPayloadRole":"wrapper"},
          "cause":{"name":"Error","message":"database is locked","code":"ERR_SQLITE_ERROR"}}}

The installed runtime bundle (vendor/zcode.cjs in zcode-app-cli 3.11.2-25) sets no busy_timeout pragma at all (grepped the bundle: zero hits), so session-DB writes run on the driver default with no backoff and no retry. WAL mode lets readers proceed, but writers still serialize; when a write at a turn boundary (turnPhase: "processing_input") waits past the timeout, the whole turn fails — and a headless agent exits mid-task with nothing written.

Environment

• zcode-app-cli 3.11.2-25, zcode-runtime 0.16.5, Linux x64 (npm)
• Shared db: ~/.zcode/cli/db/db.sqlite (~200 MB, -wal/-shm sidecars present)

Reproduction / observed scale

One host running ~10–15 concurrent zcode processes (headless CI-style agents + interactive TUIs), all writing to the single shared session DB:

• 28 distinct sessions hit by a fatal turn.failed with cause database is locked in 3 days, interactive and headless alike
• ~3× as many lock errors are absorbed at lower layers (usage.tool.write.failed ×69, usage.turn.write.failed ×4, usage.model.write.failed ×2, checkpoint.persist.failed, session_input.admit_failed) — the contention is constant; only writes landing at the turn boundary are fatal
• Failures come in bursts (e.g. 7–8 fatal turns across the 02:00–05:59 UTC hours) — one lock pileup kills several sessions at once
• Wall-time-to-death varies 4–18 min because the agent dies at whichever turn first tries to persist while another writer holds the lock

You can check any install with: grep -c 'database is locked' ~/.zcode/cli/log/zcode-*.jsonl

Suggested directions

1. Set PRAGMA busy_timeout (e.g. 5000–10000 ms) on the session-DB connections in the vendored runtime.
2. Treat SQLITE_BUSY at the persistence layer as retryable with short backoff instead of surfacing it as a fatal wrapped UNKNOWN_ERROR that kills the turn.
3. Longer term: shard the session DB per project/session so fleet-style concurrency doesn't serialize every process on one file.

Happy to share more log evidence or test a build. (Related but distinct: #160 / #161 cover the provider-registry migration fallout, not this.)

[kingsword09]

Published zcode-app-cli@3.14.0-27 (release notes), based on ZCode Desktop 3.14.0 / runtime 0.16.9, with the corrected #163 mitigation and the 3.14 compatibility work in #166.

npm install -g zcode-app-cli@3.14.0-27
zcode --version

Expected versions:

zcode-app-cli 3.14.0-27
zcode-runtime 0.16.9

Restart existing CLI/headless processes so they load the updated runtime, then retest with the original 10–15-process workload.

Each session-store connection now retains a 10-second write-lock timeout after initialization, including synchronous open, asynchronous startup, and reopen. Applying the timeout only in the constructor was insufficient because the migration code restored its own setting afterward. The existing startup migration backoff and cleanup remain in place.

This gives short lock bursts more time to clear. It does not retry entire agent turns, split the database, or guarantee that sustained contention cannot still fail. The CLI patch adds no schema migration or bulk rewrite of existing sessions; normal upstream migrations and session persistence still apply.

The actual Node runtime tests verify a native session write surviving a 6.5-second lock and committing exactly once, bounded timeout failure followed by recovery, four-process concurrent startup, and rollback/reopen after a writer is killed. The CI matrix passed on Linux with Node 22.19.0, 24, and 26, plus macOS with Node 24. Public npm installation and the published commit/tag were also verified.

I am keeping this issue open for validation on your original workload. If it still fails, please include the installed versions, Node version, approximate process count, and the timestamp plus redacted error/cause chain around turn.failed. The upstream report can use this summary:

Multiple independent ZCode runtime processes share one SQLite WAL database. In the original report, 10–15 concurrent processes produced ERR_SQLITE_ERROR: database is locked, which became fatal Turn execution failed errors and terminated headless tasks. The CLI now retains a 10-second post-initialization busy timeout, with real Node contention tests. Please investigate long write transactions and bounded recovery at persistence boundaries that can safely roll back and retry, without replaying model requests or completed tools. Results under the original workload on runtime 0.16.9 are still being collected.

Storage test details: SQLite contention regression.