Docs: geo-distribution architecture — one URL, region-local speed, single source of truth

[lopugit]

Research + architecture notes for geo-distributing Thingtime: one thingtime.com URL, region-local server + database speed for AU/US/EU users, one logical single-source-of-truth dataset. Docs only — nothing is built; this is a reading/decision document.

What's inside (docs/architecture/geo-distribution.md)

• Where we are post PRs Perf: pin functions to syd1 + edge-cache anonymous feed/search #157/Rate limiter: audible fail-open + ensureIndexes failure cooldown #159/Perf: move the ensureIndexes battery off the request path (cold-start fix) #161 (measured 2026-08-08: warm feed ~430ms from AU, fn→DB 2ms, anon edge cache ~65ms) and what a US user pays today (~one ocean crossing per request).
• The constraint set from FUNDAMENTALS/DECISIONS: single source of truth, everything-is-a-thing, load-bearing unique indexes, withMongoTransaction, test==live parity.
• The insight that picks the architecture: a social feed is global-content-read-heavy — every region needs everyone's posts. Replication (each region holds a full copy) matches that; geo-sharding (splitting by user home region) optimizes the wrong axis first.
• Three options:
  • A — read-local/write-global ⭐: one Atlas replica set, 3 electable nodes in Sydney (write-majority acks stay local — the one config mistake to avoid is spreading voters across oceans), read-only nodes in us-east/EU, readPreference=nearest + Atlas node tags. US reads drop ~200ms → ~5ms; search goes region-local too (coupled Atlas Search runs on read-only nodes). ~$112/mo (M10 + 1 US node, from the live Atlas calculator).
  • B — + write forwarding: Build Output API routes support methods matching and per-function regions, so ONE deployment can send GETs to multi-region functions and writes to a syd1-pinned copy of the same Nitro bundle. Remote writes become a single hop. No second project, no middleware.
  • C — zone-sharded Global Cluster (literal sharding): region-local writes, at the cost of schema surgery (ISO-location shard key on every doc, no resharding later), unique-index redesign (official proxy/reservation-collection pattern — conveniently the same identity-vs-content split mongodb/endpoint.ts already draws), scatter-gather feeds/search, and an ~$980/mo M30 floor. The right eventual destination, not the next step.
• A six-phase staged plan — each phase shippable + reversible, starting with the Phase-3 round-trip diets from the perf audit as prerequisite, with AU+US probe verification at every step (the methodology that caught PR Perf: move the ensureIndexes battery off the request path (cold-start fix) #161's state-C regression).
• Platform facts researched 2026-08-08 from official Vercel + MongoDB docs, the live Atlas pricing calculator, and measured Sydney↔us-east RTTs — with an explicit "could not confirm" list (e.g. Vercel's own docs disagree whether Pro allows 3 or 5 function regions).
• Open questions for the owner: budget timing, staleness tolerance, regional rate-limit semantics, which region second.

Notable existing-plumbing findings

• mongodb/endpoint.ts's home-vs-data-plane split (identity pinned home, data plane resolved per request) is exactly the seam a regional data plane generalizes.
• Fluid Compute (not yet enabled) is a free prerequisite win on any plan: in-function concurrency shrinks Mongo pool pressure and scale-to-one keeps ≥1 warm instance up to 14 days.
• Vercel's CDN cache is per-region — the anon anon=1 cache warms per region and pairs with 2025/26 cache-tags + 300ms global purge for precise invalidation later.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
thingtime	Ready	Preview	Aug 27, 2026 1:11pm
thingtime (develop)	Ready	Preview	Aug 27, 2026 1:11pm

[github-actions]

🦉 Lopu — claims verified against the code; one correction that strengthens §5.3

For a research document the review that matters is whether the factual base holds, so I re-checked every checkable claim rather than commenting on structure. Everything verified:

Claim	✓
Functions pinned to syd1	remix/vercel.json
Boot-time warmup plugin	remix/server/plugins/mongo-warmup.ts
getClientCachedFor(uri, isHome), maxPoolSize: 10	mongodb/collections.ts
withMongoTransaction — readConcern: snapshot, writeConcern: majority, readPreference: primary	collections.ts:88-95, exactly as stated
s-maxage=60, stale-while-revalidate=300	identical ANON_CACHE_CONTROL in feed/_feed.tsx:21 and search/_search.tsx:15
All ~20 relative links resolve	including the URL-encoded server/routes/api/%5B...%5D.ts

§5.3's MongoDB reasoning is accurate and unusually well-scoped — unique indexes on a sharded collection must be shard-key-prefixed; Global Clusters need M30+ and fix the shard key at creation; a query without the shard key fans out scatter-gather. Mapping the "unsharded identity/reservation collection on the primary shard" mitigation onto the endpoint split the repo already has is the sharpest observation in the document. §4's "a social feed is global-content-read-heavy" is the right decision criterion and genuinely does select A over C.

The one correction

Constraint §2.2 described things_v2 as distinguished by kind. The live discriminator is thingtime, a multikey array of type tags (['post'], ['post','comment'], …); kind is the retired v1 scalar, read only through an era-compatibility $or in things/things.ts.

Normally that'd be pedantry — but §5.3 builds its sharding argument on "this shape", and a multikey field cannot be part of a shard key in MongoDB. So thingtime — the obvious candidate a reader would reach for when asked to shard the everything-is-a-thing collection — is off the table, which strengthens your cost #1 (a new homeRegion field is genuinely required, not something derivable from what's there). I've pushed that correction with the consequence spelled out, so §5.3's first cost is now self-justifying.

Only PR in this batch with a fully green board — 18 check runs, nothing red.

[github-actions]

✅ Develop S3 preview ready

• Commit: cad8133f
• Environment: Vercel Custom Environment develop
• Preview: https://pr-185.previews.dev.thingtime.com
• Immutable deployment: https://thingtime-fr3gmglao-lopugits-projects.vercel.app
• Vercel status: open deployment

The alias passed the develop bucket CORS preflight and a final live PR/SHA fence.

Generic Vercel Preview deployments use the shared development runtime; this controller adds the stable exact-SHA alias and marker-scoped cleanup.

[github-actions]

🦉 Lopu — the reasoning holds; the branch is 962 commits old and it showed

This branch was cut 2026-08-08 and develop is 962 commits ahead. The change is one new file so the merge is trivial — but a research document is only worth what its facts are worth, so I checked every claim twice: once against the tree you measured, once against develop as it stands today.

What survived unchanged

Every code seam the argument rests on is still there and still behaves as described:

Claim	Verified on develop
Home vs data-plane split is the seam a regional data plane generalizes	✅ endpoint.ts:112 holds the AsyncLocalStorage selection; collections.ts:17 / :153 hold getClientCachedFor(uri, isHome) and getHomeThingtimeDb
withMongoTransaction pins snapshot / majority / primary	✅ collections.ts:83-95 — so §2.4's constraint is real, not hypothetical
Serverless-tuned pool: maxPoolSize: 10, 5s fail-fast, appName	✅ collections.ts:40-44
Anon edge cache contract	✅ s-maxage=60, stale-while-revalidate=300 in both _feed.tsx:21 and _search.tsx:15
Limiter: two sequential writes + a third read on reject	✅ enforce.ts — updateOne(…, {upsert:true}) :59, findOneAndUpdate :65, findOne({key}) :76. Independently matches performance/TODO.md's own entry on the same code.

The reasoning I'd most want kept is §4: a social feed is global-content-read-heavy, so replication (every region holds everything) matches the workload while geo-sharding optimizes the wrong axis first. That's the load-bearing insight and it's right. So is §5.1's warning that all three electable nodes must stay in Sydney — spreading voters across oceans puts ~200ms inside every write's majority ack, which is precisely the config mistake that would make the migration a regression. And §5.3's honesty about Option C (shard key set once, never resharded; unique indexes must be shard-key-prefixed) is what stops someone buying a $980/mo cluster to fix a read-latency problem.

What had drifted — corrected inline

1. remix/vercel.json no longer exists. §1's very first citation linked to it as the place syd1 is pinned; the pin now lives in the root vercel.json. Left as-is, the doc's opening fact would 404 on merge.
2. Fluid Compute is already enabled. The doc treats it as outstanding in three places (§5.1, §7, Phase 3's prereqs) — but the root vercel.json carries "fluid": true, enabled 2026-08-08, the same day you drafted this. Phase 3 now has one fewer prerequisite, which is a small improvement to the plan rather than a problem with it.
3. Phase 0 pointed at nothing. "The Phase-3 items from the perf audit (react 17-19→~6 RTs…)" — the audit has since been restructured, and performance/README.md records that findings assuming the old cross-region layout were restated against the current syd1 pinning. Neither a "Phase 3" nor the 17-19 figure survives anywhere in performance/TODO.md or the perf: Thingtime performance audit — findings, notes and fixes #299 notes. Since Phase 0 is the stated prerequisite for the whole plan, a dangling pointer there is the costliest staleness in the doc. It now links performance/TODO.md and names its real sections. I deliberately did not invent a replacement number.
4. Added a dated re-verification note under the header: the code seams still hold as of today, here's what moved, and §7's vendor pricing/plan-gating is still 2026-08-08 vintage and must be re-checked before anyone spends money.

I left the measurements, the three options, the cost table and the open questions alone — they're clearly scoped as "researched 2026-08-08", §7 already carries a ⚠️ re-verify warning, and re-pricing Atlas is your call at purchase time, not a reviewer's edit.

One thing on §8

The four open questions are the right ones, and the first ("park this until there's a US cohort?") is answerable cheaply now — Vercel Analytics already knows where visitors are. Worth pulling that number before Phase 1, since it also settles question four (which region second).

All checks green at this head; Lopu builds and repairs the all branch is pending, not failing — queued behind the shared fleet group. CodeQL disposition file left as [].

_{Housekeeping note for whoever maintains the review harness: the trusted CodeQL snapshot for this PR is empty, but the alert-instances API shows alert #7 open on refs/pull/185/head @ e5fda6da — this exact head. So the snapshot understates the real set, probably taken before this ref's analysis was available. I stayed inside my mandate and dispositioned nothing here, but the snapshot step looks worth hardening so a docs PR doesn't silently look cleaner than it is.}

— 🦉 Lopu, Thingtime's PR manager

[github-actions]

🤖 Lopu — your re-verification checks out; the 6 extra CodeQL alerts are not yours

This is a one-file documentation diff, so I spot-checked what the 2026-08-26 re-verification note claims against develop at c9bbd46a:

Claim	Verified
The Vercel region pin moved to the root vercel.json	✅ root vercel.json carries "regions": ["syd1"]
remix/vercel.json is gone	✅ absent from develop
Fluid Compute is enabled (one fewer Phase 3 prerequisite)	✅ "fluid": true in root vercel.json
Anonymous feed/search carry s-maxage=60, stale-while-revalidate=300	✅ ANON_CACHE_CONTROL in both things/feed/_feed.tsx:21 and things/search/_search.tsx:15

All accurate. §7's vendor pricing being stamped "as of 2026-08-08, re-check before any purchase" is the right way to age a research figure instead of letting it rot silently. The core argument holds too — colocating compute with data means a remote user pays the ocean crossing once per request, not once per database round trip, which is exactly why Option A is the right next step and why §5.4's rejection of per-region independent databases belongs in the record.

CodeQL: 22 alerts, but only 16 are the repository-wide baseline

The other 6 are worth explaining because a docs-only diff obviously did not introduce them. I read the cited code on both sides:

Alerts	Location	On this head	On develop
#30, #31	Commander/CommanderV2.tsx:152	the escapedCommandValue block still exists — replace(/"/g,'\\"').replace(/'/g,"\\'"), which never escapes backslashes	the whole block is gone; line 152 is now commandIsAction
#35–#37	Commander/CommanderV1Deprecated.tsx:184	same pattern, same era	same removal
#75	.github/workflows/resolve-pr-conflicts.yml:444 (pull_request_target checkout)	the full ~5k-line implementation is still on the branch	the file is 118 lines — the thin single-entry listener; the implementation now lives on github-actions

All six are stale-base artefacts: this branch was cut 2026-08-08 and still carries pre-fix copies of files develop has since fixed or converted. They disappear on merge, and no change here can retire them.

I left the dispositions empty deliberately. None is a false positive (the sanitization really is incomplete in the code as it stands on this head, and the privileged checkout really is there) and none is test-only — dismissing them would be greening a check with a false statement. Nor did I "fix" them: develop has already deleted the Commander block, so re-implementing an escape here would only conflict with that deletion, and resolve-pr-conflicts.yml is protected control-plane code that must never be repaired on an unrelated feature branch. The correct resolution is the merge.

Two small things

1. A broken relative link on this head, self-resolving. §1 links the region pin as [vercel.json](../../vercel.json), which from docs/architecture/ resolves to <repo-root>/vercel.json — correct on develop, but this branch predates the move and still has remix/vercel.json with no root vercel.json. So the link is dead when read on the PR branch and becomes correct on merge. Same root cause as the alerts; nothing to change, since the target it names is the right one.
2. Nothing in the repository links to this document — not README.md, not FUNDAMENTALS.md, not TODO/claude-todo/README.md. docs/ already exists on develop with several subdirectories so the location is fine, but a 482-line architecture proposal that no index points at is easy to lose. Compare docs(todo): 20 — reconcile the things index budget under MongoDB's 64 cap #405 in this batch, which added a roadmap index row for its document in the same change. I did not add the cross-reference myself: this is explicitly "a proposal for review — nothing here is built yet" with §8 open, and filing it into the roadmap index would assert a status you have not claimed. Picking its home is your call.

(The single red check is a cancellation, not a failure: Lopu builds and repairs the all branch job 98047881134 is conclusion: cancelled with steps: [] — queued in the fleet group and drained without starting. web-ci on this head is green.)

[github-actions]

🤖 Lopu — the 4h58m red mark has a root cause now, and it was fixed upstream 6 hours after this run died

Re-reviewing this head. Nothing to add on the document itself beyond my 09:15 pass — I re-checked the links against the merge commit (32188c7) rather than the branch this time, with percent-decoding, and all 11 relative links resolve in the merged state, including the ../../vercel.json one I flagged as dead-on-branch. It really is self-resolving.

What I can now add is the why behind the one red check, which I previously only characterised as "queued in the fleet group and drained without starting".

Lopu builds and repairs the all branch — 4h58m19s, and it never ran a step

Job 98047881134: conclusion: cancelled, empty step list. The workflow declares timeout-minutes: 60, so 4h58m cannot be execution time — the job was queued at 03:13:29Z and evicted at 08:11:48Z without ever starting. It sits in

concurrency:
  group: lopu-agent-fleet-${{ github.repository }}
  queue: max
  cancel-in-progress: false

— one deliberately serialized lane shared by the all-branch rebuild, PR reviews, promotions, conflict resolution, rebases and stack cascades. With 73 open PRs that lane backs up, and pending entrants get drained.

It is not about this PR. The job is a repository-wide union build of develop + main + every open PR; a single new docs file cannot influence it. The same cancellation hit promote/…, codex/… and the scheduled main runs across the same window.

It is already fixed, and the fix is already live

PR #419 (codex/lopu-coalesce-pending-runs) merged into github-actions at 2026-08-26T14:31:22Z — about six hours after this run was cancelled. And because main's listener pins the controller (uses: lopugit/thingtime/.github/workflows/resolve-pr-conflicts.yml@github-actions), that landed for new events immediately, with no promotion needed.

So: stale, externally caused, already addressed. A re-run clears it; no commit to this branch can. I deliberately did not write a controller fix into the trusted checkout for this — the owner's own fix for exactly this landed four minutes before this review session was dispatched, and a competing edit would collide with #419 rather than help.

CodeQL — 22 alerts, and I can now show the 6 extras are stale-base

I verified the stale-base claim directly against the base commit b7e2906 this time rather than by inspection:

	On this head	On develop @ b7e2906
#30, #31	CommanderV2.tsx:152 is the escapedCommandValue block	escapedCommandValue occurs 0 times; line 152 is commandIsAction
#35–#37	CommanderV1Deprecated.tsx carries the same pattern	the file is absent entirely
#75	resolve-pr-conflicts.yml is the full implementation	the file is 118 lines — the thin listener

develop deleted that Commander block rather than fixing it, so re-implementing an escape here would only conflict with the deletion. All six retire on merge. Left open, not dismissed — on this exact head the cited code really is there.

Changed from my last pass: I have now dispositioned the 11 repository-wide baseline alerts (#7, #8, #14, #26–#29, #77–#80) with per-alert evidence — the dead userGenerateJWT stub whose uuid is never read, six client-side render/debug identifiers that are DOM ids, CSS animation names and React keys rather than credentials, and four verify-*.mjs developer scripts whose random suffix only uniquifies throwaway test accounts. Those are genuinely not applicable and were sitting open across every develop-based PR in this batch.

Still left open as real: #18/#19 (smarts/index.tsx merge() walks Object.keys(value2) and assigns without a __proto__ guard — JSON.parse yields __proto__ as an own enumerable key, so this is genuine), #17, #45, #49. The prototype-pollution one wants a dedicated PR: it rides #135/#185/#382/#405 identically, so four branch-local fixes would just collide on merge.

[github-actions]

🤖 Lopu — the red control-plane / Lopu builds and repairs the all branch check on this PR is not this PR.

It is cancelled, and the job never ran a single step:

run  32925643015  "Build all branch"  pull_request_target  ->  cancelled
job  98047881134  created 03:13:29Z   completed 08:11:48Z   (4h58m)
     runner: ""    steps: []

Empty steps and no runner mean it waited its entire life in a queue and was then displaced. The chain:

1. The public .github/workflows/all-branch.yml listener fires on every pull_request_target lifecycle event across every open PR (opened, synchronize, reopened, ready_for_review, converted_to_draft, edited, closed), plus pushes to develop/main, plus an hourly cron — and it has no concurrency: block of its own, so each event creates a whole run.
2. It delegates to all-branch.yml@github-actions, whose rebuild job sits in the repository-wide group lopu-agent-fleet-${{ github.repository }} with queue: max, cancel-in-progress: false.
3. queue: max is real and supported (shipped 2026-05-07) but caps at 100 queued entries; overflow is cancelled. That one global group is shared by every Lopu lane — reviews, resolutions, promotions, rebases, CodeQL, previews and the all-branch doctor.

It is systemic, not a one-off. Last 100 runs of that listener: 45 cancelled, 31 success, 24 pending. PR #395 carries the identical symptom, and on PR #413 the same backlog has left Lopu CodeQL all branches and Web CI queued for 4+ hours against the reviewed head — so the security analysis on our most security-sensitive open PR has not actually run.

I deliberately did not change the control plane for this, because every candidate fix is an architecture decision rather than a repair: narrowing the listener's triggers means editing a workflow on main/develop (outside the protected controller checkout), and giving the rebuild its own group would break the documented "no doctor overlaps a PR review, promotion, conflict resolution, rebase, or stack cascade" invariant that resolve-pr-conflicts-routing-contract.mjs:509, promotion-worker-routing-contract.mjs:78/82/107 and build-all-branch.mjs:192 all assert.

@lopugit — your call, in increasing order of invasiveness:

1. Give the public listener its own concurrency: { group: all-branch-listener, cancel-in-progress: true }. The all branch is a fully derived artifact rebuilt from scratch each time, so superseding a queued rebuild is not merely safe, it is correct — 100 queued rebuilds produce one useful result. This alone stops PR lifecycle chatter from flooding the fleet queue.
2. Drop edited and converted_to_draft from the listener's types.
3. Split the deterministic union build from the model-backed doctor so only the doctor takes the fleet lock. Biggest win, biggest change — they are currently fused because they share a working tree.

The docs change itself is good and I have merged-in one small correction; details in my review comment.

[github-actions]

⚠️ Deterministic product contract advisory

These examples are warning-only. They do not fail the build/API contexts or block this PR.

• ✅ Web CI required-context examples
• ⚠️ Thin-listener topology examples (exit 1)

Sanitized tail

node:internal/modules/run_main:123
    triggerUncaughtException(
    ^

AssertionError [ERR_ASSERTION]: all-branch.yml must stay retired; Lopu PR manager owns its former public triggers

true !== false

    at file:///home/runner/work/thingtime/thingtime/remix/scripts/workflow-caller-contract.mjs:39:10
    at ModuleJob.run (node:internal/modules/esm/module_job:343:25)
    at async onImport.tracePromise.__proto__ (node:internal/modules/esm/loader:681:26)
    at async asyncRunEntryPointWithESMLoader (node:internal/modules/run_main:117:5) {
  generatedMessage: false,
  code: 'ERR_ASSERTION',
  actual: true,
  expected: false,
  operator: 'strictEqual',
  diff: 'simple'
}

Node.js v22.23.2

[github-actions]

🧑‍🔧 Lopu — the doc is accurate and needs no changes. The 22 CodeQL alerts on this head did need work.

On the document: I checked the load-bearing claims rather than taking them on trust. getClientCachedFor / getHomeThingtimeDb really are in mongodb/collections.ts and the per-request AsyncLocalStorage seam really is in mongodb/endpoint.ts, so §5's "the plumbing already exists" argument is real rather than aspirational. things_v2 and COLLECTION_SCHEMA_VERSIONS.things = 2 match the registry, and develop's root vercel.json does carry regions: ["syd1"] + "fluid": true with remix/vercel.json gone. Two relative links don't resolve at this head (../../vercel.json, ../../performance/TODO.md) purely because the branch is ~19 days behind — both exist on develop, so they resolve in the merged result. Nothing to fix.

On CodeQL: none of the 22 come from this PR — it adds one Markdown file — but they're attributed to this head, so I worked all of them. Ten fixed, ten dispositioned with evidence, two left open.

The one worth your attention is prototype pollution in smarts/index.tsx merge() (#18/#19). It was live, not theoretical. merge() walked Object.keys(value2) and assigned straight through, and JSON.parse('{"__proto__":{…}}') produces an own enumerable __proto__ key — so any merge of parsed data ran Object.prototype's setter. I transpiled the real module and proved it before fixing:

### before ###                                    ### after ###
not ok 1 - does not pollute Object.prototype      ok 1..4
  actual: 'yes'                                   # pass 4  # fail 0
not ok 2 - ...via overwriteAll either
not ok 3 - blocks constructor/prototype
ok   4 - still merges ordinary properties         (no regression, both runs)

Also fixed: the Commander's escaping ahead of eval() (#30/#31/#35/#36/#37) — quotes were escaped before backslashes, which threw SyntaxError on a\"b and silently corrupted C:\path\to\file → C:path\to\file; the V1 variant additionally left ${…} live inside its backtick literal. A never-implemented userGenerateJWT stub whose only body was an unused Math.random() "uuid" (#7) — zero references repo-wide, deleted. Two dead debugUuid declarations (#27/#28).

Dispositioned as not applicable: the UI-identifier Math.random() cases (#8/#14/#26/#29 — React keys, DOM ids, an undo/redo bookkeeping key, each traced to its only consumers), permissive CORS in the retired deprecated/api (#45 — not a workspace member, no root script references it, not in the deploy config), the two verification scripts' throwaway account credentials (#77–#80), and the resolve-pr-conflicts.yml untrusted-checkout alert (#75) — the flagged checkout is the file's only one and its job's if reads github.event_name != 'pull_request_target'; the query doesn't evaluate job guards. I checked the live protected controller too, which is stricter again, so no control-plane fix was warranted and I touched nothing under .github/.

Two I deliberately left open rather than dismiss:

• [codex] Document PR #16 follow-up review notes #17 (js/double-escaping, the Raycast regex converter) — the finding is correct, but it's a local dev command on the developer's own clipboard and the file already says TODO: fix this so it's not hardcoded shit. A correct fix means guessing at intended semantics, so it wants a small dedicated change, not churn from me.
• Untrack remix/.env.auto; branch-aware Vercel via VERCEL_GIT_COMMIT_REF #49 (js/stack-trace-exposure, api/http.ts:12) — the cited sink is the shared json() helper, not the leak site. The snapshot carries no code-flow data and the API returns none, and I couldn't find a path from any .stack into a response by inspection. I won't disposition what I can't demonstrate. Re-running with code flows retained would pin the caller.