[Promote] feat(mongodb): CI telemetry satellite + things index storage reclaim (#583) - #585
Merged
Merged
Conversation
… things index storage
Production audit (2026-09-02): things_v2 held 1,824,527 docs of which 99.75%
were ci-* webhook telemetry written since August, with 64 indexes totalling
3.15 GB for 1.18 GB of logical data (322 MB on disk). Five indexes served
fields no code has ever written, the five v1-era kind_* indexes held only
null entries, and every CI row paid an entry in every index plus the wildcard
text index. Ingest was ~270k rows/day and accelerating.
- Register the `ciControl` collection (ciControl_v1): every ci-* Thing now
lives there via getCiControlCollection(), on a six-index plan (unique
shareId, dashboard sort, status counts, external-id lookups, per-parent
history, TTL on root expiresAt). No text index.
- Retention policy (ciControl/retentionCore.ts): events 14d, job rows 30d,
runs/deployments/previews 90d, entities never; env overrides
THINGTIME_CI_{EVENT,JOB,ACTIVITY}_RETENTION_DAYS (0 = forever).
- Ingest de-noising (ciControl/ingestPolicyCore.ts): the repository row is
upserted by every delivery but records a ci-event only on insert or a
status transition (was 49% of all events as active→active no-ops).
- things index plan: retire the seven dead/moved indexes by name at boot,
swap the five kind_* indexes and the sandbox TTL for partial replacements
(create-then-drop), fall back to drop-then-create only at the 64 cap, and
prune leftover `__rebuild` twins so the migration runner can always start.
- Admin migrations: relocate-ci-control-telemetry (time-budgeted,
insert-if-absent by shareId, expired rows deleted without copying) and
rebuild-things-indexes (one index at a time, unique constraints held by a
same-key partial twin throughout, foreign indexes left alone).
- Migration status carries a storage census per physical collection
(dataBytes/storageBytes/indexBytes/indexes; api.admin-migrations 1.1.0) and
the panel renders it with a bloat badge; the query workbench allowlists
ciControl (api.mongodb-raw-results 1.1.0).
- Docs: audit report (docs/architecture/mongodb-index-storage-audit.md),
FUNDAMENTALS §3 row, README env + storage runbook, TESTING checklist,
DECISIONS entries, schema descriptions.
Verified: unit suites green (collections, ci-control, migrations, schemas,
capabilities), typecheck ratchet at baseline, and a live local run through
the real API: signed webhook deliveries land only on ciControl_v1 with the
expected stamps, the relocation and rebuild migrations converge, and a
concurrent duplicate-shareId probe saw 20,423 E11000 rejections and 0
accepted inserts during the rebuild.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ex storage reclaim Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…trol satellite change AST-only update through scripts/graphify; the wrapper activated one portable snapshot and pruned the superseded ones. Markdown changes in this branch were not semantically re-indexed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ist bumps through contractVersion featureVersion is not read by createApiCapabilitiesManifest; contractVersion is. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The branch tip (graphify snapshot + capability-version fix) was three commits past the PR head GitHub recorded, with no synchronize event for over 30 minutes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
scripts/graphify update pruned develop's 170 tracked snapshot files (1.4 GB, 37M lines); GitHub then refused to compute the pull request diff and stopped synchronizing its head. This PR carries no graphify-out changes; the graph refresh belongs to the merge-time hooks. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…shot finding Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…torage-dffe19 feat(mongodb): move CI telemetry to a ciControl satellite and reclaim things index storage
github-actions
Bot
added
lopu: overlapping files
github-actions
Bot
removed
the
lopu: unknown state
github-actions
Bot
removed
lopu: mergeable
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Promotion of #583 (merged into
developas f67f408) tomain, authorized by the owner for this exact PR.developwas identical tomainbefore the merge, so this promotion carries exactly the #583 changes: theciControlsatellite collection with retention, thethingsindex-plan cleanup, therelocate-ci-control-telemetryandrebuild-things-indexesadmin migrations, the storage census on/migrations, and the docs. Graphify snapshots are net-unchanged.Rollout after this deploys to production: run
relocate-ci-control-telemetry(dry run, confirm, repeat until drained) and thenrebuild-things-indexesfrom/migrations.🤖 Generated with Claude Code