feat(attachments): admin backfill re-detects legacy octet-stream crystals - #321
Open
lopugit wants to merge 1 commit into
Open
feat(attachments): admin backfill re-detects legacy octet-stream crystals#321lopugit wants to merge 1 commit into
lopugit wants to merge 1 commit into
Conversation
…tals Ready attachments finalized before PR #319's magic-byte detection keep crystal contentType application/octet-stream with no detectedContentType, so browser-playable uploads still render as file cards. Add an admin-only POST /api/v1/attachments/backfill-detected-types that re-runs detection from the object's first 8 KiB and publishes exactly what completion would have: browser-playable containers flip to their inline contentType and mediaKind, other canonical sniffed types gain detectedContentType, and undetectable bytes stay untouched so a later, wider detector can claim them. Names, sizes, object keys, and object versions never change; the crystal's JSON-payload delta settles through the same transactional storage ledger markReady uses. Each pass is bounded (at most 200 rows, 5 workers, 25s wall clock), idempotent, cursor-paginated, and supports dryRun. New store methods listReadyUndetected and upgradeReadyCrystal (markReady-modeled, name and size immutable, object-version-fenced); rate key attachments.detectionBackfill; route + import map + apiDocs registration; service and route tests mirroring the completion-test mocks; TESTING.md checklist line. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
✅ Develop S3 preview ready
The alias passed the develop bucket CORS preflight and a final live PR/SHA fence. Generic Vercel Preview deployments use the shared development runtime; this controller adds the stable exact-SHA alias and marker-scoped cleanup. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR #319 added magic-byte detection at upload completion, but only for uploads finalized after it deployed. Ready attachments finalized earlier keep
crystal.contentType: application/octet-streamwith nodetectedContentType— e.g. the owner's QuickTime-container screen recording posted 2026-08-19 still renders as a file card. This PR adds an admin-only, batched, idempotent backfill that re-runs detection for those legacy rows and publishes exactly what completion would have.What it does
POST /api/v1/attachments/backfill-detected-types— admin-only (requireAdmin), same-origin JSON, rate-limited (attachments.detectionBackfill), attachment-family private response headers. One bounded pass per call:application/octet-streamwith nodetectedContentType, inshareIdorder (cursor pagination).AttachmentS3.detectContentType) and runsdetectedAttachmentType— the exact code path completion uses (attachments.tscomplete()).contentType/mediaKind(4-key crystal → renders inline); other canonical sniffed types gaindetectedContentType(5-key-with-octet crystal → labeled download row); undetectable bytes stay untouched — deliberately, so a later pass under a wider detector can still claim them.markReadyuses.objectVersionIdadopt the exact current version first via the same HEAD-verify +setObjectVersionIdrecovery download/completion use (dry runs only read it).Bounded + safe to re-run: at most 200 rows per pass, 5 workers, 25-second wall-clock budget (mirrors
reapExpired);dryRun: truewrites nothing; upgraded rows leave the candidate set so real passes converge;nextCursor/hasMorewalk the full backlog (required for dry runs); per-row races lose cleanly — the write is fenced on ready state, object version,updatedAt, andsizeBytes.Store additions:
listReadyUndetected+upgradeReadyCrystal(modeled onmarkReady; name/size immutable so object-byte accounting cannot move; conflicts throwAttachmentStoreConflictError).Registered in all three places (route file, import map in
server/routes/api/[...].ts,apiEndpointDocsentry) — the docs entry IS the Nitro route registration and auto-generates the two-docssmoke tests.Why an endpoint, not a migration
The admin migrations framework is collection-schemaVersion-shaped (fromVersion/toVersion, global lease, no S3 dependency); this backfill never bumps
schemaVersionand needs S3 access, cursor pagination, and dry-run reporting. The attachments family already has the service dependency injection, response conventions, and thecleanupmaintenance-endpoint precedent, so the sweep lives there.Testing
pnpm run test:attachments— 123 tests pass, including 5 new ones mirroring the completion-test mock patterns:missingObject) and losing races (conflicts)storage_unconfigured503 mapping, wall-clock budget stoppnpm run test:rate-limitand the fulltest:unitbattery pass.-docs200, GET 405, anonymous POST 401, non-admin 403, admin{"dryRun":true}→ 200 zero report, unknown body key → 400.Ops notes
storage_unconfiguredwithout the private-S3 env.{"dryRun": true}to count, then POST{}repeatedly followingnextCursorwhilehasMoreis true. WatchupgradedInline/labeledOpaquevsundetected/missingObject/conflicts/failed.🤖 Generated with Claude Code