Repository navigation
Conversation
Replace autopostgresqlbackup with atomic custom-format dumps and pg_restore. Pin PostgreSQL client tools to the database major version, preserve legacy SQL restore compatibility, and repair known migration-history inconsistencies before migration.
Write downloads to the expected restore bundle path, prepare Elasticsearch storage without traversing container-owned directories, and preserve existing Borg repositories on durable storage before extraction. Document custom database dump packaging and restore behavior.
Move PostgreSQL back to an isolated host path and define canonical deployed roots for shared data, logs, and secrets. Add host preparation, preflight, rendered Compose, and live mount guards with a migration and rollback runbook.
Validate dumps before promotion, serialize repository operations, verify new Borg archives, record successful backups, and schedule guarded monthly pruning. Restore the database before rotating files, install rsync for replication work, and prevent runtime migration generation.
Document routine and maintenance backups, destructive restore procedures, monthly retention, PostgreSQL data moves and upgrades, and the canonical deployed storage contract. Align command references and record the known test-generated migration artifact.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Create shared and PostgreSQL host directories before rendering local or CI Compose configuration. Keep staging and production storage under explicit host provisioning.
Avoid emitting environment-derived paths and rendered Compose values while retaining actionable storage contract failures.
Upgrade Django REST Framework to 3.17.2 and GitPython to 3.1.59 to incorporate their current security fixes.
can switch back to comses/base:noble once it's a properly lean runtime image
- use release.submitter instead of codebase.submitter - sync discourse user on publish - fix N+1 query
…ource create_release_from_source used __dict__.update() to bulk-copy release metadata onto the new release, which bypasses Django's ForeignKey descriptor for fields like submitter: the in-memory attribute looked updated but the underlying submitter_id column was never touched on save, silently discarding any submitter override for non-initial releases. Use setattr() per field instead so FK assignment goes through the descriptor as expected. Caught by test_new_draft_attributes_submitter_override_not_codebase_submitter failing in CI.
…s/backups storage-preflight required one exact owner/mode on /srv/logs/comses and docker/shared/backups. Infrastructure bind-mounts the canonical /srv/logs/comses and /srv/backups/comses roots onto their checkout counterparts and applies its own collaborative access policy (typically debian:operators 2775) to both, which the previous exact checks could never satisfy. Validate these two collaborative roots for security and functionality instead of one exact owner/mode: existence, no symlinks, expected Cinder-backed filesystem, not world-writable, effective read/write access, and that the bind target resolves to the same filesystem object as its checkout-path source. Setgid and operators-group collaboration are accepted; unexpected broad group access still fails closed. Private/service-owned paths keep exact owner/mode checks. Add COMSES_BACKUPS_ROOT (default /srv/backups/comses) alongside the existing COMSES_LOG_ROOT, and COMSES_EXPECTED_* overrides so the pinned-path checks are testable against a fixture tree.
prepare-host-storage chowned/chmoded docker/shared/backups and /srv/logs/comses to application-chosen owners, fighting infrastructure over the canonical collaborative bind roots it also manages. Only mkdir them if missing; never chown/chmod them. Application-owned child paths inside them (e.g. backups/repo) may still be created. Also fix a latent exact-mode bug: install -d -m MODE (and a plain chmod MODE) does not reliably clear an existing setgid bit on a directory that already exists; GNU chmod requires an explicit leading zero (e.g. 00755) to force-clear special bits on a directory. Add ensure_exact_dir, which always chmods with that explicit leading zero, so every private/service-owned path ends up at precisely the requested mode even if it previously had a stray setgid bit. Parameterize the /srv mount check and the pinned APP_ROOT/SHARED_ROOT/ etc. expectations via COMSES_EXPECTED_*/COMSES_SRV_ROOT overrides so this script is testable against a disposable fixture tree instead of the real /srv layout.
Cover the new collaborative-root/private-path ownership boundary against disposable mktemp fixtures (never real /srv storage), using passwordless sudo only to set fixture ownership/mode and to run the root-required prepare-host-storage: - debian:operators 2775 logs and backups pass storage-preflight - world-writable and unexpectedly group-writable collaborative roots fail closed - symlinked collaborative roots fail closed - bind target/checkout source filesystem-object parity is enforced - private paths with incorrect owner or mode fail closed - prepare-host-storage never alters the infrastructure-owned logs/ backups roots, clears a stale setgid bit on an exact-mode private path, and is idempotent across repeated runs - the legacy/new secret authority safeguard remains intact in both scripts Add deploy/scripts/tests/run-tests.sh and a deploy/scripts/tests/run-tests.sh === test-prepare-host-storage.sh === ok - prepare-host-storage runs on a fresh fixture ok - preparation does not alter infrastructure-owned logs/backups roots ok - docker/shared gets exact private owner/mode ok - prepare-host-storage re-runs against a setgid-tainted private path ok - stale setgid bit on docker/shared is cleared to exactly 0755 ok - prepare-host-storage runs a second time ok - repeated preparation is idempotent ok - legacy build/secrets with divergent docker/secrets blocks preparation ok - COMSES_STORAGE_AUTHORITY=new allows preparation to proceed --- 9 tests, 0 failed === test-storage-preflight.sh === ok - debian:operators 2775 logs and backups pass preflight ok - repeated preflight is idempotent ok - world-writable backups root fails preflight ok - group-writable logs root with an unexpected group fails preflight ok - symlinked logs root fails preflight ok - preflight passes again once the logs bind mount is restored ok - unbound backups root does not resolve to the checkout source ok - preflight passes again once the backups bind mount is restored ok - postgres data directory with the wrong owner fails preflight ok - redis data directory with the wrong mode fails preflight ok - preflight passes again once private paths are restored ok - legacy build/secrets without COMSES_STORAGE_AUTHORITY=new fails preflight ok - legacy build/secrets with COMSES_STORAGE_AUTHORITY=new passes preflight --- 13 tests, 0 failed target to run them together.
comses/infrastructure sets up /srv/logs/comses and /srv/backups/comses with bind-mounted checkout paths and sets its own collaborative access policy, debian:operators 2775 so we should adhere to that contract within reason this application strictly validates every other private/service-specific path. logs and backups are collaborative, with operator log shipping/rotation and Borg replication, and the bind-mount identity that makes one exact application-owned mode impossible, setgid and operators-group collaboration are expected and accepted major upgrades require manual oversight
Member
Author
|
accidental PR, use #899 instead |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR improves reliability across metrics downloads and deployment operations.
pg_dumpandpg_restoreworkflows.rsyncin the server image for upcoming off-host Borg replication.Motivation
Several related failures exposed gaps in the operational workflow:
Storage Layout
Staging and production now use:
/srv/apps/comses/srv/apps/comses/docker/shared/srv/apps/comses/docker/pgdata/srv/logs/comses/srv/apps/comses/docker/secretsBackup and Restore
The new
borg.backup-alltask:/shared/backups/.backup.lock.pg_restore --list./shared/backups/last-successful-backup.Monthly pruning uses the same lock and refuses to run unless the last verified local backup is no more than 48 hours old.
Full restore now verifies the archive, restores and migrates the database, and only then rotates shared files.
Deployment Notes
Before deploying the new storage layout:
docs/agents/storage-layout.md.docker/pgdatacontains the authoritative PostgreSQL cluster.docker/shared/postgresis absent.make prepare-host-storage.make storage-preflight.Backup and recovery procedures are documented in:
docs/agents/backup-restore-runbook.mddocs/agents/postgresql-upgrade-runbook.mdVerification
rsync.