Repository navigation
Conversation
Separate routine operations from disposable acceptance testing. Prevent controller subprocesses from consuming later pasted commands and document the verified backup, transfer, scheduler, and recovery behavior.
make bootstrap failed on any workstation that already had credentials, although the runbook runs it first. Rotation still requires FORCE=1 and an incomplete configuration is still an error.
Pin comses/citation#74 merge commit fbce715.
Citation now stores year_published. Deriving it again with dateutil dropped 100 of 7673 public publications, such as "SEP-OCT 2007", from the year charts.
The tags widget bound an undefined SelectedTags observable, so knockout aborted and the filter never loaded options.
Show "curator (66%)" instead of "curator (66)%".
Long sponsor and journal names squeezed the vertical charts and were clipped by the footer.
plotly-latest is frozen at 1.58.5 and logs a console error; use the version the visualization page already pins.
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Four critical and two moderate one-vote findings remain unresolved.
Review effort: Lite
Findings: 4
Open (4)
What changed in this PR
Replaces Solr/Haystack with Elasticsearch 8 and adds digest-pinned, candidate-based deployment with restore, rollback, recovery, CI release handoffs, and visualization fixes.
Changes:
- Adds generation-based Elasticsearch indexes, aliases, synchronization, and validation.
- Adds deployment orchestration, backups, restores, migrations, rollback, and recovery.
- Updates Compose, CI, dependencies, documentation, templates, and tests while removing Solr assets.
Four critical and two moderate findings remain unresolved, each with one vote, covering Elasticsearch security, host compatibility and safety, search consistency, rollback retention, and bind validation.
| File | Summary |
|---|---|
uv.lock |
Removes legacy Solr/Haystack dependencies. |
tasks.py |
Removes legacy deployment tasks and updates Elasticsearch usage. |
staging.yml |
Adds candidate services and scheduler configuration. |
scripts/dev-restore.sh |
Adds validated local database restore. |
scripts/database.sh |
Wraps backup and restore operations. |
scripts/config.sh |
Adds safer configuration regeneration. |
scripts/compose.sh |
Protects deployment state during rendering. |
scripts/citation-config.sh |
Validates generated Citation configuration. |
scripts/checkout-citation.sh |
Checks out the pinned Citation revision. |
README.md |
Documents development and deployment workflows. |
pyproject.toml |
Replaces Solr dependencies with Elasticsearch clients. |
prod.yml |
Updates production services and scheduler settings. |
Makefile |
Adds deployment, migration, rebuild, and recovery commands. |
docs/proposals/candidate-centered-multi-host-deployment.md |
Documents the deployment design. |
dev.yml |
Updates local PostgreSQL and Elasticsearch services. |
deploy/travis/travis-solr.sh |
Removes obsolete Solr tooling. |
deploy/travis/solrconfig.xml |
Removes obsolete Solr configuration. |
deploy/solr/init.d/solr.in.sh |
Removes obsolete Solr configuration. |
deploy/solr/init.d/set-heap.sh |
Removes obsolete Solr tooling. |
deploy/solr/conf/synonyms.txt |
Removes obsolete Solr configuration. |
deploy/solr/conf/stopwords.txt |
Removes obsolete Solr configuration. |
deploy/solr/conf/solrconfig.xml |
Removes obsolete Solr configuration. |
deploy/solr/conf/schema.xml |
Removes the obsolete Solr schema. |
deploy/solr/conf/protwords.txt |
Removes obsolete Solr configuration. |
deploy/solr/conf/managed-schema |
Removes the obsolete Solr schema. |
deploy/solr/conf/lang/stopwords_en.txt |
Removes obsolete Solr configuration. |
deploy/solr/conf/currency.xml |
Removes obsolete Solr configuration. |
deploy/solr/conf/_rest_managed.json |
Removes obsolete Solr configuration. |
deploy/images/solr.Dockerfile |
Removes the Solr image. |
deploy/images/django.Dockerfile |
Pins the application image and adds scheduler tooling. |
deploy/elasticsearch.conf.d/log4j2.properties |
Removes obsolete configuration. |
deploy/elasticsearch.conf.d/elasticsearch8.yml |
Updates Elasticsearch 8 configuration. |
deploy/elasticsearch.conf.d/elasticsearch-dev.yml |
Removes obsolete development configuration. |
deploy/docker/test.sh |
Switches readiness checks to Elasticsearch. |
deploy/docker/scheduler.sh |
Adds the scheduler entrypoint. |
deploy/docker/prod.sh |
Removes Solr startup and uses Elasticsearch. |
deploy/docker/dev.sh |
Updates local startup and readiness checks. |
deploy/cron/monthly_catalog_tasks |
Runs monthly tasks through the scheduler. |
deploy/cron/daily_catalog_tasks |
Runs daily tasks with failure propagation. |
deploy/conf/config.template.ini |
Removes Solr configuration. |
deploy.sh |
Redirects operators to the Make workflow. |
core.properties |
Removes obsolete Solr metadata. |
catalog/settings/staging.py |
Adds staging email configuration. |
catalog/settings/base.py |
Removes Haystack and adds environment-based settings. |
catalog/core/visualization/plots.py |
Fixes publication years and top-ten charts. |
catalog/core/visualization/data_access.py |
Preserves dataframe columns and Citation years. |
catalog/core/views.py |
Replaces Haystack views with Elasticsearch queries. |
catalog/core/tests/test_visualization_plots.py |
Tests chart and year behavior. |
catalog/core/tests/test_visualization_cache_command.py |
Tests visualization cache rebuilding. |
catalog/core/tests/test_views.py |
Updates search and response tests. |
catalog/core/tests/test_suggested_merge_views.py |
Tests autocomplete validation. |
catalog/core/tests/test_search_sync.py |
Tests Elasticsearch synchronization and curator labels. |
catalog/core/tests/test_search_indexes.py |
Tests generation indexes and aliases. |
catalog/core/tests/test_rebuild_es_index_command.py |
Tests rebuild and validation commands. |
catalog/core/tests/test_public_layout.py |
Tests footer rendering. |
catalog/core/tests/test_export.py |
Tests CSV export behavior. |
catalog/core/tests/test_citation_integration.py |
Tests Citation integration. |
catalog/core/tests/common.py |
Fixes status comparison assertions. |
catalog/core/templates/search/indexes/citation/publication_text.txt |
Removes the obsolete Haystack template. |
catalog/core/templates/public/search.html |
Updates search documentation rendering. |
catalog/core/templates/public/home.html |
Pins Plotly and centralizes the footer. |
catalog/core/templates/public/base.html |
Adds the shared footer. |
catalog/core/search_sync.py |
Synchronizes publication changes to Elasticsearch. |
catalog/core/management/commands/validate_search_indexes.py |
Adds search-index validation. |
catalog/core/management/commands/rebuild_es_index.py |
Rebuilds public and curator indexes. |
catalog/core/management/commands/populate_visualization_cache.py |
Adds cache-clearing support. |
catalog/core/forms.py |
Reimplements curator search filters. |
catalog/core/apps.py |
Registers search synchronization signals. |
base.yml |
Replaces Solr with pinned Elasticsearch and PostgreSQL services. |
.gitignore |
Ignores dumps and private deployment data. |
.github/workflows/release-tag.yml |
Publishes stable releases without rebuilding. |
.github/workflows/docker-build.yml |
Adds testing, image publication, and handoff generation. |
.dockerignore |
Excludes dumps and private data from builds. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Candidate extraction uses filter="data", which Python 3.10.0-3.10.11 and 3.11.0-3.11.3 lack. host-check now tests for the feature, as the tarfile documentation recommends.
A missing active.json alone does not prove the host is fresh. Restore now fails closed when the current database already contains publications.
A bare port in CATALOG_HTTP_BIND published nginx on every interface. The documented override now needs an explicit address, and make status shows the bind.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
scripts/catalogctl.py) for independent staging and production hosts: digest-pinned candidates, fresh-host restore, verified on-host backups, gated schema migration and search rebuild, transactional deploy, rollback, and recovery;mainbuild publishes an immutable image andrelease-handoff.json, and a stablevYYYY.MMtag creates the GitHub Release without rebuilding;fbce715;Operator procedure:
docs/deployment-runbook.md.Disposable-host validation:
docs/deployment-acceptance-testing.md.Design and deferred work:
docs/proposals/candidate-centered-multi-host-deployment.md.Closes #193.
Closes #194.
Verification
make check,migrations-check,cite-check,cite-migrations-check,test-all(Catalog 76 and Citation 79 tests), anddeploy-controller-test(31 tests) pass.Counts were preserved (290,922 publications, 9,538 primary), and both hosts validated every search alias.
That rehearsal pinned Citation
a59d379.Migration 0038 filled 289,278 publication years in about 41 s with a peak of about 1 GiB of memory; Address #74 review: export coverage, CSV safety, and backfill memory citation#76 reduces that to 100 MiB.
Browser checks passed for search, the curator export (9,538 rows), merges, visualization, and the reviewed and unreviewed status round trip.
Deployment notes