Run Flyway scripts outside of Payara#11517
Closed
poikilotherm wants to merge 18 commits intoIQSS:developfrom
Closed
Run Flyway scripts outside of Payara#11517poikilotherm wants to merge 18 commits intoIQSS:developfrom
poikilotherm wants to merge 18 commits intoIQSS:developfrom
Conversation
poikilotherm
added
Component: Code Infrastructure
poikilotherm
force-pushed
the
11516-run-flyway
branch
from
8109a34 to
567c978
Compare
poikilotherm
force-pushed
the
11516-run-flyway
branch
from
a13f991 to
8e82b72
Compare
…e from 4.20 to 6.3 database
…) Docker and Flyway This commit introduces a new Maven module, `dataverse-migrate-db`, for database migration. It utilizes Docker for setting up a PostgreSQL environment and Flyway for managing schema migrations. The configuration supports customizable properties and includes execution phases for setting up, dumping, migrating, and cleaning up the database. The setup of Postgres in Docker can be skipped to run the migrations against some other remote database. It is up to the user to ensure no harm comes from that.
Moved SQL definitions for missing tables from `db/extra` to the new `dataverse-migrate-db/extra-migrations` module. This improves organization and separates migration scripts from other database resources for easier maintenance.
…d and CC-BY. Deleted legacy database migration scripts no longer relevant due to upstream compatibility adjustments and resolved licensing updates. This cleanup helps maintain a more streamlined and accurate codebase.
Introduces migrations for `DatasetType` and `MakeDataCountProcessState` tables. These include schema definitions, relationships, constraints, and an index to support new functionalities. The initial extra migrations were done when 6.2 was released, now adding the others.
Moved `flyway.version` and `postgresql.server.version` to the `dataverse-parent` POM for better configuration centralization. Removed their redundant declarations from the root POM to maintain consistency. Enables reusing the values in the new `dataverse-migrate-db` module Downgraded the PostgreSQL server version from 17 to the one specified in the installation guide (16) to be in sync with the official recommendations.
The Flyway Maven plugin configuration has been completely removed, including its dependencies and settings as it has been migrated to the new `dataverse-migrate-db` module.
…iguration IQSS#10288 Introduces a new migration script to update `keywordValue` fields containing HTTP values to `keywordTermUri` (as seen in the release notes). Updates the POM to include a placeholder for toggling this migration, ensuring flexibility during upgrades. Using a Flyway placeholder, we can inject a Maven property into the SQL script and conditionally execute the migration. Defaulting to false (don't execute). See also: https://documentation.red-gate.com/fd/migration-placeholders-275218550.html
…hints Added row count check and more detailed notices for the `keywordValue` to `keywordTermUri` migration process. Added guidance for re-execution if migration is skipped.
…jects This migration identifies and optionally deletes problematic saved searches, dataset links, and collection links related to issue IQSS#7398. A new Maven property, `migrate.cleanupSavedSearches`, has been added to enable or disable the cleanup process during migrations.
This script identifies datasets with mismatched file access request flags caused by the curate command. It provides a manual fixing hint for affected cases, referencing issue IQSS#7687. The future possibility of an automated migration is noted.
This script identifies and removes templates without associated dataverses, addressing issue IQSS#8600. It also updates dataverses using such templates as defaults before deleting the orphan templates to ensure data integrity.
This commit adds a new SQL migration script to drop the EJB__TIMER__TBL table if it exists. This helps clean up unused database artifacts and improves maintainability.
…issues This script identifies datasets with restricted files lacking terms of access or request access settings. It logs affected datasets with owner details and provides references for further information in the Dataverse 5.11 release notes and issue IQSS#8191. If matching files/datasets are found, the admin needs to take manual action to resolve the situation (we cannot decide here which would be the way to go for any file).
Added explanatory comments and references to Dataverse 6.3 release notes to clarify migration. Included guidance for manual loading of the keywordTermURI field and revised notices for better user direction.
Removed the '-c' and '-C' flags (adding statements to DROP and CREATE the DB) from the pg_dump command, as they are not necessary for the migration process and stop loading the database dump in a container.
Fixed a typo in the volume file name from "dumb" to "dump" in the Docker PostgreSQL initialization configuration.
poikilotherm
force-pushed
the
11516-run-flyway
branch
from
8e82b72 to
6f49209
Compare
Adjusted existing migration scripts to reference which migration any of the workarounds for missing tables and indices mitigate. Ensured compatibility and alignment with names that would be autogenerated by EclipseLink for better consistency and maintainability.
Contributor
Author
|
As per https://iqss.slack.com/archives/C010LA04BCG/p1771358921833039 it would be good to run a consistency check if all dvobjects have no owner_id == null except for the root collection. |
Contributor
Author
|
Closing in favor of gdcc/dataverse-recipes#33 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Add a new submodule to run Flyway migrations outside Payara, enabling upgrading databases, out-of-band operations, trying out new migrations without deploying etc.
Which issue(s) this PR closes:
Special notes for your reviewer:
None
Suggestions on how to test this:
See docs. You just need Docker or a live DB to run this:
Note: you will need to get a database snapshot (Postgres Dump) from somewhere first... 😉
Does this PR introduce a user interface change? If mockups are available, please link/include them here:
Nope
Is there a release notes update needed for this change?:
TBD
Additional documentation:
TBD