Repository navigation
backport: MySQL/MariaDB v3 bug fixes + extend mysql-unit-full to cover v3 - #1714
Merged
Merged
Conversation
Backports all relevant fixes from PR #1708 to the pgloader v3 CL codebase and extends the mysql-unit-full integration suite to run against v3 (MySQL and MariaDB) using variant expected files for deliberate behavioral differences (serial vs identity, bit vs bit varying, trigger naming). ## v3 fixes ### hex-to-bytea transform (src/utils/transforms.lisp) Add hex-to-bytea: converts plain / 0x-prefixed / \x-prefixed hex strings to PostgreSQL \x... bytea notation. (#1066) ### No-typemod DDL guard (src/pgsql/pgsql-ddl.lisp) Add effective-type-mod: strips typemod for types that don't accept one in PostgreSQL (bytea, boolean, text, date, uuid, json, jsonb, xml, money, oid). Prevents 'type modifier is not allowed for type bytea' errors when a column cast targets bytea but the source typemod is propagated. (#1066) ### first-match-using fix (src/sources/common/casting-rules.lisp) apply-casting-rules was computing first-match-using (the transform from a using-only column cast rule) but never passing it to make-pgsql-type. The finally clause now uses (or using first-match-using) so column-level 'using fn' casts without an explicit 'to TYPE' apply the transform. This is the v3 equivalent of the WITHOUT TYPE feature. (#1522) ### CHECK constraints for MySQL (src/sources/mysql/*) - New list-all-checks.sql: queries information_schema.TABLE_CONSTRAINTS joined with information_schema.CHECK_CONSTRAINTS (MySQL 8.0.16+; returns empty on older servers and MariaDB, safe to run anywhere). - New fetch-check-constraints method: adds constraints to the catalog by replacing MySQL backtick quoting with PostgreSQL double-quote quoting. - New add-check-constraint helper + table-check-constraint-list slot on the table struct (src/utils/catalog.lisp). - New create-pgsql-check-constraints function runs ALTER TABLE ADD CONSTRAINT ... CHECK (...) in the post-load phase (src/pgsql/pgsql-create-schema.lisp, src/load/migrate-database.lisp). (#1645) ### datetime/timestamp/time precision preserved (mysql-cast-rules.lisp) Add :drop-typemod nil to all datetime/timestamp catch-all cast rules so DATETIME(3) maps to timestamptz(3) instead of the no-typemod default of 6. Add explicit time cast rule preserving typemod so TIME(3) maps to time(3). (#1629) ### ON UPDATE CURRENT_TIMESTAMP detection for MySQL 8 (mysql-cast-rules.lisp) MySQL 8 returns 'DEFAULT_GENERATED on update CURRENT_TIMESTAMP' in the EXTRA column; the old pattern only matched 'on update CURRENT_TIMESTAMP'. Fixed with uiop:string-suffix-p so both MySQL 5/8 patterns are recognized. (#1560/#1196) ### CURRENT_TIMESTAMP default case-insensitive (mysql-cast-rules.lisp) MariaDB lowercases the default to 'current_timestamp(6)'; the original check used uiop:string-prefix-p which is case-sensitive. Switch to case-insensitive substring comparison so both MySQL and MariaDB defaults are normalized to :current-timestamp. (#1403) ## Test coverage mysql-unit-full now runs against v3 (MySQL and MariaDB) via new Makefile targets and .v3 / .v3.mariadb variant expected files. Deliberate v3 vs v4 differences captured in variants: - AUTO_INCREMENT → serial (shows nextval default) vs generated identity - BIT(8) → bit (fixed-length) vs bit varying - BIT(1) → boolean vs bit varying - Trigger named on_update_current_timestamp vs trg_<table>_<col> All tests pass (MySQL v3, MySQL v4, MariaDB v3, MariaDB v4).
…t suite
Deletes four overlapping MySQL test suites (v4only, my, source,
mysql-unit-full) and replaces them with a single mytest suite that loads
every table into one MySQL database (pgloader_mytest) and one PostgreSQL
schema (mytest).
Changes by area:
## Deleted suites
- v4only/ — orphaned; seed never loaded; identical SQL + expected
output to mysql-unit-full (just different schema name)
- my/ — legacy pgloader database; absorbed into mytest
- source/ — source/source2 databases; absorbed into mytest
- mysql-unit-full is now MySQL-side-only-via-mariadb; MySQL runs mytest
## mytest/mytest.sql (unified seed)
All tables from my.sql + history.sql + seed.sql (source + source2) +
mysql-unit-full.sql in one pgloader_mytest database. Notable changes:
- Expression index uses MySQL-version-gated conditional comment
/*!80013 CREATE INDEX ... ((LOWER(email))) */;
so MariaDB silently ignores it without a syntax error.
- SET SESSION sql_mode='' at the top allows zero-date values in legacy
tables (races.date, events.created_at).
- ALTER USER ... IDENTIFIED WITH mysql_native_password removed from seed.
Kept as a doc comment explaining the qmynd limitation. The container
already starts with --default-authentication-plugin=mysql_native_password
so the pgloader user is compatible; see load-datasets.sh for the manual
command if needed against a plain MySQL 8 instance.
- source2.things merged as a plain table; the multi-database aspect is
explicitly noted in a comment (see report section below).
## mytest/mytest.load
Merged CAST rules from my.load and mysql-unit-full.load. WITH options
use the concurrency settings from my.load (2 threads, rows-per-range).
## SQL verification queries (11 total)
01 tables — full merged table list (31 tables)
02 schema — races column types (from old my/sql/02-schema)
03 indexes — non-PK indexes in mytest schema
04 expression-index — lower() expression index exists (MySQL 8.0.13+)
05 camelcase — CamelCaseTable row count + column names
06-11 — from mysql-unit-full (datetime precision, bit defaults,
check constraints, on-update trigger, hex-to-bytea,
without-type); schema changed to mytest
## Expected files
Generated fresh for both v4 (baseline) and v3 (--variant v3).
## Infra updates
- Dockerfile: load mytest.sql instead of my.sql + history.sql + seed.sql
+ mysql-unit-full.sql
- load-datasets.sh: single mytest.tmpl step; qmynd doc comment
- docker-compose.yml: healthcheck uses pgloader_mytest; mysql-unit-full
volume mount removed (no longer needed for MySQL)
- Makefile: all → mytest + sakila + f1db + source; mytest-v3 target added
## source2 multi-database scenario (not implemented — see note)
The former source2 database existed to test loading from two separate MySQL
databases. It has been merged as the things table in pgloader_mytest.
There are currently no SQL verification queries that exercise source2 data,
so no test coverage is lost. To add a real multi-database test, a second
MySQL database (pgloader_mytest_aux) and a second LOAD DATABASE command
in the load file would be needed — that is a separate, additive change.
dimitri
force-pushed
the
feat/mysql-v3-backport
branch
from
June 22, 2026 20:02
a7592c0 to
ff17e63
Compare
MySQL 5.7 reached end of life in October 2023. The mysql57 suite ran the same three queries as the mysql8 my-suite against the same seed schema, with one behavioral difference: int(11) display widths made v3 map INT → bigint instead of integer. That quirk is a v3 limitation, not something being fixed, so maintaining a dedicated EOL-server CI job to freeze it is more cost than value. Removes: - clojure/tests/mysql57/ (Dockerfile, docker-compose, Makefile, all suites) - CI matrix entries for mysql57 (v3) and mysql57 (v4) Also reverts a half-applied fetch-check-constraints version guard that was started to work around the now-removed suite.
Unmatched close parenthesis caused SBCL 2.1.11 (Debian) to fail when compiling fetch-check-constraints, breaking all v3 test suites.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backports all relevant fixes from #1708 to the pgloader v3 Common Lisp codebase, and extends the
mysql-unit-fullintegration suite to run the same tests against both v3 and v4.v3 fixes
hex-to-byteatransformtransforms.lispfirst-match-usingbugusing fnwithoutto TYPEwas silently dropping the transformlist-all-checks.sql,fetch-check-constraints,create-pgsql-check-constraints:drop-typemod nilto all datetime/timestamp cast rules soDATETIME(3)→timestamptz(3)timecast ruleTIME(3)→time(3)DEFAULT_GENERATED on update …in EXTRA; now detected via suffix matchcurrent_timestamp(6); fixed with case-insensitive comparisonTest coverage
mysql-unit-fullnow runs against v3 (MySQL 8 and MariaDB) via newmysql-unit-full-v3Makefile targets and.v3/.v3.mariadbvariant expected files. All 7 tests pass on both engines for both v3 and v4.Deliberate v3 vs v4 behavioral differences captured in variant files (not bugs):
serial(v3) vsGENERATED … AS IDENTITY(v4)BIT(8)→bitfixed-length (v3) vsbit varying(v4)BIT(1)→boolean(v3) vsbit varying(v4)on_update_current_timestamp(v3) vstrg_<table>_<col>(v4)Closes #1066 (v3)
Closes #1522 (v3)
Closes #1560 (v3)
Closes #1196 (v3)
Closes #1403 (v3)
Closes #1629 (v3)
Closes #1645 (v3)