Skip to content

backport: MySQL/MariaDB v3 bug fixes + extend mysql-unit-full to cover v3 - #1714

Merged
dimitri merged 5 commits into
mainfrom
feat/mysql-v3-backport
Jun 22, 2026
Merged

dimitri merged 5 commits into
mainfrom
feat/mysql-v3-backport

Conversation

@dimitri

@dimitri dimitri commented Jun 22, 2026

Copy link
Copy Markdown
Owner

Backports all relevant fixes from #1708 to the pgloader v3 Common Lisp codebase, and extends the mysql-unit-full integration suite to run the same tests against both v3 and v4.

v3 fixes

Fix Issue Description
hex-to-bytea transform #1066 New transform function in transforms.lisp
No-typemod DDL guard #1066 Strip typemod for types that reject it (bytea, text, boolean, uuid, …)
first-match-using bug #1522 Column-level using fn without to TYPE was silently dropping the transform
CHECK constraints #1645 New list-all-checks.sql, fetch-check-constraints, create-pgsql-check-constraints
datetime/timestamp precision #1629 Add :drop-typemod nil to all datetime/timestamp cast rules so DATETIME(3) → timestamptz(3)
time cast rule #1629 New explicit rule so TIME(3) → time(3)
ON UPDATE CURRENT_TIMESTAMP (MySQL 8) #1560/#1196 MySQL 8 returns DEFAULT_GENERATED on update … in EXTRA; now detected via suffix match
CURRENT_TIMESTAMP default (MariaDB) #1403 MariaDB lowercases current_timestamp(6); fixed with case-insensitive comparison

Test coverage

mysql-unit-full now runs against v3 (MySQL 8 and MariaDB) via new mysql-unit-full-v3 Makefile targets and .v3 / .v3.mariadb variant expected files. All 7 tests pass on both engines for both v3 and v4.

Deliberate v3 vs v4 behavioral differences captured in variant files (not bugs):

  • AUTO_INCREMENT → serial (v3) vs GENERATED … AS IDENTITY (v4)
  • BIT(8) → bit fixed-length (v3) vs bit varying (v4)
  • BIT(1) → boolean (v3) vs bit varying (v4)
  • Trigger named on_update_current_timestamp (v3) vs trg_<table>_<col> (v4)

Closes #1066 (v3)
Closes #1522 (v3)
Closes #1560 (v3)
Closes #1196 (v3)
Closes #1403 (v3)
Closes #1629 (v3)
Closes #1645 (v3)

dimitri added 3 commits June 22, 2026 22:01
Backports all relevant fixes from PR #1708 to the pgloader v3 CL codebase
and extends the mysql-unit-full integration suite to run against v3 (MySQL
and MariaDB) using variant expected files for deliberate behavioral
differences (serial vs identity, bit vs bit varying, trigger naming).

## v3 fixes

### hex-to-bytea transform (src/utils/transforms.lisp)
Add hex-to-bytea: converts plain / 0x-prefixed / \x-prefixed hex strings
to PostgreSQL \x... bytea notation. (#1066)

### No-typemod DDL guard (src/pgsql/pgsql-ddl.lisp)
Add effective-type-mod: strips typemod for types that don't accept one
in PostgreSQL (bytea, boolean, text, date, uuid, json, jsonb, xml, money,
oid). Prevents 'type modifier is not allowed for type bytea' errors when a
column cast targets bytea but the source typemod is propagated. (#1066)

### first-match-using fix (src/sources/common/casting-rules.lisp)
apply-casting-rules was computing first-match-using (the transform from a
using-only column cast rule) but never passing it to make-pgsql-type. The
finally clause now uses (or using first-match-using) so column-level
'using fn' casts without an explicit 'to TYPE' apply the transform.
This is the v3 equivalent of the WITHOUT TYPE feature. (#1522)

### CHECK constraints for MySQL (src/sources/mysql/*)
- New list-all-checks.sql: queries information_schema.TABLE_CONSTRAINTS
  joined with information_schema.CHECK_CONSTRAINTS (MySQL 8.0.16+; returns
  empty on older servers and MariaDB, safe to run anywhere).
- New fetch-check-constraints method: adds constraints to the catalog by
  replacing MySQL backtick quoting with PostgreSQL double-quote quoting.
- New add-check-constraint helper + table-check-constraint-list slot on the
  table struct (src/utils/catalog.lisp).
- New create-pgsql-check-constraints function runs ALTER TABLE ADD
  CONSTRAINT ... CHECK (...) in the post-load phase
  (src/pgsql/pgsql-create-schema.lisp, src/load/migrate-database.lisp).
(#1645)

### datetime/timestamp/time precision preserved (mysql-cast-rules.lisp)
Add :drop-typemod nil to all datetime/timestamp catch-all cast rules so
DATETIME(3) maps to timestamptz(3) instead of the no-typemod default of 6.
Add explicit time cast rule preserving typemod so TIME(3) maps to
time(3). (#1629)

### ON UPDATE CURRENT_TIMESTAMP detection for MySQL 8 (mysql-cast-rules.lisp)
MySQL 8 returns 'DEFAULT_GENERATED on update CURRENT_TIMESTAMP' in the EXTRA
column; the old pattern only matched 'on update CURRENT_TIMESTAMP'. Fixed
with uiop:string-suffix-p so both MySQL 5/8 patterns are recognized.
(#1560/#1196)

### CURRENT_TIMESTAMP default case-insensitive (mysql-cast-rules.lisp)
MariaDB lowercases the default to 'current_timestamp(6)'; the original
check used uiop:string-prefix-p which is case-sensitive. Switch to
case-insensitive substring comparison so both MySQL and MariaDB defaults
are normalized to :current-timestamp. (#1403)

## Test coverage

mysql-unit-full now runs against v3 (MySQL and MariaDB) via new Makefile
targets and .v3 / .v3.mariadb variant expected files. Deliberate v3 vs v4
differences captured in variants:
- AUTO_INCREMENT → serial (shows nextval default) vs generated identity
- BIT(8) → bit (fixed-length) vs bit varying
- BIT(1) → boolean vs bit varying
- Trigger named on_update_current_timestamp vs trg_<table>_<col>

All tests pass (MySQL v3, MySQL v4, MariaDB v3, MariaDB v4).
…t suite

Deletes four overlapping MySQL test suites (v4only, my, source,
mysql-unit-full) and replaces them with a single mytest suite that loads
every table into one MySQL database (pgloader_mytest) and one PostgreSQL
schema (mytest).

Changes by area:

## Deleted suites
- v4only/        — orphaned; seed never loaded; identical SQL + expected
                   output to mysql-unit-full (just different schema name)
- my/            — legacy pgloader database; absorbed into mytest
- source/        — source/source2 databases; absorbed into mytest
- mysql-unit-full is now MySQL-side-only-via-mariadb; MySQL runs mytest

## mytest/mytest.sql (unified seed)
All tables from my.sql + history.sql + seed.sql (source + source2) +
mysql-unit-full.sql in one pgloader_mytest database.  Notable changes:

- Expression index uses MySQL-version-gated conditional comment
    /*!80013 CREATE INDEX ... ((LOWER(email))) */;
  so MariaDB silently ignores it without a syntax error.

- SET SESSION sql_mode='' at the top allows zero-date values in legacy
  tables (races.date, events.created_at).

- ALTER USER ... IDENTIFIED WITH mysql_native_password removed from seed.
  Kept as a doc comment explaining the qmynd limitation.  The container
  already starts with --default-authentication-plugin=mysql_native_password
  so the pgloader user is compatible; see load-datasets.sh for the manual
  command if needed against a plain MySQL 8 instance.

- source2.things merged as a plain table; the multi-database aspect is
  explicitly noted in a comment (see report section below).

## mytest/mytest.load
Merged CAST rules from my.load and mysql-unit-full.load.  WITH options
use the concurrency settings from my.load (2 threads, rows-per-range).

## SQL verification queries (11 total)
01 tables       — full merged table list (31 tables)
02 schema       — races column types (from old my/sql/02-schema)
03 indexes      — non-PK indexes in mytest schema
04 expression-index — lower() expression index exists (MySQL 8.0.13+)
05 camelcase    — CamelCaseTable row count + column names
06-11           — from mysql-unit-full (datetime precision, bit defaults,
                  check constraints, on-update trigger, hex-to-bytea,
                  without-type); schema changed to mytest

## Expected files
Generated fresh for both v4 (baseline) and v3 (--variant v3).

## Infra updates
- Dockerfile: load mytest.sql instead of my.sql + history.sql + seed.sql
              + mysql-unit-full.sql
- load-datasets.sh: single mytest.tmpl step; qmynd doc comment
- docker-compose.yml: healthcheck uses pgloader_mytest; mysql-unit-full
  volume mount removed (no longer needed for MySQL)
- Makefile: all → mytest + sakila + f1db + source; mytest-v3 target added

## source2 multi-database scenario (not implemented — see note)
The former source2 database existed to test loading from two separate MySQL
databases.  It has been merged as the things table in pgloader_mytest.
There are currently no SQL verification queries that exercise source2 data,
so no test coverage is lost.  To add a real multi-database test, a second
MySQL database (pgloader_mytest_aux) and a second LOAD DATABASE command
in the load file would be needed — that is a separate, additive change.
@dimitri
dimitri force-pushed the feat/mysql-v3-backport branch from a7592c0 to ff17e63 Compare June 22, 2026 20:02
dimitri added 2 commits June 22, 2026 22:14
MySQL 5.7 reached end of life in October 2023.  The mysql57 suite ran the
same three queries as the mysql8 my-suite against the same seed schema, with
one behavioral difference: int(11) display widths made v3 map INT → bigint
instead of integer.  That quirk is a v3 limitation, not something being
fixed, so maintaining a dedicated EOL-server CI job to freeze it is more
cost than value.

Removes:
- clojure/tests/mysql57/ (Dockerfile, docker-compose, Makefile, all suites)
- CI matrix entries for mysql57 (v3) and mysql57 (v4)

Also reverts a half-applied fetch-check-constraints version guard that was
started to work around the now-removed suite.
Unmatched close parenthesis caused SBCL 2.1.11 (Debian) to fail
when compiling fetch-check-constraints, breaking all v3 test suites.
@dimitri
dimitri merged commit 9b522ee into main Jun 22, 2026
28 checks passed
@dimitri
dimitri deleted the feat/mysql-v3-backport branch June 22, 2026 20:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant