Add pgoutput logical decoding plugin and make it the default - #58
Conversation
pgoutput ships with Postgres core, so CDC works without installing an extension on the source server. It uses a compact binary protocol, which lowers network volume and CPU on the source. The receive step decodes the binary protocol and writes a wal2json-shaped JSON message, so the transform and apply steps are unchanged. pgoutput sends only the tables of a publication. Without --publication, pgcopydb builds one from the --filters table list and drops it during "pgcopydb stream cleanup". Add --filters to "pgcopydb snapshot", which is where the publication is created in a multi-step migration. test_decoding and wal2json remain supported via --plugin.
Postgres now restricts logical decoding to the plugins listed in output_plugin_libraries, which defaults to "pgoutput, test_decoding". Add wal2json to that list on the source servers of the suites that use it.
psql widens expanded output rows to the record header width since 18.6, so compare the unit test output with "diff -w". These tests assert content. ci-parity.sh runs the suites and Postgres versions from the CI workflow locally, pulling images first and cleaning up containers and volumes.
robdaly93
left a comment
There was a problem hiding this comment.
Reviewed from the migration/field side, and ran the pgoutput path against PostgreSQL 18.6 locally.
Approach looks right to me. Decoding pgoutput into the existing wal2json JSON shape so transform and apply stay untouched keeps the change well contained, and emitting values as strings rather than JSON numbers is actually a fidelity improvement over the wal2json path for numeric. The 'K' vs 'O' identity handling and the synthesized old-key case when no old tuple is sent both look correct to me.
I spent most of my time on the new publication behavior, since pgcopydb performing DDL on the source is new. Noting what I confirmed, for anyone reading this later:
- A table in the publication with no replica identity gets
UPDATE/DELETErejected on the source:ERROR: cannot update table "..." because it does not have a replica identity and publishes updates. This is the prerequisite in the "Replica Identity and lack of Primary Keys" docs section doing its job — non-PK tables needREPLICA IDENTITY FULLorUSING INDEXfor CDC under any plugin. - A
--filters-scoped publication only covers the filtered tables, so excluded tables are unaffected. pgcopydb stream cleanupcorrectly drops the publication it created and leaves a user-supplied--publicationalone.
One small non-blocking thing you may want to look at: the publication builder applies exclude-schema and exclude-table, but not exclude-table-data. So a table excluded only from the data copy still joins the publication if someone used that on a large non-PK log table they'd hit the error above. Might be worth aligning the filter handling or adding a doc note.
LGTM.
Add pgcopydb-helpers/env-template as the reference ~/.env. The user copies it to ~/.env on the migration instance. The provisioning templates no longer create that file. The file cannot be named .env here: .gitignore blocks the name to keep credentials out of git, and the templates deploy with "cp -r pgcopydb-helpers/* /home/ubuntu/", where the shell glob does not match dotfiles. Move the remaining hardcoded tunables into ~/.env: - OUTPUT_PLUGIN, new, default pgoutput. planetscale/pgcopydb#58 adds the pgoutput plugin and makes it the pgcopydb default. It is part of PostgreSQL core, so the source server needs no extension. Set OUTPUT_PLUGIN=wal2json to keep the previous plugin. - SPLIT_TABLES_LARGER_THAN, default 50GB, was a literal in three scripts. - FILTER_FILE, default ~/filters.ini, was a literal in five scripts. run-migration.sh, resume-migration.sh and resume-cdc.sh now log the effective plugin, jobs, split size and filter path to migration.log. drop-replication-slots.sh also drops the source publication. pgcopydb creates one named after the replication slot for pgoutput, so without this it is left behind on the source after every migration. The drop needs ownership of the publication, so the script warns and continues on failure. Every script that sources ~/.env now checks the file exists first and prints the cp command if it does not. Without a guard the scripts failed with a raw "No such file or directory" from the shell, which is reachable now that no template creates the file. Checked with bash -n and shellcheck on the helper scripts and the terraform templates, and cfn-lint on the CloudFormation template.
Summary
Adds the
pgoutputlogical decoding plugin and makes it the default.pgoutputships with Postgres core, so CDC works without installing anextension on the source server. It also sends a compact binary protocol: on a
mixed INSERT/UPDATE/DELETE workload it used 4.5x less network volume and about
4x less CPU on the source than
wal2json.test_decodingandwal2jsonremain supported via--plugin.How it works
The receive step decodes the pgoutput binary protocol and writes a
wal2json-shaped JSON message, so the transform and apply steps are unchanged.
pgoutputsends only the tables of a publication. Without--publication,pgcopydb builds one from the
--filterstable list and drops it duringpgcopydb stream cleanup, so the source server does the filtering.Usage
Multi-step migrations pass the filters at the snapshot step, where the
publication is created:
New options
--publication— use an existing publication instead of letting pgcopydbmanage one
--filtersonpgcopydb snapshotTests
New suites
cdc-pgoutput,cdc-filtering-pgoutputandfollow-pgoutput, andcdc-low-levelnow runs on pgoutput. All are in the CI matrix.Coverage includes INSERT/UPDATE/DELETE/TRUNCATE, REPLICA IDENTITY DEFAULT and
FULL, unchanged TOAST values, multi-relation TRUNCATE, and both include-only
and exclude filters. Data fidelity is checked with source/target checksums over
float8 and float4 (including denormals, Infinity and NaN, and float8 used as a
WHERE-clause key), bigint and numeric ranges, text needing escaping, quoted
identifiers with embedded double quotes, and uuid, date, time, interval, inet,
arrays and jsonb.
tests/ci-parity.shruns the CI suite list and Postgres versions locally.Full matrix: 33 suites on PostgreSQL 16, 17 and 18.