Skip to content

Add pgoutput logical decoding plugin and make it the default - #58

Merged
teknogeek0 merged 3 commits into
mainfrom
feat/pgoutput-plugin
Aug 27, 2026
Merged

Add pgoutput logical decoding plugin and make it the default#58
teknogeek0 merged 3 commits into
mainfrom
feat/pgoutput-plugin

Conversation

@teknogeek0

Copy link
Copy Markdown
Collaborator

Summary

Adds the pgoutput logical decoding plugin and makes it the default.

pgoutput ships with Postgres core, so CDC works without installing an
extension on the source server. It also sends a compact binary protocol: on a
mixed INSERT/UPDATE/DELETE workload it used 4.5x less network volume and about
4x less CPU on the source than wal2json.

test_decoding and wal2json remain supported via --plugin.

How it works

The receive step decodes the pgoutput binary protocol and writes a
wal2json-shaped JSON message, so the transform and apply steps are unchanged.

pgoutput sends only the tables of a publication. Without --publication,
pgcopydb builds one from the --filters table list and drops it during
pgcopydb stream cleanup, so the source server does the filtering.

Usage

pgcopydb clone --follow --plugin pgoutput --filters filters.ini

Multi-step migrations pass the filters at the snapshot step, where the
publication is created:

pgcopydb snapshot --follow --plugin pgoutput --filters filters.ini
pgcopydb stream setup
pgcopydb clone --filters filters.ini

New options

  • --publication — use an existing publication instead of letting pgcopydb
    manage one
  • --filters on pgcopydb snapshot

Tests

New suites cdc-pgoutput, cdc-filtering-pgoutput and follow-pgoutput, and
cdc-low-level now runs on pgoutput. All are in the CI matrix.

Coverage includes INSERT/UPDATE/DELETE/TRUNCATE, REPLICA IDENTITY DEFAULT and
FULL, unchanged TOAST values, multi-relation TRUNCATE, and both include-only
and exclude filters. Data fidelity is checked with source/target checksums over
float8 and float4 (including denormals, Infinity and NaN, and float8 used as a
WHERE-clause key), bigint and numeric ranges, text needing escaping, quoted
identifiers with embedded double quotes, and uuid, date, time, interval, inet,
arrays and jsonb.

tests/ci-parity.sh runs the CI suite list and Postgres versions locally.

Full matrix: 33 suites on PostgreSQL 16, 17 and 18.

pgoutput ships with Postgres core, so CDC works without installing an
extension on the source server. It uses a compact binary protocol, which
lowers network volume and CPU on the source.

The receive step decodes the binary protocol and writes a wal2json-shaped
JSON message, so the transform and apply steps are unchanged.

pgoutput sends only the tables of a publication. Without --publication,
pgcopydb builds one from the --filters table list and drops it during
"pgcopydb stream cleanup". Add --filters to "pgcopydb snapshot", which is
where the publication is created in a multi-step migration.

test_decoding and wal2json remain supported via --plugin.
Postgres now restricts logical decoding to the plugins listed in
output_plugin_libraries, which defaults to "pgoutput, test_decoding". Add
wal2json to that list on the source servers of the suites that use it.
psql widens expanded output rows to the record header width since 18.6, so
compare the unit test output with "diff -w". These tests assert content.

ci-parity.sh runs the suites and Postgres versions from the CI workflow
locally, pulling images first and cleaning up containers and volumes.

@robdaly93 robdaly93 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed from the migration/field side, and ran the pgoutput path against PostgreSQL 18.6 locally.

Approach looks right to me. Decoding pgoutput into the existing wal2json JSON shape so transform and apply stay untouched keeps the change well contained, and emitting values as strings rather than JSON numbers is actually a fidelity improvement over the wal2json path for numeric. The 'K' vs 'O' identity handling and the synthesized old-key case when no old tuple is sent both look correct to me.

I spent most of my time on the new publication behavior, since pgcopydb performing DDL on the source is new. Noting what I confirmed, for anyone reading this later:

  • A table in the publication with no replica identity gets UPDATE/DELETE rejected on the source: ERROR: cannot update table "..." because it does not have a replica identity and publishes updates. This is the prerequisite in the "Replica Identity and lack of Primary Keys" docs section doing its job — non-PK tables need REPLICA IDENTITY FULL or USING INDEX for CDC under any plugin.
  • A --filters-scoped publication only covers the filtered tables, so excluded tables are unaffected.
  • pgcopydb stream cleanup correctly drops the publication it created and leaves a user-supplied --publication alone.

One small non-blocking thing you may want to look at: the publication builder applies exclude-schema and exclude-table, but not exclude-table-data. So a table excluded only from the data copy still joins the publication if someone used that on a large non-PK log table they'd hit the error above. Might be worth aligning the filter handling or adding a doc note.

LGTM.

@teknogeek0
teknogeek0 merged commit fef387e into main Aug 27, 2026
105 checks passed
@teknogeek0
teknogeek0 deleted the feat/pgoutput-plugin branch August 27, 2026 19:16
teknogeek0 added a commit to planetscale/migration-scripts that referenced this pull request Aug 27, 2026
Add pgcopydb-helpers/env-template as the reference ~/.env. The user copies
it to ~/.env on the migration instance. The provisioning templates no longer
create that file.

The file cannot be named .env here: .gitignore blocks the name to keep
credentials out of git, and the templates deploy with
"cp -r pgcopydb-helpers/* /home/ubuntu/", where the shell glob does not
match dotfiles.

Move the remaining hardcoded tunables into ~/.env:

- OUTPUT_PLUGIN, new, default pgoutput. planetscale/pgcopydb#58 adds the
  pgoutput plugin and makes it the pgcopydb default. It is part of
  PostgreSQL core, so the source server needs no extension. Set
  OUTPUT_PLUGIN=wal2json to keep the previous plugin.
- SPLIT_TABLES_LARGER_THAN, default 50GB, was a literal in three scripts.
- FILTER_FILE, default ~/filters.ini, was a literal in five scripts.

run-migration.sh, resume-migration.sh and resume-cdc.sh now log the
effective plugin, jobs, split size and filter path to migration.log.

drop-replication-slots.sh also drops the source publication. pgcopydb
creates one named after the replication slot for pgoutput, so without this
it is left behind on the source after every migration. The drop needs
ownership of the publication, so the script warns and continues on failure.

Every script that sources ~/.env now checks the file exists first and
prints the cp command if it does not. Without a guard the scripts failed
with a raw "No such file or directory" from the shell, which is reachable
now that no template creates the file.

Checked with bash -n and shellcheck on the helper scripts and the terraform
templates, and cfn-lint on the CloudFormation template.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants