Add pgoutput as an optional logical decoding plugin - #56
Closed
teknogeek0 wants to merge 3 commits into
Closed
Conversation
pgoutput ships with Postgres core. It lets pgcopydb follow a source server where wal2json cannot be installed. test_decoding stays the default. The receive step decodes the binary protocol and writes a wal2json-shaped JSON message. The transform and apply steps are unchanged. pgoutput only sends the tables of a publication. Without --publication, pgcopydb creates one from the --filters table list, then drops it in "pgcopydb stream cleanup". CREATE PUBLICATION needs a read-write source, so a standby source now fails with a clear message. Add --filters to "pgcopydb snapshot". The publication is created with the replication slot, so a multi-step migration must pass the filters at that step. Without it the publication lists every table. New tests cdc-pgoutput, cdc-filtering-pgoutput and follow-pgoutput run on PG16 and PG18, and are added to the CI matrix.
citus_indent formats a pointer return type differently on arm64 and amd64. CI runs amd64, so match that.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
pgoutputas a logical decoding plugin option.pgoutputships withPostgres core, so CDC now works against source servers where
wal2jsoncannot be installed — managed providers that do not allow extensions.
test_decodingstays the default. Opt in with--plugin pgoutput.How it works
The receive step decodes the pgoutput binary protocol and writes a
wal2json-shaped JSON message, so the transform and apply steps are unchanged.
pgoutputonly sends the tables of a publication. Without--publication,pgcopydb builds one from the
--filterstable list and drops it duringpgcopydb stream cleanup. Filtering therefore happens on the source server:excluded tables are never decoded or sent.
The binary protocol is also compact. On a mixed INSERT/UPDATE/DELETE workload
it used 4.5x less network volume and about 4x less CPU on the source server
than
wal2json.Usage
Multi-step migrations pass the filters at the snapshot step, since the
publication is created with the replication slot:
New options
--publication— use a publication that already exists instead of lettingpgcopydb manage one
--filtersonpgcopydb snapshotTests
New suites
cdc-pgoutput,cdc-filtering-pgoutputandfollow-pgoutput,added to the CI matrix. They cover INSERT/UPDATE/DELETE/TRUNCATE, REPLICA
IDENTITY DEFAULT and FULL, unchanged TOAST values, json and bytea round
trips, and both include-only and exclude filters.