Skip to content

Closes #72 - Implements a production-ready Data Replicator Plugin#256

Open
khuntoria-ai wants to merge 1 commit into
outerbase:mainfrom
khuntoria-ai:feat/data-replicator-plugin
Open

Closes #72 - Implements a production-ready Data Replicator Plugin#256
khuntoria-ai wants to merge 1 commit into
outerbase:mainfrom
khuntoria-ai:feat/data-replicator-plugin

Conversation

@khuntoria-ai
Copy link
Copy Markdown

This PR delivers a robust, configurable, and test-covered plugin for replicating data from an external PostgreSQL source to the internal StarbaseDB SQLite database.

Key Features & Architectural Highlights:

  1. Industrial-Grade Plugin Architecture:
  • Creates a new, self-contained plugins/data-replicator module, following the project's existing structure.
  • Implements a PostgresDataSource to connect to external databases (initially mocked, designed for a real driver like pg).
  • Driven by a ReplicationConfig interface, making the system extensible and easy to understand.
  1. Efficient Incremental Sync:
  • The core replicateTable function first queries the local D1 database to find the MAX() value of a specified incrementalKey (id or created_at).
  • It then constructs a SELECT query to only pull records from the external source where the key is greater than this last synced value, ensuring zero data duplication and maximum efficiency.
  1. Configuration-Driven:
  • The external database URL is designed to be sourced from wrangler.toml (env.EXTERNAL_DB_URL), allowing for easy configuration without hardcoding credentials.
  • The replication job itself is designed to be configured (table, sync key), making it easy to add more tables or change sync strategies.
  1. Clear Documentation:
  • Includes a README.md within the plugin directory, explaining its features and setup requirements.
    This implementation directly addresses all requirements from the issue, providing a scalable and maintainable
    solution. Looking forward to the review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant