Skip to content

fix(v4): load SQLite BLOB columns as bytea - #1784

Open
ritojoao wants to merge 1 commit into
dimitri:mainfrom
ritojoao:fix/1782-sqlite-blob-bytea
Open

ritojoao wants to merge 1 commit into
dimitri:mainfrom
ritojoao:fix/1782-sqlite-blob-bytea

Conversation

@ritojoao

Copy link
Copy Markdown

Fixes #1782

What happens

In the v4 (Clojure) SQLite source, read-rows and read-query turn each value into a string with (str v). For a BLOB the JDBC driver returns a byte[], so PostgreSQL receives the Java object name ([B@1b2c3d4) instead of the bytes. In create mode the column is also created as text, because pg-type-for in ddl/common.clj has no case for bytea.

The change

  • source/sqlite.clj: a small jdbc-value->str helper writes a byte[] as bytea hex (\x...), the same way mysql.clj and cast.clj already do. Other values still go through (str v), and NULL stays NULL.
  • ddl/common.clj: bytea is passed through in pg-type-for, next to uuid and xml, so tables created by pgloader get a bytea column.

Tests

  • New unit tests in ddl_test.clj (bytea column definition) and source/sqlite_test.clj (hex conversion, including an empty array).
  • End to end against PostgreSQL 18, in create mode and in data-only mode, with a SQLite table holding x'deadbeef', x'00ff7f80005c0a09', an empty blob and NULL: before the change the column is text and holds [B@...; with the change it is bytea, the bytes match, the empty blob has length 0 and NULL is kept. v3 gives the same result as the fixed v4.
  • Unit suite: 76 tests / 356 assertions with 1 failure on the base commit, 78 / 363 with 1 failure with the change. The failure is the same one in both, sqlite-type->pg-test for real (sqlite_test.clj:42), and is not related to this change.

Not checked

Large BLOBs (several MB), the integration suites under clojure/tests, and user cast rules such as type blob to text.

SQLite BLOB values were turned into strings with (str byte[]), so
PostgreSQL received "[B@<hash>" instead of the bytes. Emit them as
bytea hex (\x...) and map the bytea target type through as-is in
pg-type-for, so tables created by pgloader get bytea instead of text.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

v4 (Clojure) SQLite → PostgreSQL: BLOB columns loaded as [B@<hash> (JVM byte[] toString) instead of binary data

1 participant