Skip to content

[iceberg] Fix Iceberg metadata after a rollback - #9036

Open
plusplusjiajia wants to merge 1 commit into
apache:masterfrom
plusplusjiajia:gh-iceberg-schema-rollback
Open

[iceberg] Fix Iceberg metadata after a rollback#9036
plusplusjiajia wants to merge 1 commit into
apache:masterfrom
plusplusjiajia:gh-iceberg-schema-rollback

Conversation

@plusplusjiajia

@plusplusjiajia plusplusjiajia commented Aug 4, 2026

Copy link
Copy Markdown
Member

Purpose

After a rollback, later commits reuse the deleted snapshot ids, but IcebergCommitCallback finds the old vN.metadata.json and returns early — Iceberg readers keep seeing the pre-rollback rows forever (live rows (1,10),(3,30) read back as (1,10),(2,20)). The schema story also goes wrong: snapshot entries are stamped
with the latest schema id, and reused schema ids stay bound to abandoned definitions.

The fix:

  • every snapshot summary records a commit identity (the Paimon snapshot's uuid; legacy metadata without it stays trusted); existing metadata and bases are kept only on an identity match, otherwise rebuilt — inheriting the replaced metadata's table uuid and last-column-id high-water mark;
  • the abandoned suffix is retired from the actual files, only while the snapshot is still the Paimon head, after the replacement and catalog pointer are durable; retries recommit the catalog pointer with the previous version as base, and the REST committer treats an already-current catalog as a no-op instead of recreating
    the table;
  • snapshot entries record their own schema id; reused schema ids are detected by content comparison and trigger a rebuild; a plain schema-pointer rollback keeps history and refs.

@plusplusjiajia
plusplusjiajia force-pushed the gh-iceberg-schema-rollback branch 2 times, most recently from 69959a9 to 7569101 Compare August 5, 2026 09:09
@plusplusjiajia
plusplusjiajia force-pushed the gh-iceberg-schema-rollback branch from 7569101 to a53351c Compare August 5, 2026 11:34
@plusplusjiajia plusplusjiajia changed the title [iceberg] Fix Iceberg metadata after a schema rollback [iceberg] Fix Iceberg metadata after a rollback Aug 5, 2026
@plusplusjiajia plusplusjiajia reopened this Aug 5, 2026
@plusplusjiajia
plusplusjiajia marked this pull request as ready for review August 5, 2026 15:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant