[Bug] rewrite_file_index crashes or corrupts the index for files written before a column drop

[LuciferYang]

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

master

Compute Engine

Spark / Flink (CALL rewrite_file_index)

Minimal reproduce step

Create a table with a file index on a column, for example a bloom filter on v, with schema [k, a, v]. Write some rows so a data file is committed at schema 0. Then ALTER TABLE ... DROP COLUMN a. Run rewrite_file_index over the table. The rewrite reads the pre-drop file (still at schema 0) to rebuild its index.

What doesn't meet your expectations?

FileIndexProcessor reads the file with a projection and feeds the rows to the index writer. The writer projection holds positions into the file's own schema, but the same file-schema positions were also passed to ReadBuilder.withProjection, which indexes into the current table schema. Once the schema evolves the two drift apart:

• After a drop that shrinks the arity, a file-schema position can exceed the current schema. The read projects an out-of-range position and the rewrite fails with IndexOutOfBoundsException. Every file written before the drop fails.
• With same-arity drift (a drop plus a later add), the file-schema position lands on a different current column. The rewrite reads that column's values and builds the index under the original column's name, silently corrupting predicate pruning with no error.

Expected: rewrite_file_index rebuilds the index over the intended column's values for files written before a schema change.

Anything else?

Fix direction: resolve the read projection against the current schema by column name, keeping it positionally aligned with the writer projection (both collected in one pass, skipping columns absent from either schema). When the file schema is the current schema the two projections are identical and nothing changes.

Are you willing to submit a PR?

• I'm willing to submit a PR!