[iceberg-datafusion] Cache eager scan planning across equivalent scans

[toutane]

Is your feature request related to a problem or challenge?

Eager scan planning introduced by #2671 calls plan_files() during TableProvider::scan(). This includes reading manifests, binding predicates, resolving the snapshot schema, and associating applicable delete files with each FileScanTask.

If DataFusion requests the same scan multiple times during optimization, this work is repeated. The resulting eager plan is retained only by the returned IcebergTableScan and cannot be reused by a future equivalent scan() call.

This limitation was originally discussed in #2298: #2298 (comment)

We should keep in mind that there are two provider types:

• IcebergStaticTableProvider uses a fixed table and snapshot, making equivalent scans suitable for reuse.
• IcebergTableProvider must continue loading fresh table metadata so that newly committed snapshots remain visible.

Describe the solution you'd like

Reuse eager scan planning results across equivalent TableProvider::scan() calls.

A cached result must only be reused when every input affecting file planning is equivalent. This includes at least:

• the concrete snapshot ID;
• the projection;
• the pushed-down predicate.

For the catalog-backed IcebergTableProvider, the table must still be reloaded before a cached result can be reused. The concrete snapshot ID resolved from the refreshed table must participate in cache lookup so that a new snapshot naturally causes a cache miss.

Two caching strategies should be considered:

Option 1: Cache the complete EagerScanPlan

Cache the tasks after they have been grouped into DataFusion output partitions.

Option 2: Cache the planned tasks before grouping

Cache the result of Iceberg file planning before assigning the FileScanTasks to DataFusion output partitions. Perform the relatively inexpensive grouping after cache lookup.

In this design, target_partitions does not need to be part of the cache key.

Willingness to contribute

I would be willing to contribute to this feature with guidance from the Iceberg Rust community