deps: [DO NOT MERGE] bump DataFusion 55.0 and Arrow/Parqet 59.1

[mbutrovich]

Blocked until the DataFusion 55.0 crates are published (this branch pins a git commit directly). In the meantime, it doubles as a testbed for DataFusion and upstream-Comet-related fixes that need validation against the new version before it's released, most notably apache/datafusion#24090 below.

Which issue does this PR close?

Closes #4865. Closes #4859.

Rationale for this change

Bumping DataFusion this far ahead of a release requires fixing API breakage and validating that fixes landing upstream in the meantime, like apache/datafusion#24090 (nested Parquet leaf pruning, for #4859), actually work end to end against Comet.

What changes are included in this PR?

• Bump DataFusion to a commit past feat: prune unread Parquet leaves when a nested column is cast to a narrower type datafusion#24090, Arrow/Parquet to 59.1.0, and iceberg-rust/opendal to compatible commits (native/Cargo.toml, native/Cargo.lock, native/core/Cargo.toml).

• Small API-compat fixes for the new arrow-rs/DataFusion versions: FixedSizeBinaryArray::from to try_from, ShufflePartitioner: Send + Sync to Send, and GroupsAccumulator trait changes (merge_batch drops _opt_filter, new required convert_to_state) across the affected agg/array functions, shuffle writers, and execution operators. No behavior change.

• Add spark_sqrt, a Comet-only scalar UDF (native/spark-expr/src/math_funcs/sqrt.rs, wired in comet_scalar_funcs.rs and spark/.../serde/math.scala). DataFusion's own sqrt errors on negative input; Spark's Sqrt returns NaN. Unrelated to the DataFusion bump, just riding this branch.

• Comet native Parquet scan reads full nested columns despite pruned Spark ReadSchema #4859: Comet's SparkPhysicalExprAdapter swaps DataFusion's CastExpr for Comet's own CometCastColumnExpr whenever a cast touches a Struct, List, or Map, to apply Spark-specific semantics via spark_parquet_convert. #24090's leaf-pruning in build_projection_read_plan only recognizes the literal CastExpr type, so once Comet swaps it, pruning never fires and the whole column gets read regardless of the DataFusion bump. Same failure shape as perf: unwrap identity casts in schema adapter to enable Parquet stats pruning #4730, which hit an analogous problem with row-group pruning.

  Fix: add is_pure_structural_narrowing (native/core/src/parquet/schema_adapter.rs). DataFusion's own datafusion_common::nested_struct::cast_column, which ColumnarValue::cast_to runs at execution time, already matches struct fields by name, drops extra source fields, and null-fills missing target fields, exactly the shape a pruned nested ReadSchema needs. For a pure structural narrowing (fields dropped only, no leaf-type change, no Map, no Parquet field-id matching, no case-insensitive-only match), that's identical to what Comet's own path computes. When the predicate holds, replace_with_spark_cast leaves DataFusion's CastExpr in place instead of swapping in CometCastColumnExpr, so #24090's pruning sees it.

  Built as an allow list (only Struct/List/LargeList recurse, every leaf must be an exact type match), not a deny list, since a deny list fails open: a future Comet special case not mirrored there would silently produce wrong results instead of just missing an optimization. Map narrowing is still uncovered (nested_struct::cast_column has no Map arm), needs a Comet-side accommodation or an upstream extension.

• Hoist the two regression tests from fix: [DO NOT MERGE] Validate nested Parquet leaf pruning against datafusion#23391 #4866 (array of struct and array of struct of struct leaf pruning) into CometNativeReaderSuite.

• CometTPCHQuerySuite: raise MEMORY_OFFHEAP_SIZE to 4g, unrelated CI tuning found necessary while running this branch's suites.

How are these changes tested?

• 17 new Rust unit tests in schema_adapter.rs covering is_pure_structural_narrowing (allowed: struct and list-of-struct-of-struct narrowing; denied: case-insensitive-only match, missing target field, zero field overlap, Parquet field-id matching at top and nested levels, Map, Dictionary, timestamp NTZ to LTZ relabeling, leaf type change), plus two end-to-end tests through the real PhysicalExprAdapter::rewrite pipeline and two tests comparing Comet's spark_parquet_convert against the actual DataFusion functions this relies on for byte-identical output.
• The two #4859 regression tests in CometNativeReaderSuite pass; full CometNativeReaderSuite passes too, to catch any regression in other array/struct/map shapes from swapping fewer casts.
• Existing test suites cover the API-compat fixes and spark_sqrt; full CI run pending on this PR.

[andygrove]

I had a similar effort here - #4888

Happy to close mine though