Describe the bug
arrays_overlap over array<array<double>> panics in native code when both sides of the comparison are expression-constructed nested lists rather than plain columns.
org.apache.comet.CometNativeException: native panic: index out of bounds: the len is 2 but the index is 2
Comet native panic: panicked at arrow-array-58.4.0/src/array/list_array.rs:352:19
list_array.rs:352 is GenericListArray::value, so something in arrays_overlap_list_generic (native/spark-expr/src/array_funcs/arrays_overlap.rs) indexes past the end of one of its two list arrays.
Steps to reproduce
As a Comet SQL file test:
statement
CREATE TABLE t_panic(a array<array<double>>) USING parquet
statement
INSERT INTO t_panic VALUES
(array(array(-0.0))),
(array(array(0.0))),
(array(array(1.0, -0.0))),
(array(array(double('NaN')))),
(array(array(1.0))),
(array(array(-0.0), cast(NULL as array<double>))),
(array(cast(NULL as array<double>)))
query
SELECT arrays_overlap(array(array(a[0][0])), array(array(cast(0.0 as double)))) FROM t_panicReproduced on apache/main at f262b13d2, Spark 4.1 profile, JDK 17, macOS.
What I could and could not narrow down
The panic needs all seven rows. None of these reproduce it on their own:
- any single row from the table, including the two rows carrying a
NULL inner list
- a two-row table combining a multi-element inner list with a
NULL inner list
- the
int equivalent of the same query
arrays_overlap(a, array(array(cast(0.0 as double)))), a plain column against a constantarrays_overlap(array(a[0]), array(array(0))), only the outer list constructed
The failing task index moves with the number of rows, so this looks like a batch or partition boundary interacting with the per-row left.value(i) / right.value(i) slicing in arrays_overlap_list_generic, rather than anything about a specific row's contents. Worth checking whether the two sides can end up with different len() when one of them arrives as a broadcast scalar and the other is a per-row constructed array.
Additional context
Found while reviewing #5235, which changes the same function. The panic reproduces identically with and without that PR applied, so it is independent of it.
Describe the bug
arrays_overlapoverarray<array<double>>panics in native code when both sides of the comparison are expression-constructed nested lists rather than plain columns.list_array.rs:352isGenericListArray::value, so something inarrays_overlap_list_generic(native/spark-expr/src/array_funcs/arrays_overlap.rs) indexes past the end of one of its two list arrays.Steps to reproduce
As a Comet SQL file test:
Reproduced on
apache/mainatf262b13d2, Spark 4.1 profile, JDK 17, macOS.What I could and could not narrow down
The panic needs all seven rows. None of these reproduce it on their own:
NULLinner listNULLinner listintequivalent of the same queryarrays_overlap(a, array(array(cast(0.0 as double)))), a plain column against a constantarrays_overlap(array(a[0]), array(array(0))), only the outer list constructedThe failing task index moves with the number of rows, so this looks like a batch or partition boundary interacting with the per-row
left.value(i)/right.value(i)slicing inarrays_overlap_list_generic, rather than anything about a specific row's contents. Worth checking whether the two sides can end up with differentlen()when one of them arrives as a broadcast scalar and the other is a per-row constructed array.Additional context
Found while reviewing #5235, which changes the same function. The panic reproduces identically with and without that PR applied, so it is independent of it.