[python] Apply filters when reading system tables - #10164
jackylee-ch wants to merge 1 commit into
Conversation
| arrow_table = split.arrow_table() | ||
| # Filter on the full schema before projection so a predicate may | ||
| # reference columns that are not part of the projection. | ||
| if arrow_predicate is not None: |
There was a problem hiding this comment.
predicate_supports_arrow_filter only checks the predicate method, so it accepts comparisons for every field type. System tables expose array fields such as $files.write_cols and $table_indexes.dv_ranges , but PyArrow has no equality kernel for list values. For example, builder.equal("write_cols", ["a"]) reaches this line and raises ArrowNotImplementedError: Function 'equal' has no kernel matching input types (list<item: string>, list<item: string>) . Please make the support check field-type/method aware (or provide the existing row-wise fallback) and reject unsupported combinations with the documented NotImplementedError . A regression test using an array-valued system-table column would cover this
| if self.predicate is not None: | ||
| from pypaimon.read.push_down_utils import predicate_supports_arrow_filter | ||
|
|
||
| if self.predicate is not None and not predicate_supports_arrow_filter( |
There was a problem hiding this comment.
Please preserve the existing rejection path for unsupported predicate objects before calling this helper. SystemReadPipelineTest.test_filter_is_not_supported_yet passes object() and expects the public NotImplementedError , but predicate_supports_arrow_filter immediately dereferences .method , so this PR changes the result to an internal AttributeError . The focused system-table suite currently fails on this test. An isinstance(self.predicate, Predicate) guard (or making the helper return False for unknown objects) would retain the previous error contract.
76a5181 to
3902dd4
Compare
SystemReadBuilder.with_filter stores and forwards a predicate, but SystemTableRead._materialise raised NotImplementedError for any predicate, so with_filter on a system table (e.g. db.t$snapshots, db.t$files) always failed. Layers that push filters down to pypaimon (Daft / Ray) hit this on every system table. Apply the predicate to the materialised PyArrow table for the methods that convert to a safe Arrow row filter (comparisons, null checks, is-in), reusing predicate_supports_arrow_filter from the data-read path. The filter runs before projection so a predicate may reference columns that are not projected. String-match predicates (starts_with / ends_with / contains / like) are not safe as final Arrow filters and still raise a clear error.
3902dd4 to
d8b5952
Compare
|
Thanks @Akash3121 — both addressed (rebased onto master):
|
2 participants
Purpose
SystemReadBuilder.with_filterstores and forwards a predicate, butSystemTableRead._materialiseraisedNotImplementedErrorfor any predicate. Sowith_filter(...)on a system table (db.t$snapshots,db.t$files, ...) always failed, and layers that push filters down to pypaimon (Daft / Ray) hit this on every system table.This applies the predicate to the materialised PyArrow table for the methods that convert to a safe Arrow row filter (comparisons, null checks,
is_in), reusingpredicate_supports_arrow_filterfrom the data-read path. The filter runs before projection, so a predicate may reference columns that are not projected. String-match predicates (starts_with/ends_with/contains/like) are not safe as final Arrow row filters and still raise a clear error.Tests
pypaimon/tests/system/system_table_test.py: addedSystemTableFilterTestcovering equal / greater-than / is-in filtering, filtering on a column that is not in the projection, and the string-match method still raisingNotImplementedError.API and Format
No public API or format change;
with_filterwas already exposed on the system-table read builder.Written with Claude Code; verification is mine.