VARIANT: targeted extract pushdown for Parquet, unshredded read crash fix, and unshredded write support - #1
Open
fuziontech wants to merge 1 commit into
Open
Conversation
… fix, and unshredded write support Path queries on VARIANT columns (variant_extract / v.field with cast or filter) on unshredded and partially-shredded Parquet files were decoding every row's entire value twice: the reader converted the full binary value into a variant vector (re-encoding all of it), then the extract decoded it again to pick one field. The metadata dictionary was decoded (and its key strings allocated) once per row on top of that. - VariantColumnReader now performs a targeted extract when the extract is pushed down: navigate only the requested path per row (binary values via the object field table, shredded objects via the typed tree with binary-overlay fallback) and emit just the addressed value. BY_KEY path components are resolved to metadata dictionary field ids once per distinct dictionary (content-keyed), so per-row lookups are integer comparisons. This mirrors what ClickHouse implemented in fuziontech/clickhouse#1. Path group-by/filter on 3M-row unshredded files drops from ~1.2s to ~0.11s; shredded path queries already ran in ~0.02s via the direct typed_value read. - The decoded Variant metadata dictionary is cached per row-group chunk and reused across rows sharing identical metadata bytes (writers emit identical metadata per row), instead of re-decoding per row. Full scans of unshredded variant files drop from ~5.3s to ~2.6s (3M rows). - ParquetObjectIterator skips std::sort when fields are already in lexicographic key order (the common, spec-compliant case). - Fix a crash/wrong-result for pushdown extracts on unshredded (2-child metadata+value) variant schemas: the pushdown index was silently dropped, so the scan produced full (or wrongly typed) vectors - 'Vector::Shred can only be used on variant vectors' for extracts with a pushed-down cast. The 2-child case now goes through the same pushdown path as shredded schemas. - Writing 'SHREDDING {col: ''VARIANT''}' now produces a spec-compliant unshredded (metadata+value) variant column instead of an internal error ('Unsupported type ANY'); the schema analyzer no longer overrides an explicitly requested layout. 'SHREDDING {col: ''NULL''}' behavior is unchanged (analyzed partial shredding). Adds test/parquet/variant/pushdown/variant_extract_unshredded.test covering unshredded writes, pushed-down casts, filters, missing keys, nested paths, and row-group-local metadata dictionaries.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Path queries on VARIANT columns (
variant_extract(v, 'field'),v.fieldwith a cast or filter) on unshredded and partially-shredded Parquet files were decoding every row's entire value twice: the reader converted the full binary value into a variant vector (re-encoding all of it viaBuildVariant), then the extract decoded it again to pick out one field. The metadata dictionary was additionally decoded — with all key strings allocated — once per row.This PR makes extract pushdown do a targeted decode (the same approach ClickHouse took in fuziontech/clickhouse#1), fixes a crash on unshredded schemas, and adds spec-compliant unshredded writes.
Benchmarks (3M rows, aarch64, best-of-5)
Changes
VariantColumnReader(parquet_variant_iterator.cpp,variant_column_reader.cpp): when the extract is pushed down, navigate only the requested path per row — binary values via the object field table, shredded objects via the typed tree with binary-overlay fallback — and emit just the addressed value. No whole-value conversion, no double pass.BY_KEYpath components are resolved to metadata dictionary field ids once per distinct dictionary (keyed by blob content), so per-row lookups are integer comparisons.ParquetVariantIterator::GetMetadata()decoded the key dictionary per row; it's now reused across rows sharing identical metadata bytes (writers emit identical metadata per row). The resolution cache is keyed by blob content, so per-row/per-row-group dictionary changes stay correct.sorted_stringsis verified, not trusted: the flag is unreliable across writers (DuckDB itself set it without always sorting until f129a86); sortedness is checked on the decoded dictionary once per distinct dictionary instead.ParquetObjectIterator: skipstd::sortwhen fields are already in lexicographic key order (the common, spec-compliant case).metadata+value) variant schemas, so the scan produced full or wrongly typed vectors —Vector::Shred can only be used on variant vectorsfor extracts with a pushed-down cast. The 2-child case now goes through the same pushdown path as shredded schemas.SHREDDING {col: 'VARIANT'}now writes a spec-compliant unshredded (metadata+value) variant column instead of failing withUnsupported type "ANY"; the schema analyzer no longer overrides an explicitly requested layout.SHREDDING {col: 'NULL'}behavior is unchanged (analyzed partial shredding).Testing
test/parquet/variant/pushdown/variant_extract_unshredded.test: unshredded writes (schema assertion: notyped_value), pushed-down casts, filters, missing keys, nested paths, row-group-local metadata dictionaries.test/parquet/variant/*(34 cases) and the fulltest/parquet/*suite (62 cases, 9164 assertions) pass.Notes
function.statisticsoptimizer blocker, inlined-data extracts, variant stats propagation) lives in the DuckLake repo and will come as a separate PR there. Verified end-to-end through DuckLake: path group-by/filter run in ~60-80ms for the same dataset.