Skip to content

VARIANT: targeted extract pushdown for Parquet, unshredded read crash fix, and unshredded write support - #1

Open
fuziontech wants to merge 1 commit into
mainfrom
variant-extract-perf-posthog
Open

VARIANT: targeted extract pushdown for Parquet, unshredded read crash fix, and unshredded write support#1
fuziontech wants to merge 1 commit into
mainfrom
variant-extract-perf-posthog

Conversation

@fuziontech

Copy link
Copy Markdown
Member

Summary

Path queries on VARIANT columns (variant_extract(v, 'field'), v.field with a cast or filter) on unshredded and partially-shredded Parquet files were decoding every row's entire value twice: the reader converted the full binary value into a variant vector (re-encoding all of it via BuildVariant), then the extract decoded it again to pick out one field. The metadata dictionary was additionally decoded — with all key strings allocated — once per row.

This PR makes extract pushdown do a targeted decode (the same approach ClickHouse took in fuziontech/clickhouse#1), fixes a crash on unshredded schemas, and adds spec-compliant unshredded writes.

Benchmarks (3M rows, aarch64, best-of-5)

Query duckdb 1.5.5 ClickHouse (fuziontech/clickhouse#1) this branch
path groupby shredded 1740ms 562ms ~20ms
path filter shredded 1733ms 507ms ~20ms
path groupby unshredded 1537ms 590ms ~110ms
path filter unshredded 1534ms 514ms ~110ms
full scan shredded 10545ms ~4000ms ~2400ms (decode-only ~1000ms)
full scan unshredded 10015ms ~3600ms ~2600ms (decode-only ~1100ms)
pruned groupby/filter (fully-shredded) n/a ~71ms ~20ms

Changes

  • Targeted extract in VariantColumnReader (parquet_variant_iterator.cpp, variant_column_reader.cpp): when the extract is pushed down, navigate only the requested path per row — binary values via the object field table, shredded objects via the typed tree with binary-overlay fallback — and emit just the addressed value. No whole-value conversion, no double pass. BY_KEY path components are resolved to metadata dictionary field ids once per distinct dictionary (keyed by blob content), so per-row lookups are integer comparisons.
  • Metadata dictionary caching: ParquetVariantIterator::GetMetadata() decoded the key dictionary per row; it's now reused across rows sharing identical metadata bytes (writers emit identical metadata per row). The resolution cache is keyed by blob content, so per-row/per-row-group dictionary changes stay correct.
  • sorted_strings is verified, not trusted: the flag is unreliable across writers (DuckDB itself set it without always sorting until f129a86); sortedness is checked on the decoded dictionary once per distinct dictionary instead.
  • ParquetObjectIterator: skip std::sort when fields are already in lexicographic key order (the common, spec-compliant case).
  • Crash/wrong-result fix for unshredded (2-child) schemas: the pushdown index was silently dropped for 2-child (metadata+value) variant schemas, so the scan produced full or wrongly typed vectors — Vector::Shred can only be used on variant vectors for extracts with a pushed-down cast. The 2-child case now goes through the same pushdown path as shredded schemas.
  • Unshredded write support: SHREDDING {col: 'VARIANT'} now writes a spec-compliant unshredded (metadata+value) variant column instead of failing with Unsupported type "ANY"; the schema analyzer no longer overrides an explicitly requested layout. SHREDDING {col: 'NULL'} behavior is unchanged (analyzed partial shredding).

Testing

  • New test/parquet/variant/pushdown/variant_extract_unshredded.test: unshredded writes (schema assertion: no typed_value), pushed-down casts, filters, missing keys, nested paths, row-group-local metadata dictionaries.
  • test/parquet/variant/* (34 cases) and the full test/parquet/* suite (62 cases, 9164 assertions) pass.

Notes

  • Companion DuckLake work (enabling this pushdown through DuckLake scans: the function.statistics optimizer blocker, inlined-data extracts, variant stats propagation) lives in the DuckLake repo and will come as a separate PR there. Verified end-to-end through DuckLake: path group-by/filter run in ~60-80ms for the same dataset.

… fix, and unshredded write support

Path queries on VARIANT columns (variant_extract / v.field with cast or
filter) on unshredded and partially-shredded Parquet files were decoding
every row's entire value twice: the reader converted the full binary
value into a variant vector (re-encoding all of it), then the extract
decoded it again to pick one field. The metadata dictionary was decoded
(and its key strings allocated) once per row on top of that.

- VariantColumnReader now performs a targeted extract when the extract
  is pushed down: navigate only the requested path per row (binary
  values via the object field table, shredded objects via the typed
  tree with binary-overlay fallback) and emit just the addressed value.
  BY_KEY path components are resolved to metadata dictionary field ids
  once per distinct dictionary (content-keyed), so per-row lookups are
  integer comparisons. This mirrors what ClickHouse implemented in
  fuziontech/clickhouse#1. Path group-by/filter on 3M-row unshredded
  files drops from ~1.2s to ~0.11s; shredded path queries already ran
  in ~0.02s via the direct typed_value read.
- The decoded Variant metadata dictionary is cached per row-group
  chunk and reused across rows sharing identical metadata bytes
  (writers emit identical metadata per row), instead of re-decoding
  per row. Full scans of unshredded variant files drop from ~5.3s to
  ~2.6s (3M rows).
- ParquetObjectIterator skips std::sort when fields are already in
  lexicographic key order (the common, spec-compliant case).
- Fix a crash/wrong-result for pushdown extracts on unshredded
  (2-child metadata+value) variant schemas: the pushdown index was
  silently dropped, so the scan produced full (or wrongly typed)
  vectors - 'Vector::Shred can only be used on variant vectors' for
  extracts with a pushed-down cast. The 2-child case now goes through
  the same pushdown path as shredded schemas.
- Writing 'SHREDDING {col: ''VARIANT''}' now produces a spec-compliant
  unshredded (metadata+value) variant column instead of an internal
  error ('Unsupported type ANY'); the schema analyzer no longer
  overrides an explicitly requested layout. 'SHREDDING {col: ''NULL''}'
  behavior is unchanged (analyzed partial shredding).

Adds test/parquet/variant/pushdown/variant_extract_unshredded.test
covering unshredded writes, pushed-down casts, filters, missing keys,
nested paths, and row-group-local metadata dictionaries.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant