Skip to content

Support regex filters for listing table paths - #96

Merged
osipovartem merged 1 commit into
embucket-sync-df55.0.0from
df55-listing-url-regex
Sep 16, 2026
Merged

osipovartem merged 1 commit into
embucket-sync-df55.0.0from
df55-listing-url-regex

Conversation

@osipovartem

Copy link
Copy Markdown
Collaborator

Summary

  • add an optional precompiled full object-path regex to ListingTableUrl
  • apply regex filtering to the object metadata stream before files are planned or opened
  • make regex and glob matchers mutually exclusive
  • dispatch once when constructing the listing stream so the existing default/glob hot path does not pay a per-object regex branch

This is intended for consumers that need regular-expression file selection over object-store prefixes, such as Snowflake-compatible staged COPY INTO ... PATTERN handling. It avoids an application-side LIST followed by one HEAD request per selected object.

Validation

  • cargo +1.95.0 test -p datafusion-datasource (180/180 unit tests, 8/8 doctests)
  • cargo +1.95.0 clippy -p datafusion-datasource --all-targets -- -D warnings
  • cargo +1.95.0 fmt --all --check

Tests cover prefix isolation, nested-path regex selection, glob replacement, and filtering through the real list_all_files metadata stream.

@osipovartem
osipovartem merged commit 391220d into embucket-sync-df55.0.0 Sep 16, 2026
16 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant