Conversation
| .withDescription( | ||
| "Whether to ignore the order of the buckets when reading data from an append-only table."); | ||
|
|
||
| public static final ConfigOption<Boolean> BUCKET_PER_PARTITION_COUNT_ENABLED = |
There was a problem hiding this comment.
ConfigOptionsDocsCompletenessITCase discovers this new CoreOptions entry, but bucket.per-partition-count-enabled is absent from the generated configuration docs. The focused test currently fails with: Option bucket.per-partition-count-enabled in class org.apache.paimon.CoreOptions is not documented. This also explains the failing Core and integrations jobs on both JDK 8 and JDK 11. Please regenerate the configuration documentation according to paimon-docs/README.md (or explicitly exclude the option if it is intentionally internal) so the required completeness check passes.
|
Closing this PR under the end-to-end production-value criterion for this review pass. The PR introduces a public I checked the full 23-file diff and ran the focused core suite after packaging the codegen loader: Please bring the core routing contract back together with at least one complete engine write/read path, explicit behavior for the other engines, generated option docs, and a rescale-then-write/read integration test. That would make the production behavior reviewable as a whole. |
Summary
This is a focused follow-up extracted from #9370, following the merged partition-layout scan prerequisite (#10052).
This PR makes partition bucket layout a core write contract:
PartitionBucketMappingto resolve the current bucket count for each partition;write(partition, bucket, totalBuckets, data);write(partition, bucket, data)path whenbucket.per-partition-count-enabled=trueon a partitioned table.The legacy path only carries a bucket id and cannot establish which bucket count was used for hashing. Rejecting it prevents a caller from silently routing a row with the table-level bucket count while the target partition uses a different layout.
Scope
This PR intentionally does not expose complete engine support for the option:
Verification
mvn -q -pl paimon-core -DskipTests compilemvn -q -pl paimon-core spotless:checkgit diff --checkThe focused restore test was updated to cover legacy-path rejection, stale-layout rejection, and a successful explicit partition-layout write.