Parquet dictionary encoding fallback is sub-optimal, may violate writer parameters

[mzabaluev]

The dict_fallback method of GenericColumnWriter writes the dictionary page to the output even though the conditions for the fallback are reached, meaning that the dictionary encoding is unsatisfactory to encode the entire column chunk. This presents a few minor problems:

1. In the current should_dict_fallback logic, the dictionary has met or exceeded its page size limit as configured in the column properties. Oversized dictionary pages, though not violating any format constraints, may be surprising to the user.
2. The data pages of the column chunk are then encoded piecemeal using first the dictionary, then a fallback encoding, which is again legal but weird. More importantly, a larger than expected dictionary may arise from high cardinality of the values, so encoding all data pages in fallback may result in a more compact encoding.
3. More fallback strategies may be added in the future, as proposed in Parquet dictionary fallback heuristics #9699 and implemented in feat(parquet): dictionary fallback heuristic based on compression efficiency #9700. In such cases, the dictionary encoding is decided to be inefficient based on the size of a partial encoding, so it does not make sense to write out the first inefficiently encoded pages and then continue on the better encoding.

For comparison, the FallbackValuesWriter implementation in parquet-java extracts all values from the dictionary encoder to be re-encoded by the fallback encoder.

[friendlymatthew]

I've run into this well. At work, we sample the first row group and apply some simple heuristics (like distinct counts, total value offset size) to decide on encodings, then pass those into WriterPropertiesBuilder. That lets us avoid dictionary rollback when we're reasonably confident it won't help

I don't think there's anything custom about what we're doing that couldn't be supported upstream

[etseidl]

Oversized dictionary pages, though not violating any format constraints, may be surprising to the user.

I believe most size limits indicate they are a best effort, not a hard cutoff.

The data pages of the column chunk are then encoded piecemeal using first the dictionary, then a fallback encoding, which is again legal but weird.

Actually this is quite common. Even parquet-java will do this if the dictionary only becomes too large after writing several pages.