From c785bc93f24e3b984fd759ebc74cbfa4bbe22a6a Mon Sep 17 00:00:00 2001 From: Prateek Gaur Date: Mon, 9 Mar 2026 21:47:39 +0000 Subject: [PATCH 01/15] Add ALP (Adaptive Lossless floating-Point) encoding specification Add the encoding specification for ALP (encoding value 10) to Encodings.md. ALP compresses FLOAT and DOUBLE columns by converting values to integers via decimal scaling, then applying Frame of Reference encoding and bit-packing. Values that cannot be losslessly round-tripped are stored as exceptions. The spec covers: - Page layout: 7-byte header, offset array, compressed vectors - Vector format: AlpInfo, ForInfo, packed values, exception data - Encoding math: two-step multiplication for cross-language consistency - Parameter selection, exception detection, and decoding steps Based on the paper "ALP: Adaptive Lossless floating-Point Compression" (Afroozeh and Boncz, SIGMOD 2024). Wire format matches the C++ Arrow and Java parquet-java implementations. --- Encodings.md | 519 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 519 insertions(+) diff --git a/Encodings.md b/Encodings.md index 1c766fb5a..6f0758e6a 100644 --- a/Encodings.md +++ b/Encodings.md @@ -391,3 +391,522 @@ After applying the transformation, the data has the following representation: ``` Bytes AA 00 A3 BB 11 B4 CC 22 C5 DD 33 D6 ``` + + +### Adaptive Lossless floating-Point: (ALP = 10) + +Supported Types: FLOAT, DOUBLE + +This encoding is adapted from the paper +["ALP: Adaptive Lossless floating-Point Compression"](https://dl.acm.org/doi/10.1145/3626717) +by Afroozeh and Boncz (SIGMOD 2024). + +ALP works by converting floating-point values to integers using decimal scaling, +then applying Frame of Reference (FOR) encoding and bit-packing. Values that +cannot be losslessly converted are stored as exceptions. The encoding achieves +high compression for decimal-like floating-point data (e.g., monetary values, +sensor readings) while remaining fully lossless. + +#### Overview + +ALP encoding consists of a page-level header followed by an offset array and one +or more encoded vectors (batches of values). Each vector contains up to +`vector_size` elements (default 1024). + +``` ++-------------+-----------------------------+--------------------------------------+ +| Header | Offset Array | Vector Data | +| (7 bytes) | (num_vectors * 4 bytes) | (variable) | ++-------------+------+------+-----+---------+----------+----------+-----+----------+ +| Page Header | off0 | off1 | ... | off N-1 | Vector 0 | Vector 1 | ... | Vec N-1 | +| (7 bytes) | (4B) | (4B) | | (4B) |(variable)|(variable)| |(variable)| ++-------------+------+------+-----+---------+----------+----------+-----+----------+ +``` + +The compression pipeline for each vector is: + +``` + Input: float/double array + | + v + +----------------------------------------------------------+ + | 1. SAMPLING & PRESET GENERATION | + | Sample vectors from column chunk | + | Try all (exponent, factor) combinations | + | Select best k combinations for preset | + +----------------------------------------------------------+ + | + v + +----------------------------------------------------------+ + | 2. DECIMAL ENCODING | + | encoded[i] = round(value[i] * 10^e * 10^(-f)) | + | Detect exceptions where decode(encode(v)) != v | + +----------------------------------------------------------+ + | + v + +----------------------------------------------------------+ + | 3. FRAME OF REFERENCE (FOR) | + | min_val = min(encoded[]) | + | delta[i] = encoded[i] - min_val | + +----------------------------------------------------------+ + | + v + +----------------------------------------------------------+ + | 4. BIT PACKING | + | bit_width = ceil(log2(max_delta + 1)) | + | Pack each delta into bit_width bits | + +----------------------------------------------------------+ + | + v + Output: Serialized vector bytes +``` + +#### Page Layout + +##### Header (7 bytes) + +All multi-byte values are little-endian. + +``` + Byte: 0 1 2 3 4 5 6 + +----------------+---------------+--------------+----+----+----+----+ + | compression | integer | log_vector | num_elements | + | _mode | _encoding | _size | (int32 LE) | + +----------------+---------------+--------------+----+----+----+----+ +``` + +| Offset | Field | Size | Type | Description | +|--------|-------|------|------|-------------| +| 0 | compression_mode | 1 byte | uint8 | Compression mode (must be 0 = ALP) | +| 1 | integer_encoding | 1 byte | uint8 | Integer encoding (must be 0 = FOR + bit-packing) | +| 2 | log_vector_size | 1 byte | uint8 | log2(vector\_size). Must be in \[3, 15\]. Default: 10 (vector size 1024) | +| 3 | num_elements | 4 bytes | int32 | Total number of floating-point values in the page | + +The number of vectors is `ceil(num_elements / vector_size)`. The last vector may +contain fewer than `vector_size` elements. + +**Note:** The number of elements per vector and the packed data size are NOT stored +in the header. They are derived: +* Elements per vector: `vector_size` for all vectors except the last, which may be smaller. +* Packed data size: `ceil(num_elements_in_vector * bit_width / 8)`. + +##### Offset Array + +Immediately following the header is an array of `num_vectors` little-endian uint32 +values. Each offset gives the byte position of the corresponding vector's data, +measured from the start of the offset array itself. + +The first offset equals `num_vectors * 4` (pointing just past the offset array). +Each subsequent offset equals the previous offset plus the stored size of the +previous vector. + +##### Vector Format + +Each vector is self-describing and contains the encoding parameters, FOR metadata, +bit-packed encoded values, and exception data. + +``` ++-------------------+-----------------+-------------------+---------------------+-------------------+ +| AlpInfo | ForInfo | PackedValues | ExceptionPositions | ExceptionValues | +| (4 bytes) | (5B or 9B) | (variable) | (variable) | (variable) | ++-------------------+-----------------+-------------------+---------------------+-------------------+ +``` + +Vector header sizes: +| Type | AlpInfo | ForInfo | Total Header | +|--------|---------|---------|--------------| +| FLOAT | 4 bytes | 5 bytes | 9 bytes | +| DOUBLE | 4 bytes | 9 bytes | 13 bytes | + +Data section sizes: +| Section | Size Formula | Description | +|---------------------|-----------------------------|------------------------------| +| PackedValues | ceil(N * bit\_width / 8) | Bit-packed delta values | +| ExceptionPositions | num\_exceptions * 2 bytes | uint16 indices of exceptions | +| ExceptionValues | num\_exceptions * sizeof(T) | Original float/double values | + +###### AlpInfo (4 bytes, both types) + +``` + Byte: 0 1 2 3 + +----------+----------+---------+---------+ + | exponent | factor | num_exceptions | + | (uint8) | (uint8) | (uint16 LE) | + +----------+----------+---------+---------+ +``` + +| Offset | Field | Size | Type | Description | +|--------|-------|------|------|-------------| +| 0 | exponent | 1 byte | uint8 | Power-of-10 exponent *e*. Range: \[0, 10\] for FLOAT, \[0, 18\] for DOUBLE. | +| 1 | factor | 1 byte | uint8 | Power-of-10 factor *f*. Range: \[0, *e*\]. | +| 2 | num_exceptions | 2 bytes | uint16 | Number of exception values in this vector. | + +###### ForInfo for FLOAT (5 bytes) + +``` + Byte: 0 1 2 3 4 + +----+----+----+----+-----------+ + | frame_of_reference | bit_width | + | (int32 LE) | (uint8) | + +----+----+----+----+-----------+ +``` + +| Offset | Field | Size | Type | Description | +|--------|-------|------|------|-------------| +| 0 | frame_of_reference | 4 bytes | int32 | Minimum encoded integer in the vector | +| 4 | bit_width | 1 byte | uint8 | Bits per packed value. Range: \[0, 32\]. | + +###### ForInfo for DOUBLE (9 bytes) + +``` + Byte: 0 1 2 3 4 5 6 7 8 + +----+----+----+----+----+----+----+----+-----------+ + | frame_of_reference | bit_width | + | (int64 LE) | (uint8) | + +----+----+----+----+----+----+----+----+-----------+ +``` + +| Offset | Field | Size | Type | Description | +|--------|-------|------|------|-------------| +| 0 | frame_of_reference | 8 bytes | int64 | Minimum encoded long in the vector | +| 8 | bit_width | 1 byte | uint8 | Bits per packed value. Range: \[0, 64\]. | + +###### PackedValues + +The FOR-encoded deltas, bit-packed into `ceil(num_elements_in_vector * bit_width / 8)` bytes. +Values are packed from the least significant bit of each byte to the most significant bit, +in groups of 8 values, using the same bit-packing order as the +[RLE/Bit-Packing Hybrid](#RLE) encoding. + +If `bit_width` is 0, no bytes are stored (all deltas are zero, meaning all encoded +integers are equal to `frame_of_reference`). + +###### ExceptionPositions + +An array of `num_exceptions` little-endian uint16 values, each giving +the 0-based index within the vector of an exception value. + +###### ExceptionValues + +An array of `num_exceptions` values in the original floating-point type +(4 bytes little-endian IEEE 754 for FLOAT, 8 bytes for DOUBLE), stored in +the same order as the corresponding positions. + +#### Encoding + +##### Encoding Formula + +``` ++-------------------------------------------------------------------+ +| | +| encoded = round( value * 10^e * 10^(-f) ) | +| | +| decoded = encoded * 10^f * 10^(-e) | +| | ++-------------------------------------------------------------------+ +``` + +The encoding uses two separate multiplications (not a single multiplication by +`10^(e-f)`, and not division) to ensure that implementations produce identical +floating-point rounding across languages. The powers of 10 MUST be stored as +precomputed floating-point constants (i.e., literal values like `1e-3f`), not +computed at runtime. + +##### Fast Rounding + +The rounding function uses a "magic number" technique for branchless rounding: + +| Type | Magic Number | Formula | +|--------|-----------------------------------|----------------------------------| +| FLOAT | 2^22 + 2^23 = 12,582,912 | `(int)((value + magic) - magic)` | +| DOUBLE | 2^51 + 2^52 = 6,755,399,441,055,744 | `(long)((value + magic) - magic)` | + +For negative values, the signs are reversed: `(int)((value - magic) + magic)`. + +##### Parameter Selection + +The encoder selects the (exponent, factor) pair that minimizes exceptions. +Valid combinations satisfy 0 ≤ factor ≤ exponent: + +| Type | Max Exponent | Total Combinations | +|--------|--------------|--------------------| +| FLOAT | 10 | 66 | +| DOUBLE | 18 | 190 | + +To avoid the cost of exhaustive search on every vector, implementations +SHOULD use sampling to select up to 5 candidate (exponent, factor) +combinations (the "encoding preset") at the start of each column chunk. +Each vector then searches only those 5 candidates. + +Sampling parameters: + +| Parameter | Value | Description | +|----------------------|-------|-------------------------------------| +| Vector Size | 1024 | Elements compressed as a unit | +| Sample Size | 256 | Values sampled per vector | +| Max Combinations | 5 | Best (e,f) pairs kept in preset | +| Sample Vectors | 8 | Vectors sampled per row group | + +##### Exception Detection + +A value becomes an exception if any of the following is true: + +| Condition | Example | Reason | +|--------------------|----------------------------|----------------------------------| +| NaN | `NaN` | Cannot convert to integer | +| Infinity | `+Inf`, `-Inf` | Cannot convert to integer | +| Negative zero | `-0.0` | Would become `+0.0` after encoding | +| Out of range | value * 10^e > INT32\_MAX | Exceeds target integer limits | +| Round-trip failure | `0.333...` with e=1, f=0 | `decode(encode(v)) != v` | + +Exception values at positions in the vector are replaced with a placeholder +(the encoded integer of the first non-exception value, or 0 if all values +are exceptions) before FOR encoding. This keeps the FOR range tight. + +##### Frame of Reference and Bit-Packing + +After decimal encoding and exception substitution: + +``` ++---------------------------------------------------------------------+ +| Encoded: [ 123, 456, 789, 12 ] | +| | +| min_val = 12 (stored as frame_of_reference) | +| | +| Deltas: [ 111, 444, 777, 0 ] <-- all non-negative | ++---------------------------------------------------------------------+ +``` + +| Step | Formula | Example | +|------------------------|---------------------------------------|-----------------------------| +| 1. Find min | min\_val = min(encoded\[\]) | 12 | +| 2. Compute deltas | delta\[i\] = encoded\[i\] - min\_val | \[111, 444, 777, 0\] | +| 3. Calculate bit width | bit\_width = ceil(log2(max\_delta+1)) | ceil(log2(778)) = 10 | +| 4. Pack values | Each value uses bit\_width bits | 4 * 10 = 40 bits = 5 bytes | + +Special case: If all values are identical, bit\_width = 0 and no packed data is stored. + +#### Decoding + +``` + Input: Serialized vector bytes + | + v + +----------------------------------------------------------+ + | 1. BIT UNPACKING | + | Unpack num_elements values at bit_width bits each | + +----------------------------------------------------------+ + | + v + +----------------------------------------------------------+ + | 2. REVERSE FOR | + | encoded[i] = delta[i] + frame_of_reference | + +----------------------------------------------------------+ + | + v + +----------------------------------------------------------+ + | 3. DECIMAL DECODING | + | value[i] = encoded[i] * 10^factor * 10^(-exponent) | + +----------------------------------------------------------+ + | + v + +----------------------------------------------------------+ + | 4. PATCH EXCEPTIONS | + | value[pos[j]] = exception_values[j] | + +----------------------------------------------------------+ + | + v + Output: Original float/double array +``` + +For each vector: + +1. Read AlpInfo and ForInfo from the vector header. +2. Unpack `bit_width`-bit integers from PackedValues. +3. Add `frame_of_reference` to each unpacked integer. +4. Decode: multiply each integer by `10^factor` then by `10^(-exponent)`. +5. Patch exceptions: for each (position, value) in the exception arrays, + overwrite the decoded output at that position with the stored value. + +#### Example 1: Simple Decimal Values + +**Input:** `float values[4] = { 1.23, 4.56, 7.89, 0.12 }` + +**Step 1: Find Best Exponent/Factor** + +Testing (exponent=2, factor=0) means multiply by 10^2 = 100: + +| Value | value * 100 | Rounded | Verify: rounded * 1.0 * 0.01 | Match? | +|-------|-------------|---------|-------------------------------|--------| +| 1.23 | 123.0 | 123 | 1.23 | Yes | +| 4.56 | 456.0 | 456 | 4.56 | Yes | +| 7.89 | 789.0 | 789 | 7.89 | Yes | +| 0.12 | 12.0 | 12 | 0.12 | Yes | + +All values round-trip correctly -- no exceptions. + +**Step 2: Frame of Reference** + +| Encoded | min = 12 | Delta (encoded - min) | +|---------|----------|-----------------------| +| 123 | - | 111 | +| 456 | - | 444 | +| 789 | - | 777 | +| 12 | - | 0 | + +**Step 3: Bit Packing** + +max\_delta = 777, bit\_width = ceil(log2(778)) = 10 bits, +packed\_size = ceil(4 * 10 / 8) = 5 bytes + +**Serialized Vector:** + +| Section | Content | Size | +|---------------------|----------------------------------------|----------| +| AlpInfo | e=2, f=0, num\_exceptions=0 | 4 bytes | +| ForInfo | frame\_of\_reference=12, bit\_width=10 | 5 bytes | +| PackedValues | \[111, 444, 777, 0\] at 10 bits each | 5 bytes | +| ExceptionPositions | (none) | 0 bytes | +| ExceptionValues | (none) | 0 bytes | +| **Total** | | **14 bytes** | + +Compared to PLAIN encoding (4 * 4 = 16 bytes). With 1024 values, the 9-byte +vector header becomes negligible and compression ratios of 2-8x are typical. + +#### Example 2: Values with Exceptions + +**Input:** `float values[4] = { 1.5, NaN, 2.5, 0.333... }` + +**Step 1: Decimal Encoding with (e=1, f=0)** + +Multiply by 10^1 = 10: + +| Index | Value | value * 10 | Rounded | Verify | Exception? | +|-------|----------|------------|---------|----------------|------------| +| 0 | 1.5 | 15.0 | 15 | 1.5 = 1.5 | No | +| 1 | NaN | - | - | - | Yes (NaN) | +| 2 | 2.5 | 25.0 | 25 | 2.5 = 2.5 | No | +| 3 | 0.333... | 3.333... | 3 | 0.3 != 0.333...| Yes (round-trip) | + +**Step 2: Handle Exceptions** + +Exception positions: \[1, 3\] +Exception values: \[NaN, 0.333...\] +Placeholder: 15 (first non-exception encoded value) +Encoded with placeholders: \[15, 15, 25, 15\] + +**Step 3: Frame of Reference** + +| Encoded | min = 15 | Delta | +|------------------|----------|-------| +| 15 | - | 0 | +| 15 (placeholder) | - | 0 | +| 25 | - | 10 | +| 15 (placeholder) | - | 0 | + +**Step 4: Bit Packing** + +max\_delta = 10, bit\_width = ceil(log2(11)) = 4 bits, +packed\_size = ceil(4 * 4 / 8) = 2 bytes + +**Serialized Vector:** + +| Section | Content | Size | +|---------------------|----------------------------------------|----------| +| AlpInfo | e=1, f=0, num\_exceptions=2 | 4 bytes | +| ForInfo | frame\_of\_reference=15, bit\_width=4 | 5 bytes | +| PackedValues | \[0, 0, 10, 0\] at 4 bits each | 2 bytes | +| ExceptionPositions | \[1, 3\] | 4 bytes | +| ExceptionValues | \[NaN, 0.333...\] | 8 bytes | +| **Total** | | **23 bytes** | + +#### Example 3: Monetary Data (1024 values) + +1024 price values ranging from $0.01 to $999.99 (e.g., product prices). + +Optimal encoding: (exponent=2, factor=0) + +| Metric | Value | Calculation | +|---------------|-------------|--------------------------------------| +| Exponent | 2 | Multiply by 100 for 2 decimal places | +| Factor | 0 | No additional scaling needed | +| Encoded range | 1 to 99,999 | $0.01 -> 1, $999.99 -> 99999 | +| FOR min | 1 | Assuming $0.01 is present | +| Delta range | 0 to 99,998 | After FOR subtraction | +| Bit width | 17 | ceil(log2(99999)) = 17 bits | +| Packed size | 2,176 bytes | ceil(1024 * 17 / 8) | + +**Size Comparison:** + +| Encoding | Size | Ratio | +|---------------|--------------|----------------------| +| PLAIN (float) | 4,096 bytes | 1.0x | +| ALP | ~2,185 bytes | 0.53x (47% smaller) | + +#### Characteristics + +| Property | Description | +|----------------|----------------------------------------------------------------------------------------| +| Lossless | All original floating-point values are perfectly recoverable, including NaN, Inf, -0.0 | +| Adaptive | Exponent/factor selection adapts per vector based on data characteristics | +| Vectorized | Fixed-size vectors enable SIMD-optimized bit packing/unpacking | +| Exception-safe | Values that don't fit decimal model are stored separately | + +**Best use cases:** + +* Monetary/financial data (prices, transactions) +* Sensor readings with fixed precision +* Scientific measurements with limited decimal places +* GPS coordinates and geographic data +* Normalized scores and percentages + +**Worst case scenarios:** + +* Random floating-point values (high exception rate) +* High-precision scientific data (many decimal places) +* Data with many special values (NaN, Inf) +* Very small datasets (header overhead dominates) + +**Comparison with other encodings:** + +| Encoding | Type Support | Compression | Best For | +|---------------------|--------------|-------------|---------------------| +| PLAIN | All | None | General purpose | +| BYTE\_STREAM\_SPLIT | Float/Double | Moderate | Random floats | +| ALP | Float/Double | High | Decimal-like floats | +| DELTA\_BINARY\_PACKED | Int32/Int64 | High | Sequential integers | + +Unlike [Byte Stream Split](#BYTE_STREAM_SPLIT), ALP does not require a subsequent +compression step to achieve size reduction -- the bit-packing directly reduces the +encoded size. However, ALP and Byte Stream Split can be complementary: ALP +exploits decimal structure while Byte Stream Split exploits byte-level correlation. + +#### Size Calculations + +##### Vector Size Formula + +``` +vector_bytes = vector_header_size // FLOAT: 9, DOUBLE: 13 + + ceil(num_elements * bit_width / 8) // packed values + + num_exceptions * 2 // exception positions (uint16) + + num_exceptions * sizeof(T) // exception values (4 or 8) +``` + +##### Page Size Formula + +``` +page_bytes = 7 // page header + + num_vectors * 4 // offset array + + sum(vector_bytes for each vector) // all vectors +``` + +#### Constants Reference + +| Constant | Value | Description | +|-------------------|---------|-----------------------------------------| +| Vector size | 1024 | Default elements per compressed vector | +| Max combinations | 5 | Max (e,f) pairs in preset | +| Samples per vector| 256 | Values sampled per vector | +| Sample vectors | 8 | Vectors sampled per row group | +| FLOAT max exponent| 10 | 10^10 ~ 10 billion | +| DOUBLE max exponent| 18 | 10^18 ~ 1 quintillion | From 69aaf624a9edfe56134cbd86269aa5b9d98a226d Mon Sep 17 00:00:00 2001 From: Prateek Gaur Date: Wed, 29 Apr 2026 20:01:08 +0000 Subject: [PATCH 02/15] Address review feedback on ALP encoding specification Incorporate review comments from emkornfield and alamb on PR #557: --- Encodings.md | 56 ++++++++++++++++++++++++---------------------------- 1 file changed, 26 insertions(+), 30 deletions(-) diff --git a/Encodings.md b/Encodings.md index 6f0758e6a..5db977775 100644 --- a/Encodings.md +++ b/Encodings.md @@ -38,6 +38,7 @@ For details on current implementation status, see the [Implementation Status](ht | [Delta-length byte array](#DELTALENGTH) | DELTA_LENGTH_BYTE_ARRAY = 6 | BYTE_ARRAY | | [Delta Strings](#DELTASTRING) | DELTA_BYTE_ARRAY = 7 | BYTE_ARRAY, FIXED_LEN_BYTE_ARRAY | | [Byte Stream Split](#BYTESTREAMSPLIT) | BYTE_STREAM_SPLIT = 9 | INT32, INT64, FLOAT, DOUBLE, FIXED_LEN_BYTE_ARRAY | +| [ALP](#ALP) | ALP = 10 | FLOAT, DOUBLE | ### Deprecated Encodings @@ -45,7 +46,6 @@ For details on current implementation status, see the [Implementation Status](ht | ------------------------------------- | -------------- | | [Bit-packed (Deprecated)](#BITPACKED) | BIT_PACKED = 4 | - ### Plain: (PLAIN = 0) @@ -401,11 +401,13 @@ This encoding is adapted from the paper ["ALP: Adaptive Lossless floating-Point Compression"](https://dl.acm.org/doi/10.1145/3626717) by Afroozeh and Boncz (SIGMOD 2024). -ALP works by converting floating-point values to integers using decimal scaling, -then applying Frame of Reference (FOR) encoding and bit-packing. Values that -cannot be losslessly converted are stored as exceptions. The encoding achieves -high compression for decimal-like floating-point data (e.g., monetary values, -sensor readings) while remaining fully lossless. +ALP works by converting floating-point values to integers using decimal scaling +(controlled by an *exponent* `e` and *factor* `f`), then applying Frame of +Reference (FOR) encoding and bit-packing. Values that cannot be losslessly +converted are stored separately as *exceptions*. The encoding achieves high +compression for decimal-like floating-point data (e.g., monetary values, sensor +readings) while remaining fully lossless. Each value is encoded independently, +enabling random access to individual vectors and parallel encode/decode. #### Overview @@ -430,16 +432,14 @@ The compression pipeline for each vector is: | v +----------------------------------------------------------+ - | 1. SAMPLING & PRESET GENERATION | - | Sample vectors from column chunk | - | Try all (exponent, factor) combinations | - | Select best k combinations for preset | + | 1. CHOOSE PARAMETERS | + | Select (exponent, factor) pair for this vector | +----------------------------------------------------------+ | v +----------------------------------------------------------+ | 2. DECIMAL ENCODING | - | encoded[i] = round(value[i] * 10^e * 10^(-f)) | + | encoded[i] = fast_round(value[i] * 10^e * 10^(-f)) | | Detect exceptions where decode(encode(v)) != v | +----------------------------------------------------------+ | @@ -465,7 +465,7 @@ The compression pipeline for each vector is: ##### Header (7 bytes) -All multi-byte values are little-endian. +All multi-byte values are stored in little-endian order. ``` Byte: 0 1 2 3 4 5 6 @@ -477,18 +477,16 @@ All multi-byte values are little-endian. | Offset | Field | Size | Type | Description | |--------|-------|------|------|-------------| -| 0 | compression_mode | 1 byte | uint8 | Compression mode (must be 0 = ALP) | +| 0 | compression_mode | 1 byte | uint8 | Compression mode (0 = ALP). Reserved for future variants (e.g., ALP-RD). | | 1 | integer_encoding | 1 byte | uint8 | Integer encoding (must be 0 = FOR + bit-packing) | -| 2 | log_vector_size | 1 byte | uint8 | log2(vector\_size). Must be in \[3, 15\]. Default: 10 (vector size 1024) | +| 2 | log_vector_size | 1 byte | uint8 | log2(vector\_size). Must be in the inclusive range \[3, 15\]. Recommended default: 10 (vector size 1024) | | 3 | num_elements | 4 bytes | int32 | Total number of floating-point values in the page | The number of vectors is `ceil(num_elements / vector_size)`. The last vector may contain fewer than `vector_size` elements. -**Note:** The number of elements per vector and the packed data size are NOT stored -in the header. They are derived: -* Elements per vector: `vector_size` for all vectors except the last, which may be smaller. -* Packed data size: `ceil(num_elements_in_vector * bit_width / 8)`. +**Note:** The number of elements per vector is NOT stored in the header — it is +derived: `vector_size` for all vectors except the last, which may be smaller. ##### Offset Array @@ -496,7 +494,7 @@ Immediately following the header is an array of `num_vectors` little-endian uint values. Each offset gives the byte position of the corresponding vector's data, measured from the start of the offset array itself. -The first offset equals `num_vectors * 4` (pointing just past the offset array). +The first offset always equals `num_vectors * 4` (pointing just past the offset array). Each subsequent offset equals the previous offset plus the stored size of the previous vector. @@ -521,9 +519,9 @@ Vector header sizes: Data section sizes: | Section | Size Formula | Description | |---------------------|-----------------------------|------------------------------| -| PackedValues | ceil(N * bit\_width / 8) | Bit-packed delta values | +| PackedValues | ceil(num\_elements\_in\_vector * bit\_width / 8) | Bit-packed delta values | | ExceptionPositions | num\_exceptions * 2 bytes | uint16 indices of exceptions | -| ExceptionValues | num\_exceptions * sizeof(T) | Original float/double values | +| ExceptionValues | num\_exceptions * sizeof(type) (4 for FLOAT, 8 for DOUBLE) | Original float/double values | ###### AlpInfo (4 bytes, both types) @@ -574,8 +572,7 @@ Data section sizes: ###### PackedValues The FOR-encoded deltas, bit-packed into `ceil(num_elements_in_vector * bit_width / 8)` bytes. -Values are packed from the least significant bit of each byte to the most significant bit, -in groups of 8 values, using the same bit-packing order as the +Values are bit-packed using the same LSB-first packing order as the [RLE/Bit-Packing Hybrid](#RLE) encoding. If `bit_width` is 0, no bytes are stored (all deltas are zero, meaning all encoded @@ -599,7 +596,7 @@ the same order as the corresponding positions. ``` +-------------------------------------------------------------------+ | | -| encoded = round( value * 10^e * 10^(-f) ) | +| encoded = fast_round( value * 10^e * 10^(-f) ) | | | | decoded = encoded * 10^f * 10^(-e) | | | @@ -608,9 +605,8 @@ the same order as the corresponding positions. The encoding uses two separate multiplications (not a single multiplication by `10^(e-f)`, and not division) to ensure that implementations produce identical -floating-point rounding across languages. The powers of 10 MUST be stored as -precomputed floating-point constants (i.e., literal values like `1e-3f`), not -computed at runtime. +floating-point rounding across languages. Implementations must ensure that the +encoder and decoder use identical power-of-10 values for a given exponent. ##### Fast Rounding @@ -618,10 +614,10 @@ The rounding function uses a "magic number" technique for branchless rounding: | Type | Magic Number | Formula | |--------|-----------------------------------|----------------------------------| -| FLOAT | 2^22 + 2^23 = 12,582,912 | `(int)((value + magic) - magic)` | -| DOUBLE | 2^51 + 2^52 = 6,755,399,441,055,744 | `(long)((value + magic) - magic)` | +| FLOAT | 2^22 + 2^23 = 12,582,912 | `(int32_t)((value + magic) - magic)` | +| DOUBLE | 2^51 + 2^52 = 6,755,399,441,055,744 | `(int64_t)((value + magic) - magic)` | -For negative values, the signs are reversed: `(int)((value - magic) + magic)`. +For negative values, the signs are reversed: `(int32_t)((value - magic) + magic)` for FLOAT, `(int64_t)((value - magic) + magic)` for DOUBLE. ##### Parameter Selection From 095a0e50731c29fa62257b85d1240a65d7731f9a Mon Sep 17 00:00:00 2001 From: Prateek Gaur Date: Thu, 7 May 2026 03:32:18 +0000 Subject: [PATCH 03/15] Address remaining review feedback on ALP spec - Clarify no padding between vectors in offset array description - Use 'sizeof(encoded type) (float=4 and double=8)' per reviewer suggestion --- Encodings.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Encodings.md b/Encodings.md index 5db977775..b7cea34a7 100644 --- a/Encodings.md +++ b/Encodings.md @@ -496,7 +496,7 @@ measured from the start of the offset array itself. The first offset always equals `num_vectors * 4` (pointing just past the offset array). Each subsequent offset equals the previous offset plus the stored size of the -previous vector. +previous vector. No padding is inserted between vectors. ##### Vector Format @@ -521,7 +521,7 @@ Data section sizes: |---------------------|-----------------------------|------------------------------| | PackedValues | ceil(num\_elements\_in\_vector * bit\_width / 8) | Bit-packed delta values | | ExceptionPositions | num\_exceptions * 2 bytes | uint16 indices of exceptions | -| ExceptionValues | num\_exceptions * sizeof(type) (4 for FLOAT, 8 for DOUBLE) | Original float/double values | +| ExceptionValues | num\_exceptions * sizeof(encoded type) (float=4 and double=8) | Original float/double values | ###### AlpInfo (4 bytes, both types) From ccb6674758c19dd7cc7c36a5ed5ba3e234013890 Mon Sep 17 00:00:00 2001 From: Prateek Gaur Date: Thu, 14 May 2026 00:28:24 +0000 Subject: [PATCH 04/15] Address alamb's second review: trim spec, fix wording, rework example MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Remove Characteristics, Size Calculations, Constants Reference sections - Consolidate three examples into one worked example with f!=0 and exceptions - Remove incorrect sign-reversal claim for fast_round on negative values - Soften sampling recommendation from SHOULD to suggestion - Fix "individual vectors" → "individual values" for random access - Clarify power-of-10 interop as MUST requirement - Use consistent fast_round terminology throughout --- Encodings.md | 223 +++++++++++---------------------------------------- 1 file changed, 46 insertions(+), 177 deletions(-) diff --git a/Encodings.md b/Encodings.md index b7cea34a7..ddb8e7025 100644 --- a/Encodings.md +++ b/Encodings.md @@ -407,7 +407,7 @@ Reference (FOR) encoding and bit-packing. Values that cannot be losslessly converted are stored separately as *exceptions*. The encoding achieves high compression for decimal-like floating-point data (e.g., monetary values, sensor readings) while remaining fully lossless. Each value is encoded independently, -enabling random access to individual vectors and parallel encode/decode. +enabling random access to individual values and parallel encode/decode. #### Overview @@ -605,20 +605,20 @@ the same order as the corresponding positions. The encoding uses two separate multiplications (not a single multiplication by `10^(e-f)`, and not division) to ensure that implementations produce identical -floating-point rounding across languages. Implementations must ensure that the -encoder and decoder use identical power-of-10 values for a given exponent. +floating-point results. All implementations MUST use the exact same floating-point +arithmetic and power-of-10 constants to guarantee cross-language interoperability. ##### Fast Rounding -The rounding function uses a "magic number" technique for branchless rounding: +The `fast_round` function uses a "magic number" technique for branchless rounding. + +`fast_round(value)` is defined as follows: | Type | Magic Number | Formula | |--------|-----------------------------------|----------------------------------| | FLOAT | 2^22 + 2^23 = 12,582,912 | `(int32_t)((value + magic) - magic)` | | DOUBLE | 2^51 + 2^52 = 6,755,399,441,055,744 | `(int64_t)((value + magic) - magic)` | -For negative values, the signs are reversed: `(int32_t)((value - magic) + magic)` for FLOAT, `(int64_t)((value - magic) + magic)` for DOUBLE. - ##### Parameter Selection The encoder selects the (exponent, factor) pair that minimizes exceptions. @@ -630,15 +630,15 @@ Valid combinations satisfy 0 ≤ factor ≤ exponent: | DOUBLE | 18 | 190 | To avoid the cost of exhaustive search on every vector, implementations -SHOULD use sampling to select up to 5 candidate (exponent, factor) -combinations (the "encoding preset") at the start of each column chunk. -Each vector then searches only those 5 candidates. +can use a sampling approach. One such approach, described in the paper, is to +select up to 5 candidate (exponent, factor) combinations (the "encoding preset") +at the start of each column chunk, and when encoding each vector, +test each of the 5 candidates for the fewest exceptions. -Sampling parameters: +Suggested sampling parameters (from the paper): | Parameter | Value | Description | |----------------------|-------|-------------------------------------| -| Vector Size | 1024 | Elements compressed as a unit | | Sample Size | 256 | Values sampled per vector | | Max Combinations | 5 | Best (e,f) pairs kept in preset | | Sample Vectors | 8 | Vectors sampled per row group | @@ -659,9 +659,9 @@ Exception values at positions in the vector are replaced with a placeholder (the encoded integer of the first non-exception value, or 0 if all values are exceptions) before FOR encoding. This keeps the FOR range tight. -##### Frame of Reference and Bit-Packing +##### Example: Frame of Reference and Bit-Packing -After decimal encoding and exception substitution: +Given the following data after decimal encoding and exception substitution: ``` +---------------------------------------------------------------------+ @@ -724,185 +724,54 @@ For each vector: 5. Patch exceptions: for each (position, value) in the exception arrays, overwrite the decoded output at that position with the stored value. -#### Example 1: Simple Decimal Values - -**Input:** `float values[4] = { 1.23, 4.56, 7.89, 0.12 }` - -**Step 1: Find Best Exponent/Factor** - -Testing (exponent=2, factor=0) means multiply by 10^2 = 100: - -| Value | value * 100 | Rounded | Verify: rounded * 1.0 * 0.01 | Match? | -|-------|-------------|---------|-------------------------------|--------| -| 1.23 | 123.0 | 123 | 1.23 | Yes | -| 4.56 | 456.0 | 456 | 4.56 | Yes | -| 7.89 | 789.0 | 789 | 7.89 | Yes | -| 0.12 | 12.0 | 12 | 0.12 | Yes | - -All values round-trip correctly -- no exceptions. - -**Step 2: Frame of Reference** - -| Encoded | min = 12 | Delta (encoded - min) | -|---------|----------|-----------------------| -| 123 | - | 111 | -| 456 | - | 444 | -| 789 | - | 777 | -| 12 | - | 0 | - -**Step 3: Bit Packing** - -max\_delta = 777, bit\_width = ceil(log2(778)) = 10 bits, -packed\_size = ceil(4 * 10 / 8) = 5 bytes +#### Worked Example: Exceptions and Non-Zero Factor -**Serialized Vector:** - -| Section | Content | Size | -|---------------------|----------------------------------------|----------| -| AlpInfo | e=2, f=0, num\_exceptions=0 | 4 bytes | -| ForInfo | frame\_of\_reference=12, bit\_width=10 | 5 bytes | -| PackedValues | \[111, 444, 777, 0\] at 10 bits each | 5 bytes | -| ExceptionPositions | (none) | 0 bytes | -| ExceptionValues | (none) | 0 bytes | -| **Total** | | **14 bytes** | - -Compared to PLAIN encoding (4 * 4 = 16 bytes). With 1024 values, the 9-byte -vector header becomes negligible and compression ratios of 2-8x are typical. - -#### Example 2: Values with Exceptions - -**Input:** `float values[4] = { 1.5, NaN, 2.5, 0.333... }` +**Input:** `double values[4] = { 1500.0, NaN, 2500.0, 333.3 }` -**Step 1: Decimal Encoding with (e=1, f=0)** +Best encoding found: (exponent=4, factor=3). This means: +`encoded = fast_round(value * 10^4 * 10^(-3)) = fast_round(value * 10)` -Multiply by 10^1 = 10: +**Step 1: Decimal Encoding** -| Index | Value | value * 10 | Rounded | Verify | Exception? | -|-------|----------|------------|---------|----------------|------------| -| 0 | 1.5 | 15.0 | 15 | 1.5 = 1.5 | No | -| 1 | NaN | - | - | - | Yes (NaN) | -| 2 | 2.5 | 25.0 | 25 | 2.5 = 2.5 | No | -| 3 | 0.333... | 3.333... | 3 | 0.3 != 0.333...| Yes (round-trip) | +| Index | Value | value * 10^4 * 10^(-3) | Rounded | Decoded: rounded * 10^3 * 10^(-4) | Exception? | +|-------|---------|------------------------|---------|------------------------------------|------------| +| 0 | 1500.0 | 15000.0 | 15000 | 1500.0 | No | +| 1 | NaN | - | - | - | Yes (NaN) | +| 2 | 2500.0 | 25000.0 | 25000 | 2500.0 | No | +| 3 | 333.3 | 3333.0 | 3333 | 333.3 | No | **Step 2: Handle Exceptions** -Exception positions: \[1, 3\] -Exception values: \[NaN, 0.333...\] -Placeholder: 15 (first non-exception encoded value) -Encoded with placeholders: \[15, 15, 25, 15\] +Exception positions: \[1\] +Exception values: \[NaN\] +Placeholder: 15000 (first non-exception encoded value) +Encoded with placeholders: \[15000, 15000, 25000, 3333\] **Step 3: Frame of Reference** -| Encoded | min = 15 | Delta | -|------------------|----------|-------| -| 15 | - | 0 | -| 15 (placeholder) | - | 0 | -| 25 | - | 10 | -| 15 (placeholder) | - | 0 | +| Encoded | min = 3333 | Delta | +|--------------------|------------|-------| +| 15000 | - | 11667 | +| 15000 (placeholder)| - | 11667 | +| 25000 | - | 21667 | +| 3333 | - | 0 | **Step 4: Bit Packing** -max\_delta = 10, bit\_width = ceil(log2(11)) = 4 bits, -packed\_size = ceil(4 * 4 / 8) = 2 bytes +max\_delta = 21667, bit\_width = ceil(log2(21668)) = 15 bits, +packed\_size = ceil(4 * 15 / 8) = 8 bytes **Serialized Vector:** -| Section | Content | Size | -|---------------------|----------------------------------------|----------| -| AlpInfo | e=1, f=0, num\_exceptions=2 | 4 bytes | -| ForInfo | frame\_of\_reference=15, bit\_width=4 | 5 bytes | -| PackedValues | \[0, 0, 10, 0\] at 4 bits each | 2 bytes | -| ExceptionPositions | \[1, 3\] | 4 bytes | -| ExceptionValues | \[NaN, 0.333...\] | 8 bytes | -| **Total** | | **23 bytes** | - -#### Example 3: Monetary Data (1024 values) - -1024 price values ranging from $0.01 to $999.99 (e.g., product prices). - -Optimal encoding: (exponent=2, factor=0) - -| Metric | Value | Calculation | -|---------------|-------------|--------------------------------------| -| Exponent | 2 | Multiply by 100 for 2 decimal places | -| Factor | 0 | No additional scaling needed | -| Encoded range | 1 to 99,999 | $0.01 -> 1, $999.99 -> 99999 | -| FOR min | 1 | Assuming $0.01 is present | -| Delta range | 0 to 99,998 | After FOR subtraction | -| Bit width | 17 | ceil(log2(99999)) = 17 bits | -| Packed size | 2,176 bytes | ceil(1024 * 17 / 8) | - -**Size Comparison:** +| Section | Content | Size | +|---------------------|--------------------------------------------------|----------| +| AlpInfo | e=4, f=3, num\_exceptions=1 | 4 bytes | +| ForInfo | frame\_of\_reference=3333, bit\_width=15 | 9 bytes | +| PackedValues | \[11667, 11667, 21667, 0\] at 15 bits each | 8 bytes | +| ExceptionPositions | \[1\] | 2 bytes | +| ExceptionValues | \[NaN\] | 8 bytes | +| **Total** | | **31 bytes** | -| Encoding | Size | Ratio | -|---------------|--------------|----------------------| -| PLAIN (float) | 4,096 bytes | 1.0x | -| ALP | ~2,185 bytes | 0.53x (47% smaller) | - -#### Characteristics - -| Property | Description | -|----------------|----------------------------------------------------------------------------------------| -| Lossless | All original floating-point values are perfectly recoverable, including NaN, Inf, -0.0 | -| Adaptive | Exponent/factor selection adapts per vector based on data characteristics | -| Vectorized | Fixed-size vectors enable SIMD-optimized bit packing/unpacking | -| Exception-safe | Values that don't fit decimal model are stored separately | - -**Best use cases:** - -* Monetary/financial data (prices, transactions) -* Sensor readings with fixed precision -* Scientific measurements with limited decimal places -* GPS coordinates and geographic data -* Normalized scores and percentages - -**Worst case scenarios:** - -* Random floating-point values (high exception rate) -* High-precision scientific data (many decimal places) -* Data with many special values (NaN, Inf) -* Very small datasets (header overhead dominates) - -**Comparison with other encodings:** - -| Encoding | Type Support | Compression | Best For | -|---------------------|--------------|-------------|---------------------| -| PLAIN | All | None | General purpose | -| BYTE\_STREAM\_SPLIT | Float/Double | Moderate | Random floats | -| ALP | Float/Double | High | Decimal-like floats | -| DELTA\_BINARY\_PACKED | Int32/Int64 | High | Sequential integers | - -Unlike [Byte Stream Split](#BYTE_STREAM_SPLIT), ALP does not require a subsequent -compression step to achieve size reduction -- the bit-packing directly reduces the -encoded size. However, ALP and Byte Stream Split can be complementary: ALP -exploits decimal structure while Byte Stream Split exploits byte-level correlation. - -#### Size Calculations - -##### Vector Size Formula - -``` -vector_bytes = vector_header_size // FLOAT: 9, DOUBLE: 13 - + ceil(num_elements * bit_width / 8) // packed values - + num_exceptions * 2 // exception positions (uint16) - + num_exceptions * sizeof(T) // exception values (4 or 8) -``` - -##### Page Size Formula - -``` -page_bytes = 7 // page header - + num_vectors * 4 // offset array - + sum(vector_bytes for each vector) // all vectors -``` - -#### Constants Reference +Compared to PLAIN encoding (4 * 8 = 32 bytes). With 1024 values, the 13-byte +vector header becomes negligible and compression ratios of 2-8x are typical. -| Constant | Value | Description | -|-------------------|---------|-----------------------------------------| -| Vector size | 1024 | Default elements per compressed vector | -| Max combinations | 5 | Max (e,f) pairs in preset | -| Samples per vector| 256 | Values sampled per vector | -| Sample vectors | 8 | Vectors sampled per row group | -| FLOAT max exponent| 10 | 10^10 ~ 10 billion | -| DOUBLE max exponent| 18 | 10^18 ~ 1 quintillion | From 270d455eb7b32e6e1a851e66337f7e42436aa18b Mon Sep 17 00:00:00 2001 From: Prateek Gaur Date: Wed, 3 Jun 2026 14:49:57 +0000 Subject: [PATCH 05/15] Address review: clarify power-of-10 constants and add fast_round sign branching MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Specify that power-of-10 constants must match IEEE 754 correctly-rounded decimal-to-binary conversion of literals (§5.12.2), not runtime pow() - Add negative-value branch to fast_round formula table to avoid landing in a binade where ULP < 1.0, which causes unnecessary exceptions --- Encodings.md | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/Encodings.md b/Encodings.md index ddb8e7025..f554bbc99 100644 --- a/Encodings.md +++ b/Encodings.md @@ -607,6 +607,11 @@ The encoding uses two separate multiplications (not a single multiplication by `10^(e-f)`, and not division) to ensure that implementations produce identical floating-point results. All implementations MUST use the exact same floating-point arithmetic and power-of-10 constants to guarantee cross-language interoperability. +The power-of-10 constants MUST be the correctly-rounded IEEE 754 values of the +decimal literals `1e0`, `1e1`, ..., `1e18` and `1e-1`, `1e-2`, ..., `1e-18` as +defined by the decimal-to-binary conversion in IEEE 754-2008 §5.12.2. +Implementations MUST NOT compute these constants at runtime via `pow()` or +equivalent functions, which are not guaranteed to be correctly rounded. ##### Fast Rounding @@ -614,10 +619,19 @@ The `fast_round` function uses a "magic number" technique for branchless roundin `fast_round(value)` is defined as follows: -| Type | Magic Number | Formula | -|--------|-----------------------------------|----------------------------------| -| FLOAT | 2^22 + 2^23 = 12,582,912 | `(int32_t)((value + magic) - magic)` | -| DOUBLE | 2^51 + 2^52 = 6,755,399,441,055,744 | `(int64_t)((value + magic) - magic)` | +| Type | Magic Number | Formula (value ≥ 0) | Formula (value < 0) | +|--------|-----------------------------------|----------------------------------|----------------------------------| +| FLOAT | 2^22 + 2^23 = 12,582,912 | `(int32_t)((value + magic) - magic)` | `(int32_t)((value - magic) + magic)` | +| DOUBLE | 2^51 + 2^52 = 6,755,399,441,055,744 | `(int64_t)((value + magic) - magic)` | `(int64_t)((value - magic) + magic)` | + +The sign branching is necessary because the technique relies on `value ± magic` +landing in a binade where the unit in the last place (ULP) equals 1.0. For +non-negative values, `value + magic` lands in [2^23, 2^24) for floats or +[2^52, 2^53) for doubles. For negative values, `value - magic` lands in +[-2^24, -2^23) or [-2^53, -2^52) respectively, where ULP is also 1.0. Without +sign branching, negative values with magnitude beyond 2^22 (float) or 2^51 +(double) fall into a lower binade where ULP < 1.0, producing incorrect +rounding and a higher exception rate. ##### Parameter Selection From 87f1630c01ef44723297e7a349d58baae4aafea3 Mon Sep 17 00:00:00 2001 From: Prateek Gaur Date: Wed, 3 Jun 2026 14:55:51 +0000 Subject: [PATCH 06/15] Address review: clarify parameter selection is encoder-only optimization MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Any valid (e,f) pair produces correct output — the decoder is agnostic. Reframe "minimize exceptions" as one heuristic; the actual target is smallest encoded size (bit-width + exception overhead). --- Encodings.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/Encodings.md b/Encodings.md index f554bbc99..1e5570053 100644 --- a/Encodings.md +++ b/Encodings.md @@ -635,7 +635,15 @@ rounding and a higher exception rate. ##### Parameter Selection -The encoder selects the (exponent, factor) pair that minimizes exceptions. +Any valid (exponent, factor) pair produces a correct encoding — the decoder is +agnostic to the selection strategy, and the exception mechanism guarantees +round-trip fidelity regardless of which pair is chosen. The choice only affects +compression ratio. + +The encoder SHOULD select the (exponent, factor) pair that produces the smallest +encoded output. A simple heuristic is to minimize exception count; a more precise +approach accounts for both bit-width and exception overhead. + Valid combinations satisfy 0 ≤ factor ≤ exponent: | Type | Max Exponent | Total Combinations | @@ -647,7 +655,7 @@ To avoid the cost of exhaustive search on every vector, implementations can use a sampling approach. One such approach, described in the paper, is to select up to 5 candidate (exponent, factor) combinations (the "encoding preset") at the start of each column chunk, and when encoding each vector, -test each of the 5 candidates for the fewest exceptions. +evaluate each candidate for the best compression. Suggested sampling parameters (from the paper): From 2169e26d9d8ba9993592749fe3ccb3f385d5bd8a Mon Sep 17 00:00:00 2001 From: Prateek Gaur Date: Wed, 3 Jun 2026 14:57:50 +0000 Subject: [PATCH 07/15] Address review: fix out-of-range condition to reference correct integer types The example incorrectly referenced only INT32_MAX for all types. Reworded to specify int32 for FLOAT and int64 for DOUBLE, avoiding exact numeric limits that differ from INT_MAX due to float representability. --- Encodings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Encodings.md b/Encodings.md index 1e5570053..05d9a5e0e 100644 --- a/Encodings.md +++ b/Encodings.md @@ -674,7 +674,7 @@ A value becomes an exception if any of the following is true: | NaN | `NaN` | Cannot convert to integer | | Infinity | `+Inf`, `-Inf` | Cannot convert to integer | | Negative zero | `-0.0` | Would become `+0.0` after encoding | -| Out of range | value * 10^e > INT32\_MAX | Exceeds target integer limits | +| Out of range | scaled value outside int32 (FLOAT) or int64 (DOUBLE) | Exceeds target integer type range | | Round-trip failure | `0.333...` with e=1, f=0 | `decode(encode(v)) != v` | Exception values at positions in the vector are replaced with a placeholder From 55a503d326fb615e59bf46c5bb436e97b8e84893 Mon Sep 17 00:00:00 2001 From: Prateek Gaur Date: Mon, 20 Jul 2026 05:54:06 +0000 Subject: [PATCH 08/15] Address review: mark encoder pipeline as informative, clarify example and layout --- Encodings.md | 46 ++++++++++++++++++++++++++++++---------------- 1 file changed, 30 insertions(+), 16 deletions(-) diff --git a/Encodings.md b/Encodings.md index 05d9a5e0e..40c8511c6 100644 --- a/Encodings.md +++ b/Encodings.md @@ -425,7 +425,10 @@ or more encoded vectors (batches of values). Each vector contains up to +-------------+------+------+-----+---------+----------+----------+-----+----------+ ``` -The compression pipeline for each vector is: +The compression pipeline below describes *one* way to produce a conforming +vector. It is informative, not normative: an encoder may use any strategy as long +as it emits the byte layout defined in [Page Layout](#page-layout). Only that +byte layout and the [Decoding](#decoding) procedure are normative. ``` Input: float/double array @@ -446,7 +449,7 @@ The compression pipeline for each vector is: v +----------------------------------------------------------+ | 3. FRAME OF REFERENCE (FOR) | - | min_val = min(encoded[]) | + | min_val = min(encoded[:]) | | delta[i] = encoded[i] - min_val | +----------------------------------------------------------+ | @@ -461,6 +464,10 @@ The compression pipeline for each vector is: Output: Serialized vector bytes ``` +The `fast_round` used in step 2 is defined precisely in +[Fast Rounding](#fast-rounding) below, including the exact magic-number values +and the sign branching required for negative values. + #### Page Layout ##### Header (7 bytes) @@ -498,6 +505,10 @@ The first offset always equals `num_vectors * 4` (pointing just past the offset Each subsequent offset equals the previous offset plus the stored size of the previous vector. No padding is inserted between vectors. +A vector's absolute byte position within the page is +`page_data_start + 7 + offset`, where `page_data_start` is the first byte after +the page's Thrift header and `7` is the size of the ALP header. + ##### Vector Format Each vector is self-describing and contains the encoding parameters, FOR metadata, @@ -573,7 +584,9 @@ Data section sizes: The FOR-encoded deltas, bit-packed into `ceil(num_elements_in_vector * bit_width / 8)` bytes. Values are bit-packed using the same LSB-first packing order as the -[RLE/Bit-Packing Hybrid](#RLE) encoding. +[RLE/Bit-Packing Hybrid](#RLE) encoding. When the total number of packed bits is +not a multiple of 8, the final byte is padded with zero bits in its most +significant positions. If `bit_width` is 0, no bytes are stored (all deltas are zero, meaning all encoded integers are equal to `frame_of_reference`). @@ -625,7 +638,8 @@ The `fast_round` function uses a "magic number" technique for branchless roundin | DOUBLE | 2^51 + 2^52 = 6,755,399,441,055,744 | `(int64_t)((value + magic) - magic)` | `(int64_t)((value - magic) + magic)` | The sign branching is necessary because the technique relies on `value ± magic` -landing in a binade where the unit in the last place (ULP) equals 1.0. For +landing in a [binade](https://en.wikipedia.org/wiki/Binade) where the unit in the +last place (ULP) equals 1.0. For non-negative values, `value + magic` lands in [2^23, 2^24) for floats or [2^52, 2^53) for doubles. For negative values, `value - magic` lands in [-2^24, -2^23) or [-2^53, -2^52) respectively, where ULP is also 1.0. Without @@ -697,7 +711,7 @@ Given the following data after decimal encoding and exception substitution: | Step | Formula | Example | |------------------------|---------------------------------------|-----------------------------| -| 1. Find min | min\_val = min(encoded\[\]) | 12 | +| 1. Find min | min\_val = min(encoded\[:\]) | 12 | | 2. Compute deltas | delta\[i\] = encoded\[i\] - min\_val | \[111, 444, 777, 0\] | | 3. Calculate bit width | bit\_width = ceil(log2(max\_delta+1)) | ceil(log2(778)) = 10 | | 4. Pack values | Each value uses bit\_width bits | 4 * 10 = 40 bits = 5 bytes | @@ -748,7 +762,7 @@ For each vector: #### Worked Example: Exceptions and Non-Zero Factor -**Input:** `double values[4] = { 1500.0, NaN, 2500.0, 333.3 }` +**Input:** `double values[4] = { 1500.0, NaN, 2500.0, 333.5 }` Best encoding found: (exponent=4, factor=3). This means: `encoded = fast_round(value * 10^4 * 10^(-3)) = fast_round(value * 10)` @@ -760,27 +774,27 @@ Best encoding found: (exponent=4, factor=3). This means: | 0 | 1500.0 | 15000.0 | 15000 | 1500.0 | No | | 1 | NaN | - | - | - | Yes (NaN) | | 2 | 2500.0 | 25000.0 | 25000 | 2500.0 | No | -| 3 | 333.3 | 3333.0 | 3333 | 333.3 | No | +| 3 | 333.5 | 3335.0 | 3335 | 333.5 | No | **Step 2: Handle Exceptions** Exception positions: \[1\] Exception values: \[NaN\] Placeholder: 15000 (first non-exception encoded value) -Encoded with placeholders: \[15000, 15000, 25000, 3333\] +Encoded with placeholders: \[15000, 15000, 25000, 3335\] **Step 3: Frame of Reference** -| Encoded | min = 3333 | Delta | +| Encoded | min = 3335 | Delta | |--------------------|------------|-------| -| 15000 | - | 11667 | -| 15000 (placeholder)| - | 11667 | -| 25000 | - | 21667 | -| 3333 | - | 0 | +| 15000 | - | 11665 | +| 15000 (placeholder)| - | 11665 | +| 25000 | - | 21665 | +| 3335 | - | 0 | **Step 4: Bit Packing** -max\_delta = 21667, bit\_width = ceil(log2(21668)) = 15 bits, +max\_delta = 21665, bit\_width = ceil(log2(21666)) = 15 bits, packed\_size = ceil(4 * 15 / 8) = 8 bytes **Serialized Vector:** @@ -788,8 +802,8 @@ packed\_size = ceil(4 * 15 / 8) = 8 bytes | Section | Content | Size | |---------------------|--------------------------------------------------|----------| | AlpInfo | e=4, f=3, num\_exceptions=1 | 4 bytes | -| ForInfo | frame\_of\_reference=3333, bit\_width=15 | 9 bytes | -| PackedValues | \[11667, 11667, 21667, 0\] at 15 bits each | 8 bytes | +| ForInfo | frame\_of\_reference=3335, bit\_width=15 | 9 bytes | +| PackedValues | \[11665, 11665, 21665, 0\] at 15 bits each | 8 bytes | | ExceptionPositions | \[1\] | 2 bytes | | ExceptionValues | \[NaN\] | 8 bytes | | **Total** | | **31 bytes** | From 273536da08fa683f3c4c81cfb928d68f4b13dbac Mon Sep 17 00:00:00 2001 From: Prateek Gaur Date: Thu, 23 Jul 2026 16:23:17 +0000 Subject: [PATCH 09/15] Clarify fast_round arithmetic precision in ALP spec State explicitly that the magic-number add/subtract are floating-point operations (only the final cast is integer), and that the constants are written as integers merely because they are exact. Add a MUST requiring FLOAT fast_round to be evaluated in single precision and DOUBLE in double precision, since a promotion of the FLOAT path to double drops the value into a binade with ULP below 1.0, defeating the rounding entirely. --- Encodings.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/Encodings.md b/Encodings.md index 40c8511c6..5458688a6 100644 --- a/Encodings.md +++ b/Encodings.md @@ -637,6 +637,22 @@ The `fast_round` function uses a "magic number" technique for branchless roundin | FLOAT | 2^22 + 2^23 = 12,582,912 | `(int32_t)((value + magic) - magic)` | `(int32_t)((value - magic) + magic)` | | DOUBLE | 2^51 + 2^52 = 6,755,399,441,055,744 | `(int64_t)((value + magic) - magic)` | `(int64_t)((value - magic) + magic)` | +The `value ± magic` additions and subtractions are floating-point operations, +not integer arithmetic; only the final cast converts to an integer. The +magic-number constants are given as integers because they are exact integers +(each is a sum of two powers of two, so it is representable exactly), but they +are operands of floating-point arithmetic. + +Implementations MUST perform this arithmetic in the precision matching the value +type: FLOAT `fast_round` in IEEE 754 single precision (binary32), and DOUBLE +`fast_round` in double precision (binary64). This is required for the technique +to work at all, not merely for cross-language reproducibility. The method +depends on `value ± magic` landing in a binade where the ULP equals 1.0; that +holds only when the operation is carried out in the value's own precision. If a +FLOAT computation is instead evaluated in double precision (for example, through +an implicit promotion), `12,582,912` lands in a double binade whose ULP is far +below 1.0, no rounding occurs, and the result is wrong. + The sign branching is necessary because the technique relies on `value ± magic` landing in a [binade](https://en.wikipedia.org/wiki/Binade) where the unit in the last place (ULP) equals 1.0. For From 1b63f9a28ddac282db786f3c94eed1306d68f947 Mon Sep 17 00:00:00 2001 From: Prateek Gaur Date: Thu, 23 Jul 2026 17:09:48 +0000 Subject: [PATCH 10/15] Use 2^n magic constant for ALP fast_round instead of 1.5*2^n The 1.5*2^n midpoint constant exists to make the magic-number rounding branchless, covering a symmetric range in a single formula. Since this spec branches on sign, the midpoint is the wrong choice: with a branch, 2^n doubles the usable domain (to +/-2^23 for FLOAT and +/-2^52 for DOUBLE) and makes the branch actually necessary, whereas with the midpoint the branch was redundant. Update the constants, the domain description, and the prose (the technique is no longer branchless). --- Encodings.md | 30 +++++++++++++++++------------- 1 file changed, 17 insertions(+), 13 deletions(-) diff --git a/Encodings.md b/Encodings.md index 5458688a6..f14b49918 100644 --- a/Encodings.md +++ b/Encodings.md @@ -628,20 +628,22 @@ equivalent functions, which are not guaranteed to be correctly rounded. ##### Fast Rounding -The `fast_round` function uses a "magic number" technique for branchless rounding. +The `fast_round` function uses a "magic number" technique for fast rounding that +avoids a division or a call to a library rounding function. It branches on the +sign of `value`. `fast_round(value)` is defined as follows: | Type | Magic Number | Formula (value ≥ 0) | Formula (value < 0) | |--------|-----------------------------------|----------------------------------|----------------------------------| -| FLOAT | 2^22 + 2^23 = 12,582,912 | `(int32_t)((value + magic) - magic)` | `(int32_t)((value - magic) + magic)` | -| DOUBLE | 2^51 + 2^52 = 6,755,399,441,055,744 | `(int64_t)((value + magic) - magic)` | `(int64_t)((value - magic) + magic)` | +| FLOAT | 2^23 = 8,388,608 | `(int32_t)((value + magic) - magic)` | `(int32_t)((value - magic) + magic)` | +| DOUBLE | 2^52 = 4,503,599,627,370,496 | `(int64_t)((value + magic) - magic)` | `(int64_t)((value - magic) + magic)` | The `value ± magic` additions and subtractions are floating-point operations, not integer arithmetic; only the final cast converts to an integer. The magic-number constants are given as integers because they are exact integers -(each is a sum of two powers of two, so it is representable exactly), but they -are operands of floating-point arithmetic. +(each is a power of two, representable exactly), but they are operands of +floating-point arithmetic. Implementations MUST perform this arithmetic in the precision matching the value type: FLOAT `fast_round` in IEEE 754 single precision (binary32), and DOUBLE @@ -650,18 +652,20 @@ to work at all, not merely for cross-language reproducibility. The method depends on `value ± magic` landing in a binade where the ULP equals 1.0; that holds only when the operation is carried out in the value's own precision. If a FLOAT computation is instead evaluated in double precision (for example, through -an implicit promotion), `12,582,912` lands in a double binade whose ULP is far +an implicit promotion), `8,388,608` lands in a double binade whose ULP is far below 1.0, no rounding occurs, and the result is wrong. The sign branching is necessary because the technique relies on `value ± magic` landing in a [binade](https://en.wikipedia.org/wiki/Binade) where the unit in the -last place (ULP) equals 1.0. For -non-negative values, `value + magic` lands in [2^23, 2^24) for floats or -[2^52, 2^53) for doubles. For negative values, `value - magic` lands in -[-2^24, -2^23) or [-2^53, -2^52) respectively, where ULP is also 1.0. Without -sign branching, negative values with magnitude beyond 2^22 (float) or 2^51 -(double) fall into a lower binade where ULP < 1.0, producing incorrect -rounding and a higher exception rate. +last place (ULP) equals 1.0. For non-negative values, `value + magic` lands in +[2^23, 2^24) for floats or [2^52, 2^53) for doubles. For negative values, +`value - magic` lands in (-2^24, -2^23] or (-2^53, -2^52] respectively, where the +ULP is also 1.0. Without the branch — that is, applying the non-negative formula +to a negative value — `value + magic` falls below 2^23 (float) or 2^52 (double) +into a lower binade where the ULP is 0.5 or smaller, producing incorrect +rounding. The branch therefore gives a symmetric valid domain of (-2^23, 2^23) +for floats and (-2^52, 2^52) for doubles; scaled values outside this domain +round incorrectly and are caught by the round-trip check, becoming exceptions. ##### Parameter Selection From 47bde32ba89c9793fc89b88b691f54874cad0309 Mon Sep 17 00:00:00 2001 From: Prateek Gaur Date: Mon, 27 Jul 2026 23:13:50 +0000 Subject: [PATCH 11/15] Address review: add ALP=10 thrift enum and clarify spec Resolve open review comments on PR #557 (GH-533): - parquet.thrift: add missing Encoding.ALP = 10 (thread 50). The enum value was never added; prose and the summary table said "ALP = 10" but the machine-readable contract lacked it. - Fix paper authorship to three authors: Afroozeh, Kuffo, Boncz (62). - "For each data page..." intro, drop misleading "page-level header" (16/49). - num_elements is the non-null value count (56). - Correct vector offset base for compressed / rep-def pages: use alp_data_start (decoded page data) instead of page_data_start+7 (55). - Note Vector Format applies to compression_mode=0 / integer_encoding=0 (59). - Label vector header vs data section; forward-ref num_exceptions/bit_width (60/61). - Clarify deltas are non-negative/unsigned with no sign extension (64). - Exceptions stored as raw IEEE 754 bits, MUST NOT canonicalize NaN (63). - Header byte-diagram alignment and "parallel encoding/decoding" wording (49/52). Leaves the contested fast_round sign-branching (threads 31/47/48) untouched pending reviewer consensus. --- Encodings.md | 42 +++++++++++++++++++++++++--------- src/main/thrift/parquet.thrift | 8 +++++++ 2 files changed, 39 insertions(+), 11 deletions(-) diff --git a/Encodings.md b/Encodings.md index f14b49918..bfe273893 100644 --- a/Encodings.md +++ b/Encodings.md @@ -399,7 +399,7 @@ Supported Types: FLOAT, DOUBLE This encoding is adapted from the paper ["ALP: Adaptive Lossless floating-Point Compression"](https://dl.acm.org/doi/10.1145/3626717) -by Afroozeh and Boncz (SIGMOD 2024). +by Afroozeh, Kuffo, and Boncz (SIGMOD 2024). ALP works by converting floating-point values to integers using decimal scaling (controlled by an *exponent* `e` and *factor* `f`), then applying Frame of @@ -407,12 +407,12 @@ Reference (FOR) encoding and bit-packing. Values that cannot be losslessly converted are stored separately as *exceptions*. The encoding achieves high compression for decimal-like floating-point data (e.g., monetary values, sensor readings) while remaining fully lossless. Each value is encoded independently, -enabling random access to individual values and parallel encode/decode. +enabling random access to individual values and parallel encoding/decoding. #### Overview -ALP encoding consists of a page-level header followed by an offset array and one -or more encoded vectors (batches of values). Each vector contains up to +For each data page, ALP encoding consists of a header followed by an offset array +and one or more encoded vectors (batches of values). Each vector contains up to `vector_size` elements (default 1024). ``` @@ -475,7 +475,7 @@ and the sign branching required for negative values. All multi-byte values are stored in little-endian order. ``` - Byte: 0 1 2 3 4 5 6 + Byte: 0 1 2 3 4 5 6 +----------------+---------------+--------------+----+----+----+----+ | compression | integer | log_vector | num_elements | | _mode | _encoding | _size | (int32 LE) | @@ -487,7 +487,7 @@ All multi-byte values are stored in little-endian order. | 0 | compression_mode | 1 byte | uint8 | Compression mode (0 = ALP). Reserved for future variants (e.g., ALP-RD). | | 1 | integer_encoding | 1 byte | uint8 | Integer encoding (must be 0 = FOR + bit-packing) | | 2 | log_vector_size | 1 byte | uint8 | log2(vector\_size). Must be in the inclusive range \[3, 15\]. Recommended default: 10 (vector size 1024) | -| 3 | num_elements | 4 bytes | int32 | Total number of floating-point values in the page | +| 3 | num_elements | 4 bytes | int32 | Total number of non-null floating-point values in the page | The number of vectors is `ceil(num_elements / vector_size)`. The last vector may contain fewer than `vector_size` elements. @@ -505,14 +505,21 @@ The first offset always equals `num_vectors * 4` (pointing just past the offset Each subsequent offset equals the previous offset plus the stored size of the previous vector. No padding is inserted between vectors. -A vector's absolute byte position within the page is -`page_data_start + 7 + offset`, where `page_data_start` is the first byte after -the page's Thrift header and `7` is the size of the ALP header. +Offsets are relative to the start of the offset array. A vector's absolute byte +position is `alp_data_start + 7 + offset`, where `alp_data_start` is the first +byte of the ALP header within the *decoded* page data — that is, after the page +has been decompressed and after any repetition/definition levels — and `7` is the +size of the ALP header. (When the page is uncompressed and carries no repetition +or definition levels, `alp_data_start` coincides with the first byte after the +page's Thrift header.) ##### Vector Format Each vector is self-describing and contains the encoding parameters, FOR metadata, -bit-packed encoded values, and exception data. +bit-packed encoded values, and exception data. The layout described here applies +when `compression_mode` = 0 (ALP) and `integer_encoding` = 0 (FOR + bit-packing); +future modes may define different vector contents and need not include `AlpInfo` +or `ForInfo`. ``` +-------------------+-----------------+-------------------+---------------------+-------------------+ @@ -521,6 +528,10 @@ bit-packed encoded values, and exception data. +-------------------+-----------------+-------------------+---------------------+-------------------+ ``` +The first two components (`AlpInfo` and `ForInfo`) form the *vector header*; the +remaining three (`PackedValues`, `ExceptionPositions`, `ExceptionValues`) form the +*data section*. + Vector header sizes: | Type | AlpInfo | ForInfo | Total Header | |--------|---------|---------|--------------| @@ -534,6 +545,9 @@ Data section sizes: | ExceptionPositions | num\_exceptions * 2 bytes | uint16 indices of exceptions | | ExceptionValues | num\_exceptions * sizeof(encoded type) (float=4 and double=8) | Original float/double values | +Here `bit_width` and `num_exceptions` are read from the vector header (`ForInfo` +and `AlpInfo` respectively), described below. + ###### AlpInfo (4 bytes, both types) ``` @@ -588,6 +602,10 @@ Values are bit-packed using the same LSB-first packing order as the not a multiple of 8, the final byte is padded with zero bits in its most significant positions. +Because `frame_of_reference` is the minimum encoded integer in the vector, every +delta is non-negative. Deltas are packed and interpreted as unsigned integers; no +sign extension is applied when unpacking. + If `bit_width` is 0, no bytes are stored (all deltas are zero, meaning all encoded integers are equal to `frame_of_reference`). @@ -600,7 +618,9 @@ the 0-based index within the vector of an exception value. An array of `num_exceptions` values in the original floating-point type (4 bytes little-endian IEEE 754 for FLOAT, 8 bytes for DOUBLE), stored in -the same order as the corresponding positions. +the same order as the corresponding positions. Each value is stored as its exact +IEEE 754 bit pattern; implementations MUST NOT canonicalize NaN or otherwise alter +the bits, so that decoding reproduces the original value bit-for-bit. #### Encoding diff --git a/src/main/thrift/parquet.thrift b/src/main/thrift/parquet.thrift index 883264c32..cc8d1b7ba 100644 --- a/src/main/thrift/parquet.thrift +++ b/src/main/thrift/parquet.thrift @@ -628,6 +628,14 @@ enum Encoding { Support for INT32, INT64 and FIXED_LEN_BYTE_ARRAY added in 2.11. */ BYTE_STREAM_SPLIT = 9; + + /** Adaptive Lossless floating-Point (ALP) encoding for FLOAT and DOUBLE. + Losslessly converts decimal-like floating-point values to integers via + decimal scaling, then applies Frame of Reference (FOR) encoding and + bit-packing; values that cannot be converted losslessly are stored as + exceptions. See Encodings.md for the detailed specification. + */ + ALP = 10; } /** From 5ae5435be69eecf4642eef0299898f0d25647aed Mon Sep 17 00:00:00 2001 From: Prateek Gaur Date: Mon, 27 Jul 2026 23:49:59 +0000 Subject: [PATCH 12/15] Address review: pipeline wording, unsigned deltas, vector diagram Follow-up to review comments on PR #557 (GH-533): - Pipeline step 1 wording: "for this array" to match the diagram's "Input: float/double array" (thread 54). - Clarify FOR deltas are computed and stored as unsigned integers, explicitly to avoid signed overflow when max-min exceeds the encoded type's signed maximum; no sign extension on unpack (thread 64). - Annotate the Vector Format diagram with a bracket row marking the vector header vs the data section (thread 61). --- Encodings.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/Encodings.md b/Encodings.md index bfe273893..1e5b73d0e 100644 --- a/Encodings.md +++ b/Encodings.md @@ -436,7 +436,7 @@ byte layout and the [Decoding](#decoding) procedure are normative. v +----------------------------------------------------------+ | 1. CHOOSE PARAMETERS | - | Select (exponent, factor) pair for this vector | + | Select (exponent, factor) pair for this array | +----------------------------------------------------------+ | v @@ -522,6 +522,7 @@ future modes may define different vector contents and need not include `AlpInfo` or `ForInfo`. ``` +<----------- Vector Header -----------><----------------------- Data Section -----------------------> +-------------------+-----------------+-------------------+---------------------+-------------------+ | AlpInfo | ForInfo | PackedValues | ExceptionPositions | ExceptionValues | | (4 bytes) | (5B or 9B) | (variable) | (variable) | (variable) | @@ -602,9 +603,12 @@ Values are bit-packed using the same LSB-first packing order as the not a multiple of 8, the final byte is padded with zero bits in its most significant positions. -Because `frame_of_reference` is the minimum encoded integer in the vector, every -delta is non-negative. Deltas are packed and interpreted as unsigned integers; no -sign extension is applied when unpacking. +Each delta is `encoded[i] - frame_of_reference`, computed in unsigned (wrapping) +arithmetic and stored as an unsigned integer. Computing it as unsigned avoids +signed-integer overflow when the vector's range (`max - min`) exceeds the signed +maximum of the encoded type, and it means no sign extension is applied when +unpacking. Because `frame_of_reference` is the minimum encoded integer in the +vector, every delta is non-negative. If `bit_width` is 0, no bytes are stored (all deltas are zero, meaning all encoded integers are equal to `frame_of_reference`). From 1293332211d2df932f719f61618521b17eddf0f3 Mon Sep 17 00:00:00 2001 From: Prateek Gaur Date: Wed, 29 Jul 2026 16:15:26 +0000 Subject: [PATCH 13/15] Mark ALP fast_round rounding as informative --- Encodings.md | 86 +++++++++++++++++++++++++++------------------------- 1 file changed, 45 insertions(+), 41 deletions(-) diff --git a/Encodings.md b/Encodings.md index 1e5b73d0e..b8315a8c0 100644 --- a/Encodings.md +++ b/Encodings.md @@ -640,56 +640,60 @@ the bits, so that decoding reproduces the original value bit-for-bit. +-------------------------------------------------------------------+ ``` -The encoding uses two separate multiplications (not a single multiplication by -`10^(e-f)`, and not division) to ensure that implementations produce identical -floating-point results. All implementations MUST use the exact same floating-point -arithmetic and power-of-10 constants to guarantee cross-language interoperability. -The power-of-10 constants MUST be the correctly-rounded IEEE 754 values of the -decimal literals `1e0`, `1e1`, ..., `1e18` and `1e-1`, `1e-2`, ..., `1e-18` as -defined by the decimal-to-binary conversion in IEEE 754-2008 §5.12.2. +The formula uses two separate multiplications (not a single multiplication by +`10^(e-f)`, and not division). This is a requirement of the **decode** path, which +is normative: to reconstruct a value every reader MUST compute +`decoded = encoded * 10^f * 10^(-e)` using the same two-step multiplication and the +same power-of-10 constants, so that all implementations reproduce the stored value +bit-for-bit. The power-of-10 constants MUST be the correctly-rounded IEEE 754 +values of the decimal literals `1e0`, `1e1`, ..., `1e18` and `1e-1`, `1e-2`, ..., +`1e-18` as defined by the decimal-to-binary conversion in IEEE 754-2008 §5.12.2. Implementations MUST NOT compute these constants at runtime via `pow()` or equivalent functions, which are not guaranteed to be correctly rounded. -##### Fast Rounding - -The `fast_round` function uses a "magic number" technique for fast rounding that -avoids a division or a call to a library rounding function. It branches on the -sign of `value`. - -`fast_round(value)` is defined as follows: +The **encode** direction — mapping each value to the integer it will be stored as, +via `fast_round(value * 10^e * 10^(-f))` — is informative, not normative. An +encoder MAY choose that integer by any means, because every value is checked +against the normative decode above and any value that does not round-trip exactly +is stored as an exception. The rounding method therefore affects only compression +ratio and exception count, never correctness or what a reader decodes. The +`fast_round` technique below is one recommended implementation. + +##### Fast Rounding (informative) + +`fast_round` recovers the integer intended by `value * 10^e * 10^(-f)` — which +carries floating-point rounding noise — by rounding it to the nearest integer, +without a division or a call to a library rounding function. It is **not** +normative: as noted above, an encoder MAY use any rounding method, since values +that do not round-trip under the normative decode are stored as exceptions. + +The technique relies on a "magic number" — a power of two whose +[binade](https://en.wikipedia.org/wiki/Binade) has a unit in the last place (ULP) +of exactly 1.0 — so that adding then subtracting it discards the fractional bits +and leaves a nearest (ties-to-even) integer. Implementations vary: some apply it +directly in a single branch-free form; others add a sign test so that both positive +and negative values land in a binade with ULP 1.0: | Type | Magic Number | Formula (value ≥ 0) | Formula (value < 0) | |--------|-----------------------------------|----------------------------------|----------------------------------| | FLOAT | 2^23 = 8,388,608 | `(int32_t)((value + magic) - magic)` | `(int32_t)((value - magic) + magic)` | | DOUBLE | 2^52 = 4,503,599,627,370,496 | `(int64_t)((value + magic) - magic)` | `(int64_t)((value - magic) + magic)` | -The `value ± magic` additions and subtractions are floating-point operations, -not integer arithmetic; only the final cast converts to an integer. The -magic-number constants are given as integers because they are exact integers -(each is a power of two, representable exactly), but they are operands of -floating-point arithmetic. - -Implementations MUST perform this arithmetic in the precision matching the value -type: FLOAT `fast_round` in IEEE 754 single precision (binary32), and DOUBLE -`fast_round` in double precision (binary64). This is required for the technique -to work at all, not merely for cross-language reproducibility. The method -depends on `value ± magic` landing in a binade where the ULP equals 1.0; that -holds only when the operation is carried out in the value's own precision. If a -FLOAT computation is instead evaluated in double precision (for example, through -an implicit promotion), `8,388,608` lands in a double binade whose ULP is far -below 1.0, no rounding occurs, and the result is wrong. - -The sign branching is necessary because the technique relies on `value ± magic` -landing in a [binade](https://en.wikipedia.org/wiki/Binade) where the unit in the -last place (ULP) equals 1.0. For non-negative values, `value + magic` lands in -[2^23, 2^24) for floats or [2^52, 2^53) for doubles. For negative values, -`value - magic` lands in (-2^24, -2^23] or (-2^53, -2^52] respectively, where the -ULP is also 1.0. Without the branch — that is, applying the non-negative formula -to a negative value — `value + magic` falls below 2^23 (float) or 2^52 (double) -into a lower binade where the ULP is 0.5 or smaller, producing incorrect -rounding. The branch therefore gives a symmetric valid domain of (-2^23, 2^23) -for floats and (-2^52, 2^52) for doubles; scaled values outside this domain -round incorrectly and are caught by the round-trip check, becoming exceptions. +The `value ± magic` operations are floating-point, not integer, arithmetic; only +the final cast converts to an integer. The arithmetic should be evaluated in the +value's own precision (FLOAT in binary32, DOUBLE in binary64), because it relies on +`value ± magic` landing in a binade where the ULP equals 1.0, which holds only in +the matching precision — evaluating a FLOAT computation in double precision, for +example, places `8,388,608` in a binade whose ULP is far below 1.0 and no rounding +occurs. + +Each form has a limited valid domain — roughly (-2^23, 2^23) for floats and +(-2^52, 2^52) for doubles with the sign-test form, and a different range for the +branch-free form — and the two disagree on some inputs (for instance, the +branch-free form is exact for some large-magnitude values where the sign-test form +is off by one, and vice versa). Because any value a given variant rounds incorrectly +is stored as an exception, the choice never affects correctness — only compression +ratio — which is why it is left to the encoder rather than mandated. ##### Parameter Selection From 4752c5115601e35823fdc503156527dcdf8276bf Mon Sep 17 00:00:00 2001 From: Prateek Gaur Date: Wed, 29 Jul 2026 16:25:56 +0000 Subject: [PATCH 14/15] Address review: raw-bit exceptions, fast_round reference wording --- Encodings.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/Encodings.md b/Encodings.md index b8315a8c0..d14924746 100644 --- a/Encodings.md +++ b/Encodings.md @@ -464,9 +464,8 @@ byte layout and the [Decoding](#decoding) procedure are normative. Output: Serialized vector bytes ``` -The `fast_round` used in step 2 is defined precisely in -[Fast Rounding](#fast-rounding) below, including the exact magic-number values -and the sign branching required for negative values. +The `fast_round` used in step 2 is one recommended rounding technique, described in +[Fast Rounding](#fast-rounding) below; it is informative, not normative. #### Page Layout @@ -544,7 +543,7 @@ Data section sizes: |---------------------|-----------------------------|------------------------------| | PackedValues | ceil(num\_elements\_in\_vector * bit\_width / 8) | Bit-packed delta values | | ExceptionPositions | num\_exceptions * 2 bytes | uint16 indices of exceptions | -| ExceptionValues | num\_exceptions * sizeof(encoded type) (float=4 and double=8) | Original float/double values | +| ExceptionValues | num\_exceptions * sizeof(encoded type) (float=4 and double=8) | Original values, stored as their exact IEEE-754 bits (NaN not canonicalized) | Here `bit_width` and `num_exceptions` are read from the vector header (`ForInfo` and `AlpInfo` respectively), described below. From 2447379702987dc9aac8e3bf52dced4059e40859 Mon Sep 17 00:00:00 2001 From: Prateek Gaur Date: Wed, 29 Jul 2026 17:00:33 +0000 Subject: [PATCH 15/15] Address review: trim informative Fast Rounding section --- Encodings.md | 39 +++++++++++++-------------------------- 1 file changed, 13 insertions(+), 26 deletions(-) diff --git a/Encodings.md b/Encodings.md index d14924746..78af7ea17 100644 --- a/Encodings.md +++ b/Encodings.md @@ -661,38 +661,25 @@ ratio and exception count, never correctness or what a reader decodes. The ##### Fast Rounding (informative) `fast_round` recovers the integer intended by `value * 10^e * 10^(-f)` — which -carries floating-point rounding noise — by rounding it to the nearest integer, -without a division or a call to a library rounding function. It is **not** -normative: as noted above, an encoder MAY use any rounding method, since values -that do not round-trip under the normative decode are stored as exceptions. - -The technique relies on a "magic number" — a power of two whose -[binade](https://en.wikipedia.org/wiki/Binade) has a unit in the last place (ULP) -of exactly 1.0 — so that adding then subtracting it discards the fractional bits -and leaves a nearest (ties-to-even) integer. Implementations vary: some apply it -directly in a single branch-free form; others add a sign test so that both positive -and negative values land in a binade with ULP 1.0: +carries floating-point rounding noise — by rounding it to the nearest integer +(ties to even), without a division or a call to a library rounding function. It is +**not** normative: an encoder MAY use any rounding method, since values that do not +round-trip under the normative decode are stored as exceptions. + +The technique adds then subtracts a "magic number" (a power of two large enough to +discard the fractional bits), leaving the nearest integer. Implementations vary: +some apply it in a single branch-free form, others add a sign test. | Type | Magic Number | Formula (value ≥ 0) | Formula (value < 0) | |--------|-----------------------------------|----------------------------------|----------------------------------| | FLOAT | 2^23 = 8,388,608 | `(int32_t)((value + magic) - magic)` | `(int32_t)((value - magic) + magic)` | | DOUBLE | 2^52 = 4,503,599,627,370,496 | `(int64_t)((value + magic) - magic)` | `(int64_t)((value - magic) + magic)` | -The `value ± magic` operations are floating-point, not integer, arithmetic; only -the final cast converts to an integer. The arithmetic should be evaluated in the -value's own precision (FLOAT in binary32, DOUBLE in binary64), because it relies on -`value ± magic` landing in a binade where the ULP equals 1.0, which holds only in -the matching precision — evaluating a FLOAT computation in double precision, for -example, places `8,388,608` in a binade whose ULP is far below 1.0 and no rounding -occurs. - -Each form has a limited valid domain — roughly (-2^23, 2^23) for floats and -(-2^52, 2^52) for doubles with the sign-test form, and a different range for the -branch-free form — and the two disagree on some inputs (for instance, the -branch-free form is exact for some large-magnitude values where the sign-test form -is off by one, and vice versa). Because any value a given variant rounds incorrectly -is stored as an exception, the choice never affects correctness — only compression -ratio — which is why it is left to the encoder rather than mandated. +The `value ± magic` operations must be evaluated in the value's own precision +(FLOAT in binary32, DOUBLE in binary64); only the final cast converts to an integer. +The two forms round some large-magnitude inputs differently, but since any value +that fails to round-trip is stored as an exception, the choice affects only +compression ratio, never correctness. ##### Parameter Selection