Specify storage inheritance for FILE self-references by alkis · Pull Request #603 · apache/parquet-format

alkis · 2026-07-29T17:09:01Z

Specify storage inheritance semantics for FILE self-references:

require an inline field for schemas permitting self-references
inherit compression and encryption from the corresponding inline column chunk
define each referenced range as an independent compression/encryption block
define offset and size over the stored representation
require Parquet-mediated resolution of stored bytes
define modular-encryption serialization and a page-independent AAD identity
leave external references unaffected

Discussion: https://lists.apache.org/thread/zrzc7t9fccg92rx3h4fw3ndw3bdo5xr7

rok · 2026-07-31T18:32:48Z

+compressed block or the uncompressed bytes. For a compressed block, the complete range
+is supplied to the codec, and its decompressed output is the resolved value.


Should we also store the decompressed size somewhere?

I have thought about this but it complicates the format - we will need framing for the entry. For some compressors this exists already (Snappy, optionally ZSTD), for the rest we can require a dynamically sized decompressor output as discussed here https://lists.apache.org/thread/rfz37kt9bdjw6m7o6mwf8f7nr712c0m8. If there is strong push for uncompressed size we can add a header but that's a larger change.

Perhaps we cal leave this open to be specified later if needed?

Agreed. I made that explicit in the spec rather than leaving it unsaid: the decompressed size is not stored, readers rely on the codec's framing where it provides one or decompress into a dynamically sized buffer, and a future revision may add an explicit decompressed size.

I don't think this will likely ever be needed. It's trivial to workaround the framing issues and working with File should indicate that there is an expectation that large values may be present. Readers need to be very permissive if they're going to support this type.

Agreed, and that is the conclusion I came to as well. The paragraph no longer speculates about a future field — per your other comment it just states that the decompressed size is not stored and that readers use the codec framing or a dynamically sized buffer.

@rok, flagging since removing the sentence also removes the "may be specified later" hedge you asked for. The substance you wanted is unchanged: nothing here precludes adding an explicit decompressed size in a later revision, we just are not advertising it in the spec text.

rok · 2026-07-31T18:42:16Z

+For a self-reference, `offset` points to the beginning of the 4-byte length and `size`
+is the size of the complete encrypted module, including the length, nonce, ciphertext,
+and GCM tag when present.


4-byte length limits us to 2GiB payload. Are we expecting use cases with greater size requirements?

I expect 2GiB payloads to be in their own external file reference not embedded inside parquet.

And that means we can't use parquet modular encryption, right? Should we note that?

The opposite — with this PR self-references are modular encryption modules (module type 10, keyed on the inline column key), so modular encryption works for them. This PR removes the prohibition that the merged spec had.

Two limits are worth noting and are now stated:

The encryption buffer must fit the 4-byte length field, so an encrypted self-reference is capped at 2 GiB as for any other module. Larger values use an external reference.

The AAD includes the row group and column ordinals, so an encrypted stored representation is bound to one column chunk. A writer must not share one encrypted byte range between column chunks.

wgtmac · 2026-08-03T15:21:00Z

+| OffsetIndex           |       yes        |   yes (7)   |        yes        |      yes       |     no       |          no            |
+| BloomFilter Header    |       yes        |   yes (8)   |        yes        |      yes       |     no       |          no            |
+| BloomFilter Bitset    |       yes        |   yes (9)   |        yes        |      yes       |     no       |          no            |
+| Self-Reference        |       yes        |   yes (10)  |        yes        |      yes       |     no       |          yes           |


Why Page ordinal is not used here? When only a few pages are required to read, we still need to count number of values of skipped pages in order to decrypt a self-referenced value?

You are right that a derived ordinal is wrong for exactly this reason. I switched the AAD to key on the self-reference offset instead. The offset is a field of the FILE value being read, so the reader has it in hand and never counts values in pages it skips.

Page ordinal would also work for a reader — you know the ordinal of the page you are decoding — but it ties the out-of-band range to page layout, which the rest of this PR deliberately avoids: compression and encryption are properties of the column chunk, not of the page a value happens to land in. Keying on offset keeps the stored representation independent of page boundaries, and the offset already uniquely identifies it within the file.

Key the self-reference AAD on `offset` rather than a derived ordinal, so a reader can resolve a self-reference without decoding the pages it skips. State that the encryption buffer of a self-reference must fit the 4-byte length field, and that larger values must use an external reference. State that the decompressed size is not stored and how readers obtain it. State that an encrypted stored representation is bound to a single column chunk and must not be shared between column chunks.

danielcweeks · 2026-08-06T14:38:02Z

+A value of 0 indicates an empty stored representation. `size` must be set whenever
+`offset` is set. It may be omitted only for a whole-file external reference (`uri` set,
+`offset` not set), in which case the range runs to the end of the referenced file.
+Because a self-reference always sets `offset`, it always sets `size` as well.


Suggested change

Because a self-reference always sets `offset`, it always sets `size` as well.

Because a self-reference always sets `offset`, it must always set `size` as well.

Applied in aac4d72. It also lines up with the validation bullet further down, which already stated this with "must".

danielcweeks · 2026-08-06T14:45:57Z

+so all self-references of a column chunk share them regardless of the page a value is
+stored in.


Suggested change

so all self-references of a column chunk share them regardless of the page a value is

stored in.

so all self-references of a column chunk share them regardless of where in the file the referenced value is

stored.

This seems to indicate that the value is stored in a page, but it's actually outside the page data.

Good catch — applied in aac4d72. You are right that the old wording implied the payload lives in a page, which is exactly what it does not do. The point I wanted to make is that the settings come from the column chunk, so your phrasing says it without the misleading implication.

danielcweeks · 2026-08-06T14:48:54Z

+
+The decompressed size of a self-reference is not stored. Readers must rely on the
+framing of the codec where it provides one, or decompress into a dynamically sized
+buffer. A future revision of this specification may add an explicit decompressed size.


Suggested change

buffer. A future revision of this specification may add an explicit decompressed size.

buffer.

I'd prefer not to include forward-looking statements.

Applied in aac4d72 — the forward-looking sentence is gone. The paragraph now just states that the decompressed size is not stored and how a reader copes.

danielcweeks · 2026-08-06T14:56:51Z

+construction.
+
+A self-reference identifies a stored representation, not necessarily the resolved
+bytes. Consumers must use a Parquet reader to resolve a self-reference; copying


Consumers must use a Parquet reader to resolve a self-reference

I think this is an unnecessary MUST statement. I think it's fine to just say: "copying [offset, offset + size) directly may return compressed or encrypted data".

The intent is to describe what needs to be done to correctly interpret the bytes, not enforce who does it.

Applied in aac4d72. The sentence now reads "Copying [offset, offset + size) directly may return compressed or encrypted data." — describing the consequence rather than mandating who does the resolution.

danielcweeks · 2026-08-06T14:59:33Z

+encryption. If the column chunk is not encrypted, its self-references are not
+encrypted. An encrypted self-reference is limited to 2 GiB by the length field of the


It might be good to call out explicitly here that for encrypted columns, the offset+size represents the post-encrypted size (similar to the statement we make around line 772).

Added in aac4d72: "For an encrypted self-reference, offset and size identify the encrypted module, so size is the size after compression and encryption."

One caveat on precision: size also covers the 4-byte length prefix and the nonce, which are not themselves products of compression or encryption, so "after compression and encryption" is a shorthand. Encryption.md has the exact layout — offset points at the 4-byte length, and size spans the length, nonce, ciphertext, and GCM tag when present. Happy to replace the shorthand with a pointer to that layout if you would rather the normative statement live in one place.

Took my own suggestion in ad3d1fa. The shorthand is replaced with a pointer:

For an encrypted self-reference, offset and size identify the encrypted module, whose exact extent is defined in Encrypted module serialization.

That keeps the normative definition of the extent in Encryption.md section 5.1 — offset at the 4-byte length, size spanning length, nonce, ciphertext, and GCM tag when present — instead of restating a lossy version of it here.

Align the wording with the validation rules, which state the requirement normatively. Co-authored-by: Isaac

Describe self-reference storage as being anywhere in the file rather than in a page, since the referenced bytes are outside the page data. Drop the forward-looking note about a possible decompressed size field. State that `offset` and `size` cover the encrypted module for an encrypted self-reference, as already stated for the unencrypted case. Describe rather than mandate how the stored bytes are interpreted. Co-authored-by: Isaac

The previous wording described `size` as the size after compression and encryption, which omits the length prefix and nonce that the range also covers. Reference the layout in Encryption.md so the extent is defined in one place. Co-authored-by: Isaac

alkis added 3 commits July 29, 2026 20:02

Specify compression for FILE self-references

53dc52b

Define FILE self-reference compression semantics

7b729c6

Define inherited encryption for FILE self-references

f5eb885

alkis changed the title ~~Specify compression for FILE self-references~~ Specify storage inheritance for FILE self-references Jul 30, 2026

alkis added 2 commits July 30, 2026 09:42

Avoid prescribing FILE page boundaries

86479df

Make FILE storage inheritance page-independent

3047a52

brkyvz reviewed Jul 30, 2026

View reviewed changes

Comment thread LogicalTypes.md

Align FILE resolution table

81f61ed

rok reviewed Jul 31, 2026

View reviewed changes

wgtmac reviewed Aug 3, 2026

View reviewed changes

This was referenced Aug 5, 2026

Add threshold-driven inline vs self-reference storage for FILE values brkyvz/parquet-java#1

Merged

Introduce implementation for FileType apache/parquet-java#3608

Open

danielcweeks reviewed Aug 6, 2026

View reviewed changes

alkis added 3 commits August 6, 2026 18:27

Require size for FILE self-references

c9e6381

Align the wording with the validation rules, which state the requirement normatively. Co-authored-by: Isaac

		compressed block or the uncompressed bytes. For a compressed block, the complete range
		is supplied to the codec, and its decompressed output is the resolved value.

	Because a self-reference always sets `offset`, it always sets `size` as well.
	Because a self-reference always sets `offset`, it must always set `size` as well.

		so all self-references of a column chunk share them regardless of the page a value is
		stored in.

	buffer. A future revision of this specification may add an explicit decompressed size.
	buffer.

		encryption. If the column chunk is not encrypted, its self-references are not
		encrypted. An encrypted self-reference is limited to 2 GiB by the length field of the

Uh oh!

Conversation

alkis commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alkis Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

alkis commented Jul 29, 2026 •

edited

Loading

alkis Aug 3, 2026 •

edited

Loading