icechest

A hybrid store where Icechunk manages array data and Apache Iceberg manages the tabular metadata describing it.

Prototype for NASA-IMPACT/veda-odd#460.

The problem

Collections of heterogeneous arrays can't be managed as a single data cube and instead need a queryable index. Today that index often lives in a separate system (STAC, CMR, etc) pointing at data in object storage, and keeping the two consistent takes fragile orchestration.

An earlier experiment, zarr-datafusion-search, solved the consistency problem by encoding columnar metadata as 1-D Zarr arrays inside the Icechunk store. That works, but it re-implements a lot of Parquet functionality by hand and needs a bespoke query engine.

This prototype relies on Iceberg to manage tabular information and Icechunk to manages arrays so we take advantage of both ecosystems' strengths without reinventing the wheel.

How it works

Iceberg resolves "what is the current version of this table?" through a catalog service. This has the same issues as the current "separate system" metadata stores in the community. Changes to array data and metadata can easily become desynchronized without fragile orchestration.

With Icechest the Icechunk snapshot is the catalog. Every commit carries the location of the Iceberg metadata.json describing the arrays in that same commit, so a single Icechunk commit publishes the chunks and the table version together or not at all — and an Icechunk tag pins both.

with repo.transaction("main", "ingest granule G-100") as tx:
    tx.group["reflectance"][0:10] = chunk      # array data
    tx.append("granules", granule_row)          # table row
# one commit; both or neither

The parent Zarr group declares its tables through the Icechunk–Iceberg convention, so the store is self-describing — a reader discovers the table from the Zarr hierarchy alone, with no catalog and no out-of-band configuration.

Where the pointer lives, and why

The convention's attributes declare which tables exist and where their files live. They deliberately do not carry the corresponding metadata.json location. This information lives in Icechunk commit metadata, keyed iceberg:table_versions.

The architecture uses this split model to support concurrency. If the live pointer lived in the group attributes, every writer would rewrite the same Zarr node on every commit. Icechunk reports that as a ZarrMetadataDoubleUpdate conflict on the group, which the BasicConflictSolver can't resolve. So two writers appending to disjoint array regions would be unable to rebase, even though their array writes didn't actually conflict.

Commit metadata doesn't have that contention. It is per-snapshot and set at commit time and can be set after any Iceberg conflict-driven replay onto another writer's table version.

Concurrent writing

Ben commits first. Then Anna's metadata commit conflicts so her Iceberg operation is replayed onto Ben's table version and her staged chunks are rebased onto his snapshot:

Recording table work as intents rather than applying it eagerly is what makes this possible. Replaying an intent rebuilds the metadata.json on the winning parent and nothing needs unwinding because nothing was ever published. The abandoned metadata files become dereferenced garbage that can be collected and compacted later.

This automatic recovery is for appends. An operation that depends on what the other writer changed (deleting rows for array data they rewrote) must fail and re-plan. I'll be working on supporting this in more automatic way with subsequent PRs.

Garbage collection

Iceberg's expiry and orphan-file cleanup judge if files can be deleted from the table's current version alone. So they would delete files that older active Icechunk tag still pins. Tables are therefore created with gc.enabled=false and write.metadata.delete-after-commit.enabled=false, and IcechunkCatalog blocks drop_table, purge_table and rename_table operations

Table properties only constrain well-behaved clients, so a real deployment should also deny delete permissions on the warehouse prefix at the storage layer. Reclaiming space needs a sweeper that treats Icechunk refs as the source of truth.

Layout

src/icechest/
  catalog.py       An Icechunk managed PyIceberg catalog with no catalog service
  convention.py    The Iceberg pointer Zarr convention usage
  pointer.py       The table pointer in Icechunk commit metadata
  transaction.py   Iceberg commit management. Handles intents, atomic commit and conflict replay
conventions/icechunk-iceberg/
  README.md        The Iceberg pointer convention specification
  schema.json      JSON schema; examples/ validated against it in tests

Try it

uv sync
uv run pytest
uv run python examples/concurrent_writers.py

Status and open questions

Todo:

• Garbage collection. The garbage-collection story is currently "disable Iceberg's cleanup"; the Icechunk-ref-aware collector that makes space reclaimable is unimplemented.
• Non-append conflict re-planning. Appends recover automatically. Deletes and overwrites currently just fail.