Add HiGlass-compatible tileset endpoints for Hi-C contact maps — Closes #82 - #118
Draft
conradbzura wants to merge 14 commits into
Draft
Add HiGlass-compatible tileset endpoints for Hi-C contact maps — Closes #82#118conradbzura wants to merge 14 commits into
conradbzura wants to merge 14 commits into
Conversation
The mongodb image baked the sample dump in with ``COPY database/``, but that directory has been gitignored since the 4DN backup was removed from the repo, so on a clean checkout the COPY failed the build outright. The build only appeared to work on machines where a stray file such as ``.DS_Store`` happened to keep the directory alive. The dump is now mounted read-only at run time instead: the image no longer varies with whatever a developer has on disk, and obtaining a dump later costs a container restart rather than a rebuild. A missing or empty ``database/`` degrades to "empty database" with a pointer at ``/sync``, while a dump that IS present and fails to restore still stops the container rather than leaving a half-loaded database looking healthy. An empty database then surfaced a second bootstrap break: ``getIndexes`` raises NamespaceNotFound on a collection that has never been written to, which under ``set -e`` took the whole container down. A namespace that does not exist trivially holds no conflicting index, so the helper treats it as the empty list and falls through to ``createIndex``, which creates the collection along with the index. Only that one error code is swallowed -- anything else still propagates.
Contact maps need a third artifact kind alongside ``data`` and ``index``: a file prepared for random-access tile reads. Unlike the other two it is never streamed to a client -- it exists to be opened locally by the tile server and read a 256x256 block at a time, which is why no router serves it and why the tile endpoints 404 rather than 202 when it is absent. ``LocalFsCache`` grows ``path_for`` because h5py opens a file rather than consuming a byte stream, so ``get`` is the wrong shape for the tile server; on this backend the cached artifact already is a local file, so it can be opened in place. That is also what lets the tile subsystem be unit-tested without S3. ``CFDB_TILESET_MAX_SOURCE_BYTES`` (default 20 GiB) bounds the source a tileset artifact may be built from, checked before a byte is downloaded: the artifact for an ``.mcool`` is a copy of the upstream file, so an oversized or mislabelled source would otherwise fill the worker's disk and fail somewhere far less legible than at admission.
The worker-side half of matrix tile serving. For an ``.mcool`` the artifact is a byte-identical copy of the upstream file -- both formats are already multi-resolution pyramids built at creation time, so no pre-aggregation is required. A flat ``.cool`` is coarsened with ``cooler zoomify`` first, because it has no ``resolutions`` group and cannot be tiled as-is; balancing is skipped deliberately, since ICE on a large flat cooler costs minutes to hours and clodius falls back cleanly to unbalanced when no ``weight`` column exists. The processor claims files by filename extension rather than by ``file_format.name``, because the ontology mapping folds ``hic``, ``cool``, ``mcool`` and ``h5ad`` all onto HDF5 -- format name alone cannot distinguish a contact map from an AnnData file.
The API-side machinery a tile request needs between a cached artifact and a HiGlass payload, layered so that only ``service`` and ``store`` ever touch a tile backend: - ``formats`` decides which files are contact maps, and of what kind. The ontology cannot: ``hic``, ``cool``, ``mcool`` and ``h5ad`` all map onto the single EDAM HDF5 term, so the filename extension is the only discriminator the file document carries. Stdlib-only, so the worker-side processor can import it. - ``backend`` is the single import gate for clodius. The fork is private, so production images do not carry it; concentrating the import lets that degrade to a clear 501 on the tile routes instead of an ImportError at startup or a subsystem that half-works. - ``store`` turns a cached artifact into a local filesystem path. h5py needs random access to real bytes, so ``CacheBackend.get``'s byte stream is the wrong shape; local artifacts open in place, S3 artifacts are pulled onto an LRU-bounded local disk budget. - ``service`` owns the open-handle pool and the byte-bounded tile cache, and runs the synchronous h5py reads on a dedicated thread pool so a multi-second tile read cannot stall the event loop. - ``wire`` supplies what clodius's file-derived ``TilesetInfo`` deliberately leaves to the server: the dataset's name, its uid, and the coordinate system. - ``errors`` is cfdb's own failure vocabulary -- reasons a tileset cannot be produced at all, translated by the routers into status codes, distinct from clodius's per-tile request errors.
/data and /index each carried their own copy of the per-request preamble -- DCC validation against the registry, the database-ready check, the file lookup, and HuBMAP access control -- and the tileset routers would have made it four copies. ``resolve_file_doc`` now holds that sequence in the one ordering all of them agree on, alongside a shared path-param constraint (accessions across every DCC are subsets of ``[A-Za-z0-9._-]``; the length cap defends Mongo and log lines from unbounded input). ``FILE_DOC_PROJECTION`` additionally carries ``genome_assembly``, which every DCC's sync promotes to the top level of the file document and which the tileset routers read to derive HiGlass's ``coordSystem`` -- the ``extra.*.genome_assembly`` fields declared on the enriched models are never written, so this is the only path that carries an assembly.
Gosling's ``MatrixData`` compiles to HiGlass's native ``heatmap`` track,
which issues thousands of small tile requests and cannot be handed a job
id to poll -- neither /data nor /index can satisfy it. Two channels,
deliberately separate:
- ``/tileset_info`` and ``/tiles`` speak the HiGlass wire protocol
(repeatable ``d`` parameters, capped at 64; uid ``{dcc}/{local_id}``)
and are pure readers that never dispatch a workflow -- a contact map
whose artifact has not been built is a 404. A tile position the
resolution ladder does not hold is absent from the response rather
than reported, and a tile id that fails to parse gets an ``error``
object in its place so one malformed id cannot blank the track.
- ``POST /tilesets/{dcc}/{local_id}`` builds the artifact, and its
``/status`` probe mirrors the /data and /index readiness probes.
POST is idempotent: an artifact that already exists answers
``200 {"ready": true}`` rather than creating a no-op job.
``.hic`` is recognized and refused with a specific 501 rather than
served: hictkpy reads local paths only, and the ENCODE corpus (~78 TB,
largest file 315 GB) cannot be materialized onto Fargate ephemeral
storage at any budget. The lifespan wires the service only when the
workflow subsystem is enabled and the build carries clodius; the
``tiles`` extra stays optional because the fork is private and requires
Python 3.12, so images built today answer 501 on these routes.
``make api`` builds an image with no clodius, so its tile routes answer 501 -- correct for production while the fork is private, but it leaves the whole tile chain untestable through the documented Docker flow. ``make api-tiles`` layers the backend on and adds the one piece ``make api`` does not need: a cache volume shared with a worker, because a tileset artifact is built by the worker and then opened locally by the API with h5py. The sibling ``../clodius`` checkout is vendored into ``.clodius-src/`` (gitignored) first, because a ``[tool.uv.sources]`` path dependency pointing outside the build context is invisible to COPY. The worker reuses the API image rather than ``cfdb-wool``: the matrix processor needs cooler and h5py, which arrive with clodius and are absent from the worker image. All of this is interim scaffolding, retired once the fork is public and ``tiles`` becomes an ordinary dependency.
h5py opens a file, so under the S3 profile an mcool must be pulled onto local disk before any tile can be read from it. The Fargate task's ephemeral storage is raised above the 20 GiB default (``ApiEphemeralStorageGiB``, default 50) and the tile subsystem's byte budget rides along as ``TilesetDiskCacheBytes`` -- the two are passed together so the budget stays comfortably below the disk it shares with ``SYNC_DATA_DIR`` and the container image, rather than filling the tile cache taking /data and /sync down with it.
Formats classification, the local store's hydration and LRU eviction,
and the service's handle pool, tile cache, and thread offload -- run
against real coolers rather than mocks. The synthetic builders are
ported from the clodius fork's test harness, because the payload
assertions (a 256x256 dense block, base64-encoded, with no ``shape``
field) are only meaningful against bytes clodius actually produced.
The HDF5 routing regression suite pins what /data and /index do once a
processor claims the HDF5 format: ``MatrixTilesetProcessor`` declares
``supported_formats = {"HDF5"}``, so ``ProcessorRegistry.lookup_for``
now returns a processor for files it previously returned ``None`` for,
and those endpoints must keep streaming contact maps from upstream
rather than dispatching a workflow for them.
The tile endpoints are exercised over HTTP against real mcool fixtures: the HiGlass wire shape (repeatable ``d`` parameters, the 64-id cap, absent-vs-error tile semantics, no ``shape`` field), the uid parse rules, and the status codes the README table promises. The preparation channel's POST idempotency, the ``.hic`` 501, the md5-less 409, and the never-dispatching status probe are pinned alongside. The shared ``resolve_file_doc`` preamble gets its own suite so the four routers that consume it agree on ordering and error codes by construction.
The processor suite runs real coolers through the artifact build -- the mcool byte-copy path, the flat-cool zoomify path, and the source-size admission guard. The lifespan suite pins when the service exists at all: wired only when the workflow subsystem is enabled and the build carries clodius, and closed -- not merely nulled -- during teardown, so an open h5py handle or a live thread pool cannot leak into the next app instantiation.
ApiEphemeralStorageGiB and TilesetDiskCacheBytes exist, carry the documented defaults and bounds, and are actually wired through to the task definition -- a parameter that renders but reaches no container would pass every deploy while changing nothing.
The whole chain against bytes rather than mocks: build a contact map, put it through the matrix processor into a real cache, then serve /tileset_info and /tiles from the resulting artifact and compare tile payloads against a direct cooler read.
The README section carries the parts a consumer cannot recover from the code: why the endpoints exist at all (Gosling's MatrixData compiles to HiGlass's native heatmap track, which cannot poll a job id), the two-channel contract and its status tables, the wire-compatibility findings against higlass.io and a local higlass-server (cfdb sends the full symmetric matrix where higlass-server sends half; verified invisible to the client, and why ``mirror_tiles`` must never be emitted), the local-dev flow, and the limitations -- the private-fork 501, the deferred ``.hic`` corpus with its partial-coverage cost table, and the per-task locality of the tileset copies. CLAUDE.md points agents at the README rather than duplicating it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Gosling compiles
MatrixDatato HiGlass's nativeheatmaptrack, which issues thousands of small, latency-sensitive tile requests and cannot be handed a job id to poll. Neither/datanor/indexcan satisfy that access pattern, so 4,364 indexed contact maps are unreachable from Gosling. Implement the HiGlass wire protocol directly in FastAPI over the existing workflow cache, rather than deployinghiglass-server— a Django application whose database and tileset registry would duplicate what thefilescollection already is.The design separates two channels.
GET /tileset_infoandGET /tilesare pure readers that never dispatch work: a contact map whose artifact has not been built is a 404. Building that artifact belongs to a preparation channel —POST /tilesets/{dcc}/{local_id}plus a side-effect-free readiness probe — that a UI drives before mounting the Gosling spec. The artifact itself is a third workflow artifact kind,tileset, produced by a worker-side matrix processor: a byte-identical copy for.mcool, acooler zoomifycoarsening for the 10 flat.coolfiles, which carry no resolution pyramid and cannot be tiled as-is.Wire compatibility is verified rather than assumed. Against
higlass.io,server.gosling-lang.org, and a localhiglass-serverrun over the same.mcool, payload shapes match exactly (dense/dtype/min_value/max_value/size, 256×256, noshapefield), and tile values are bit-identical to directcoolerreads at every intrachromosomal resolution, modulo the per-tile float16 quantization both reference servers also perform. One representational difference is documented and shown to be client-invisible: cfdb sends the full symmetric matrix wherehiglass-serversends half, and the HiGlass client's own mirroring and symmetrization make the two indistinguishable.Two deliberate deviations from the issue as filed. The endpoints speak HiGlass's actual wire form (repeatable
dquery parameters) rather than the path-style routes the issue sketched, because the client drives/tilesitself and expects that contract. And.hicis recognized but refused with a specific 501 rather than served viahicstraw: every available reader consumes local paths only, and the ENCODE corpus (~78 TB, median 10.3 GB, largest 315 GB) exceeds Fargate ephemeral storage at any budget. The issue's open design decision — read in place versus convert once — is resolved as defer, with partial-coverage cost quantified in the README for when it is revisited.The tile backend is the private
abdenlab/clodiusfork'stiles_v2, consumed as an optionaltilesextra behind a single import gate; images built from the Dockerfiles answer 501 on the tile routes until the fork is published, which is tracked as follow-up. The branch also carries an independent fix: a clean checkout could not boot MongoDB at all once the sample dump left the repo.Closes #82
Proposed changes
Workflow layer: the
tilesetartifact kindAdd
ArtifactKind.TILESETfor files prepared for random-access tile reads. Unlikedataandindexit is never streamed: it exists to be opened locally by the tile server and read a 256×256 block at a time, which is why no router serves it and why the tile endpoints 404 rather than 202 in its absence. ExposeLocalFsCache.path_for, since h5py opens a file rather than consuming a byte stream, and bound artifact sources withCFDB_TILESET_MAX_SOURCE_BYTES(default 20 GiB), checked before a byte is downloaded.Worker:
MatrixTilesetProcessorBoth contact-map formats are multi-resolution pyramids built at creation time, so no pre-aggregation is performed — the artifact for an
.mcoolis a copy, and a flat.coolis coarsened withcooler zoomify(unbalanced, deliberately: ICE on a large flat cooler costs minutes to hours and clodius falls back cleanly when noweightcolumn exists). The processor claims files by filename extension because the ontology cannot discriminate:hic,cool,mcool, andh5adall map onto the single EDAM HDF5 term.API: the tileset read stack (
cfdb.tilesets)Six modules, layered so only
serviceandstoretouch a tile backend:formats(contact-map classification, stdlib-only so the worker can import it),backend(the single clodius import gate — absence degrades to a clear 501 instead of an ImportError at startup),store(cached artifact to local path; local artifacts open in place, S3 artifacts hydrate onto an LRU-bounded disk budget),service(handle pool, byte-bounded tile cache, and a dedicated thread pool so synchronous h5py reads cannot stall the event loop),wire(the fields clodius's file-derivedTilesetInfoleaves to the server: name, uid,coordSystem), anderrors(cfdb's failure vocabulary, distinct from clodius's per-tile request errors). Tunables:CFDB_TILESET_DISK_CACHE_BYTES,CFDB_TILESET_OPEN_MAX,CFDB_TILE_CACHE_BYTES,CFDB_TILE_THREADS,CFDB_TILESET_HYDRATE_TIMEOUT_S.API: routers and lifespan
Serve
/tileset_infoand/tileswith the batch semantics HiGlass relies on: up to 64dparameters, absent tiles omitted rather than reported, and a malformed id yielding a per-tileerrorobject so it cannot blank the track. Serve the preparation channel with an idempotent POST (an existing artifact answers200 {"ready": true}) and a probe that never dispatches. Wire the service in the lifespan only when the workflow subsystem is enabled and the build carries clodius, and close it — not merely null it — on teardown, so no h5py handle or thread pool leaks into the next app instantiation.Routers: shared request preamble
Extract the per-request sequence
/dataand/indexeach duplicated — DCC validation, database check, lookup, HuBMAP access control — intoresolve_file_doc, which the tileset routers would otherwise have made a fourth copy of. Add a shared path-param constraint and carrygenome_assemblyin the file projection; it is the only path that holds an assembly, andcoordSystemderives from it.Build and local dev
Add the
tilesextra with apython_version >= '3.12'marker (the fork's floor; cfdb still supports 3.11) and a non-editable path source pointed at the sibling checkout. Addmake api-tiles/make worker-tiles, which vendor the checkout into the build context and share a cache volume between API and worker — interim scaffolding, retired when the fork is public.Infrastructure
Raise the API task's ephemeral storage (
ApiEphemeralStorageGiB, default 50) and pass the tile disk budget alongside it (TilesetDiskCacheBytes), so the budget and the disk it shares withSYNC_DATA_DIRmove together.Independent fix: clean-checkout MongoDB bootstrap
The mongodb image baked the sample dump in with
COPY database/, but that directory is gitignored and absent on a clean checkout, failing the build outright. Mount the dump read-only at run time instead, degrade a missing dump to an empty database with a pointer at/sync, and treatgetIndexesNamespaceNotFound as the empty list so index creation survives never-written collections.Test cases
TestMatrixSourceKind/TestResolution.mcool/.cool/.hicare recognized as contact maps and.h5adis notTestLocalFsFastPathLocalFsCacheTestHydration/TestEvictionTestTilesetInfo/TestTiles(service)TestTileCache/TestOpenTilesetLifetimeTestOverHttp/TestRequestValidation/tileset_infoand/tilesreceive repeatabledparametersTestHicIsDeferred.hicfile documentTestDegradedModesTestStatusProbe/TestPrepareTestDataFallThrough/TestIndexFallThrough/TestRegistryLookup/dataand/indexTestApplicability/TestSourceSizeGuardTestRun/TestZoomLaddertest_lifespan_tilesetsSYNC_DATA_DIRTestFileDocProjection/TestResolveFileDocresolve_file_docrunstest_cloudformationtest_tiles_e2e