release: GeoBrix 0.5.0 — virtual tiles, v2 tile struct, CRS families by mjohns-databricks · Pull Request #72 · databrickslabs/geobrix

mjohns-databricks · 2026-08-13T14:34:58Z

GeoBrix 0.5.0 — release PR (`beta/0.5.0` → `main`)

Promotes the 0.5.0 line to main (~496 commits). Versions are bumped (pom.xml + __version__ = 0.5.0). Full change list: docs/docs/release-notes.mdx → "What's new in v0.5.0".

Highlights

Virtual tiles — bytes-free windowed reads for large rasters. Lightweight cog_gbx/raster_gbx/gtiff_gbx readers can emit virtual tiles (source path + pixel window, no bytes); pixels are read lazily one window at a time. A virtual row is ~100 B vs 148–527 KB materialized (~1,400–5,000× smaller), dissolving Serverless OOM on large-raster fan-out. Lightweight raster readers now default to virtual (breaking behavior change).
v2 8-field tile struct — cellid, raster (nullable), path, window, clip_polygon, clip_crs, crs, metadata. Both tiers read v1 and v2 and always emit v2.
Every lightweight rst_* is virtual-tile-aware via one shared open path (header-only accessors; reference/passthrough ops stay virtual; pixel ops materialize only their window). New force-output params: virtualize_dir / virtualize_prefix / materialize.
COG preparation lane — new file_gbx (path lister) + cog_gbx writer (master-COG prep) + cog_gbx reader (COG-aware windowed read: tileSize/overlapPercent/clipPolygons/windows). Reader splitStrategy now defaults to none.
CRS families (both tiers) — gbx_st_{crs,setcrs,transformcrs} (VectorX) and gbx_rst_{crs,setcrs,transformcrs} (RasterX): authority-string CRS handling that survives non-EPSG round-trips.
Plus per-function 4-tab docs examples, Scala-derived DESCRIBE FUNCTION signatures, and many fixes.

Pre-merge gate (requested)

Complete the eo-series' use of virtual tiles (lightweight tier / Serverless): notebooks showcase virtual tiles (narrative + functions used), the eo-series doc page (docs/docs/notebooks/eo-series.mdx) and the eo-series README.md reflect the support, validated on Serverless.

Opened as draft until the gate above is met. beta/0.5.0 CI is the dev-branch signal (not a merge gate); the main build runs on merge.

This pull request and its description were written by Isaac.

…rx+tests Docker lint (the CI-authoritative isort/black/flake8) surfaced a batch of pre-existing violations that blocked the branch push, plus two orphaned by R2's removal of from_epsg call sites: - F401 unused imports: edit.py (rasterio), open_tile.py (rasterio.crs), removed after R2 dropped their from_epsg uses; plus resample.py (np), functions.py (_raster_field), stac/_download.py (np), bench/compression_sweep.py, and several test files. - F841 dead locals (compression_sweep decoded_bytes, test_compression_sites pred, test_compress_option ds), E226 spacing, E402 post-importorskip imports (add # noqa: E402 per the repo convention), F811 redefined MemoryFile and E402 mid-file imports in test_pending_instructions (hoisted to the top). - black line-wrapping on 7 test files whose committed form failed Docker black. Import removals smoke-tested; 111 affected light tests green; full Docker python lint now passes (isort + black + flake8, 445 files). Co-authored-by: Isaac

Follow-on to Spec R2. Standardizes two CRS roles for every geometry argument (source vs target/output), maps them to a GeoBrix-wide naming rule (out_srid/ out_crs for output projection; srid/crs/clip_crs for input source), and closes the remaining RasterX gaps: clip_crs on rst_clip Column API, crs on rst_sample/ rst_viewshed, out_srid/out_crs on the produce-new-raster + grid rasterize/spec functions (with Rule-2 geometry reprojection), the rst_{h3,quadbin}_rastertogrid silent-4326 correctness fix (auto-reproject to grid-native + crs override), and the heavy reader clipCrs option. Never-error invariant: absent CRS always degrades to a sensible assumption; only explicit conflict or explicit garbage raises. Co-authored-by: Isaac

…GridX Audited every function directly against Scala + Python sources: the recon table's tier labels were wrong — rst_sample, rst_viewshed, and rst_gridfrompoints(+_agg) are ALL both-tiers, not light-only. Added a verified tier-map section so the implementation plan isn't misled; every function is both-tiers except rst_h3_gridspec (light-only DataFrame helper). Added Group E: relocate gbx_h3_cell_bbox (a gbx_h3_* GridX-namespace function with no raster dependency) from the raster package to GridX, while doing its out_srid/out_crs CRS work here since it serves the rst_h3_* 100% goal (Q11-3). Co-authored-by: Isaac

…r cache Q12: the explicit srid/crs source param is a PER-GEOM fallback, not a per-row error — a geometry with an embedded SRID uses it, a plain-WKB geometry uses the param, and a mixed column (some EWKB, some WKB) is first-class. Only the call-level both-source-params / both-out-params / unresolvable-string conditions throw. Supersedes the earlier Q8 EWKB+param error. Q13/Q14: add a Performance section — a thread-local, LRU-bounded transformer cache in the centralized helpers (crs.get_transformer / SpatialRefOps.getTransformer), keyed by canonical CRS pair, always_xy, size _TRANSFORMER_CACHE_SIZE=128 (120 UTM zones + 4326/27700/3857 + headroom). No user-facing strict/permissive mode; correctness identical with or without the cache. Added mixed-column + cache tests. Co-authored-by: Isaac

The nuance of all decisions (two-role source/output model plus out_ naming, the per-geom source fallback for mixed columns, the never-error invariant and error matrix, produce-new-raster reprojection, rastertogrid auto-reproject, the transformer-cache perf note, and per-function cross-links) must land user-visible on coordinate-reference-systems.mdx, backed by runnable doc tests so the page cannot drift from behavior. Added as an itemized Group F deliverable with docs-test requirements, and elevated to goal (d). Co-authored-by: Isaac

Group F item 7: a single tabular enumeration of every CRS-touching function across all of GeoBrix (Package, Function, Tiers, CRS params, Role, CRS-in behavior, Notes). Seeded here with the complete RasterX surface (Groups A-E plus the shipped R/R2 functions); VectorX and GridX rows are appended by their own follow-on specs, which inherit the fixed column shape and source/output-role vocabulary. Every table row links to the function reference entry and vice versa. Recorded the append contract in the out-of-scope notes for both follow-on specs. Co-authored-by: Isaac

Task-by-task plan for the RasterX-CRS-100 spec. Two foundational tasks (light + heavy transformer cache + resolve_source_crs / resolveSourceSR), then Group C rastertogrid correctness fix, Group A source params (both tiers), Group B out_srid/out_crs + Rule-2 reprojection, gbx_h3_cell_bbox relocation to GridX, heavy reader clipCrs, and the CRS docs page + master cross-reference table. Shared-core code given verbatim; per-function param threading enumerated. Self-review covers spec coverage, arity-bump and rename risks, nearest-neighbour warp, and the light-tier no-heavy-deps constraint.

Task 1 of RasterX CRS to 100%. Two shared primitives in the light resolver: - get_transformer(src, dst): thread-local, LRU-bounded (_TRANSFORMER_CACHE_SIZE =128) pyproj Transformer keyed by canonical CRS pair, always_xy=True. Reuses one transformer per thread per CRS pair instead of rebuilding per row. - resolve_source_crs(embedded_srid, srid, crs): Rule-1 per-geom source-CRS resolution — embedded SRID wins; else single explicit param; both -> error; neither -> None. Mixed-column safe (embedded present ignores the param, no error). Consumed by Tasks 3-7. 34 tests green (7 new + crs_resolve/crs_ops regression). Co-authored-by: Isaac

…tion) Task 2 of RasterX CRS to 100%, heavy mirror of Task 1: - getTransformer(srcKey, dstKey): thread-local, LRU-bounded (cap 128) CoordinateTransformation keyed by canonical CRS pair. Reuses one transformation per thread per pair instead of rebuilding per row. - resolveSourceSR(embeddedSrid, srid, crs): Rule-1 per-geom source resolution — embedded SRID wins; else single explicit param; both -> error; neither -> None. Mixed-column safe. Consumed by Tasks 3/5/6. 12/12 SpatialRefOpsTest green; scalastyle clean. Co-authored-by: Isaac

Three pre-existing failures on branch/0.5.0 surfaced by CI (run 30925679475): - test_set_srid_rejects_nonpositive asserted set_srid(0) raises, but Spec R2 changed semantics to srid>=0 (0 clears the CRS). Split into test_set_srid_rejects_negative (-1 raises) + test_set_srid_zero_clears_crs. - bench REGISTRY was missing FnSpecs for rst_crs/rst_setcrs/rst_transformcrs (added as registered fns in Sub-spec R but never benched) -> full-set count 126 != 129 registered. Added the three specs (transformcrs core=False; the warp family's core representative is rst_transform). - test_pending_instructions _a_tif() IndexError'd when neither the sample-data bundle nor the Maven MODIS fixture is present (lightweight CI has no data); now pytest.skip() instead of crashing. 136 tests green locally (bench spec + core_ops + pending_instructions). Co-authored-by: Isaac

…tive (Group C) Task 3 of RasterX CRS to 100%. H3/quadbin raster-to-grid silently read a non-4326 raster's easting/northing as lon/lat (wrong cell assignments). Now the raster is auto-reprojected to grid-native EPSG:4326 (nearest-neighbour, so pixel statistics are never interpolated) when it carries a differing CRS — mirroring what BNG already does for 27700 — with an optional source override for a CRS-less-but-known raster. A CRS-less raster with no override is assumed grid-native and processed unchanged (never errors). Light: gridagg.raster_to_grid gains crs param + _warp_to_4326_if_needed; BNG path gains the same override. Heavy: shared GridReprojection.toGridCrs guard wired into RST_H3_RasterToGrid + RST_Quadbin_RasterToGrid (releases the warped Dataset). Light 5/5 new tests; heavy grid suites 25/25; cross-tier parity 21/21; scalastyle clean. Co-authored-by: Isaac

VizX already reprojects for basemaps/PMTiles but uses a parallel ds.crs read (not the canonical helper) and offers no crs override for a CRS-less raster, so a CRS-less-but-known raster gets a correct grid via Group C yet no correct basemap plot. Group G adds an optional crs source param to the plot entry points (plot_tile/cog/raster/file/static) plus routes VizX CRS through crs_to_canonical/resolve_crs. Never-error preserved (absent CRS yields a basemap-less plot, as today). Completes read to operate to visualize. Co-authored-by: Isaac

Task 10 covers VizX CRS consistency: a _resolve_plot_crs helper routing through crs_to_canonical/resolve_crs, an optional crs source param on the plot entry points (plot_tile/cog/raster/file/static), never-error preserved. The docs task (now Task 11) also seeds the VizX rows into the master cross-reference table. Self-review updated: seven groups + two foundations across 11 tasks. Co-authored-by: Isaac

…shed (Group A) Task 4 of RasterX CRS to 100%. A plain WKB/WKT geometry can now declare its source CRS on the light Column API: - rst_clip gains clip_crs; rst_sample and rst_viewshed gain crs. - edit.clip_to_geom / ops.sample / _viewshed_bytes route the source CRS through crs.resolve_source_crs (Rule 1: embedded EWKB SRID wins per-geom; else the explicit param incl. ESRI/WKT; else assume aligned) and reproject via the cached crs.get_transformer. Never errors on a CRS-less geom. - Replaces the old int-only CRS.from_epsg sites in sample/viewshed (ESRI/WKT source CRS now work). Params default None so SQL registration + existing calls are unchanged (arity-compatible). 24 + 56 + 136 tests green (source-crs, clip/edit/pipeline regression, Spark-UDF + SQL registration). flake8 clean. Co-authored-by: Isaac

…roup A) Task 5 of RasterX CRS to 100%, heavy mirror of Task 4: - RST_Clip gains a 4th arg clipCrs (String); execute resolves the cutline source CRS via SpatialRefOps.resolveCrs (embedded SRID wins, else clipCrs, else the raster CRS fallback) — ESRI/WKT cutline CRS now work. - RST_Sample gains a 3rd arg crs; doInvoke now reprojects the point from its source CRS to the raster CRS (previously heavy assumed pre-aligned) via a CoordinateTransformation; execute keeps its pre-aligned contract. - RST_Viewshed gains a 6th arg crs; observer point reprojected the same way. All three: builder accepts both the old arity and the new (crs) arity, so existing registered calls are unaffected. Never-error preserved (missing source or raster CRS / transform failure -> point/cutline used as-is). RST_CrsOpsTest 17 (+2), PixelOpsTest 11, AnalysisTest 4 green; scalastyle clean. Co-authored-by: Isaac

…ter (Group B, light) Task 6 (light, produce-new-raster family) of RasterX CRS to 100%: - features.rasterize_geom: rename srid -> out_srid, add out_crs; resolve target (out_crs wins, both -> error, neither -> geometry's carried source CRS). Rule-2: reproject the geometry from its source CRS (embedded EWKB SRID) into the output CRS before burning — fixes the silent-garbage case for an EWKB geom + differing output CRS. - tin.idw_grid / delaunay_dtm: srid -> out_srid + out_crs via shared _resolve_out_crs -> canonical string; _write_float64_grid now takes a CRS string (None -> CRS-less). Points are assumed already in the output CRS (label-only). - Column + agg wiring: rst_rasterize, rst_gridfrompoints(+_agg), rst_dtmfromgeoms(+_agg) gain out_srid/out_crs; UDFs guard int(None) and thread out_crs. Positional callers still bind srid->out_srid (backward-compatible). Grid rasterize_agg out_crs (B4) + heavy Group B are the remaining T6 pieces. 21 out_crs+tin tests green; flake8 clean. Co-authored-by: Isaac

Refactor cellraster to accept an output-CRS spec that is an int SRID OR a CRS string (per the user's request to do the refactor, not just the rename): - _reproject normalises both sides via pyproj CRS so the identity short-circuit survives across spellings (4326 == "EPSG:4326" == CRS(4326)). - _norm_out_crs / _is_geographic replace the srid==4326 pixel-size branch with a proper is_geographic check (any geographic CRS, not only 4326). - cells_to_raster stamps the canonical CRS string (EPSG/ESRI/WKT), not EPSG:{int}. Thread out_crs through _rst_{h3,quadbin,bng}_rasterize_agg_udf + rename srid -> out_srid and add out_crs on the three public functions (out_crs wins; grid-native default preserved; bng out params documented no-op, always 27700). 13 cellraster regression + 6 out_crs (incl. ESRI:54008) + h3-rasterize-validate green; flake8 clean. Co-authored-by: Isaac

…avy) Route the heavy vector->raster output-CRS construction through the shared SpatialRefOps.resolveCrs instead of raw ImportFromEPSG, so an ESRI code passed as the int srid (e.g. 54008) classifies correctly and stamps ESRI:54008 — matching the light tier's out_srid handling. Changed VectorRasterBridge.buildOgrLayer / buildEmptyRaster and RST_GridFromPoints.writeGeoJson. Heavy RST_Rasterize/gridfrompoints/dtm/grid-aggs use one int srid for the output CRS (geom assumed already in it — no source/target mismatch, so no Rule-2 reprojection needed on the heavy tier); a separate out_crs string param is not added (the int srid already accepts EPSG+ESRI via the resolver). Signatures unchanged — no arity/registration churn. 8 rasterize/gridfrompoints/polygonize + 18 RST_CrsOps (incl. new ESRI bridge test) green; scalastyle clean. Co-authored-by: Isaac

…roup B5, light) Task 7 of RasterX CRS to 100%: - gbx_h3_cell_bbox: rename srid -> out_srid, add out_crs (string wins). The _h3_cell_bbox_udf gains an optional out_crs; SQL-registered arity unchanged for the 4-arg form. - rst_h3_gridspec (light-only DataFrame helper): out_srid/out_crs; bbox coords computed in the resolved CRS, grid.srid INT field carries the EPSG code when the CRS has one (else out_srid); pixel-size geographic branch now keys on is_geographic (any geographic CRS), not srid==4326. - FIX cellraster._reproject: route int/int-like through resolve_crs so a bare ESRI int (54008) classifies as ESRI (pyproj's lenient from_user_input would reject EPSG:54008); _norm_out_crs passes a rasterio CRS through unchanged. 7 out_crs + h3_gridspec + cellraster regression green; flake8 clean. (Heavy RST_H3_CellBBox int srid already ESRI-correct via ImportFromEPSG; its string out_crs + relocation land together in Task 8.) Co-authored-by: Isaac

…tion deferred) Task 8 of RasterX CRS to 100%, reduced scope (Q(T8)-1 + Q(T8h)-1): the planned relocation of gbx_h3_cell_bbox to GridX is DEFERRED to the future H3-GridX phase -- neither tier has an H3 expression subpackage yet (light pygx has zero H3; heavy has gridx/grid/H3.scala math but no H3 expression dir), so moving it now would seed a lone one-function corner (+ a pyrx->pygx back-dep for rst_h3_gridspec). The function stays in place both tiers; registered SQL name gbx_h3_cell_bbox unchanged. CRS work done in place: RST_H3_CellBBox gains a 5th arg out_crs (String, wins over the int srid); output CRS resolved via SpatialRefOps.resolveCrs (ESRI codes + WKT/PROJ4 work, not just EPSG), identity short-circuit at grid-native 4326. Builder accepts 1-5 args (old arities unchanged); added a 5-arg Scala Column overload. 9/9 RST_H3_RasterizeAggTest (+2 new: out_crs web-mercator, ESRI srid); scalastyle clean. Co-authored-by: Isaac

… light) Task 9 of RasterX CRS to 100%. The heavy GDAL/GTiff reader now honours a clipCrs option (the light reader already had it): GDAL_Batch resolves the option to a canonical CRS string once (via SpatialRefOps.resolveCrs/crsToCanonical — ESRI/WKT work), threads it through GDAL_Partition to GDAL_Reader, and a new tileToRow overload stamps it into the v2 tile's clip_crs field (position 5). Absent -> None -> clip_crs stays null (never errors); an explicitly-unresolvable value raises. 4/4 GDAL_DataSourceTest (+2: clipCrs populates ESRI:54008, absent stays null); scalastyle clean. Co-authored-by: Isaac

…roup G) Task 10 of RasterX CRS to 100%. VizX now shares the canonical CRS authority instead of a parallel ds.crs read, and offers a crs override for a CRS-less raster (consistency with Group C's rastertogrid override): - _resolve_plot_crs(ds_crs, crs_override): raster CRS canonicalised via crs_to_canonical/resolve_crs (ESRI/WKT consistent); else the override; else None -> basemap-less plot. Never errors on absent CRS. - plot_tile and plot_cog gain an optional crs param, routed through the helper to the contextily basemap. Scope: plot_tile/plot_cog are the raster-over-basemap plots that read ds.crs; plot_raster/plot_file are basemap-less array renders (no CRS role); plot_static is vector (fixed 3857 + geopandas to_crs — VectorX CRS territory, out of scope). 4/4 _resolve_plot_crs tests + 255 vizx suite green; flake8 clean. (The 1 vizx failure, test_no_folium_import, is a pre-existing host-dep issue, not CRS.) Co-authored-by: Isaac

…/PROJ4 examples Task 11 (Group F) of RasterX CRS to 100%. Coordinate Reference Systems page gains: - "The four CRS-string forms" near the top: authority code / int-castable / WKT (a genuinely authority-less custom Transverse Mercator) / PROJ4 (Albers), all verified to resolve — makes WKT + PROJ4 prominent per request. - "Source CRS vs output CRS": the two-role model + srid/crs/clip_crs (source) vs out_srid/out_crs (output) naming standard. - Per-geom source resolution + first-class mixed columns; the never-error invariant note (only 3 conditions raise); produce-new-raster Rule-2 reprojection; rastertogrid auto-reproject-to-grid-native; a performance note (internal transformer caching — no need to pre-warp). - Master "CRS function reference" table: every CRS-touching function across GeoBrix with param + role, each linked to its reference entry; VectorX/GridX rows deferred with a note. Plus per-function CRS-page links + updated signatures on rst_clip/sample/ viewshed/rasterize/gridfrompoints/dtmfromgeoms in raster-functions.mdx. Voice-grep clean; docs build green (MDX compiles, links + new anchors resolve). Co-authored-by: Isaac

Explains grid-shift datum/vertical transforms (NTv2/NADCON/PROJ .tif grids; NAD27<->NAD83, OSGB36<->ETRS89 via OSTN15): reprojection delegates to PROJ so any grid-referencing WKT/PROJ4 CRS is honored when the grid is found; how PROJ finds grids (PROJ_DATA search path + PROJ_NETWORK=ON CDN); a prominent warning that a missing grid degrades SILENTLY to lower accuracy (does not error) + how to confirm; and a Serverless/no-egress note pointing at Volume-staged grids and a PROJ_DATA prepend (with the standardized customer-grid mechanism flagged as a planned follow-up). Docs build green; links resolve; voice-grep clean. Co-authored-by: Isaac

…inal review) Final whole-branch review of RasterX CRS-100 found a native-memory leak in RST_Sample.reprojectToRaster and RST_Viewshed.reprojectObserver (Task 5): the resolved source SpatialReference was deleted only on the transform branch, not when source==raster CRS (IsSame==1), leaking one SpatialReference per identity-CRS sample/viewshed call. Moved sr.delete() into a finally so it's released on every path. 33/33 heavy Sample/Viewshed/CrsOps tests green; scalastyle clean. Co-authored-by: Isaac

Design for VectorX's CRS-string complement to the product's int-SRID built-ins (never duplicating st_srid/st_setsrid/st_transform), both tiers, mirroring the RasterX rst_crs/rst_setcrs/rst_transformcrs split with geometry-honest semantics: - st_transformcrs(geom, target_crs, source_crs=None): reproject to an EPSG/ESRI/ WKT/PROJ4 string; optional source_crs for plain WKB/WKT; a 12-cell return matrix (encoding follows input text-vs-binary; carried SRID follows the target — authority-coded upgrades plain->E and carries the int code, authority-less yields the plain form with any stale SRID cleared). - st_setcrs(geom, crs): string->SRID stamper, EPSG/ESRI only (WKT/PROJ4 raise — a geometry can't hold an authority-less CRS); encoding-preserving. - st_crs(geom): embedded SRID -> canonical EPSG/ESRI string; plain WKB/WKT -> null. Architecture: Q4-2 tier-neutral resolver refactor (extract pyrx.core.crs -> gbx.core.crs + heavy SpatialRefOps -> neutral home, with re-export shims so the shipped CRS-100 importers keep working) as a prerequisite task; shared parse_geom + an encoding classifier driving the matrix; never-error invariant. Co-authored-by: Isaac

CI (run 30999099989) caught a missed call site from Task 6a's srid -> out_srid rename: the bench REGISTRY FnSpec for rst_h3_rasterize_agg still passed srid= as a kwarg -> "unexpected keyword argument 'srid'". Changed to out_srid=. The quadbin/bng/rasterize/gridfrompoints/dtm bench col_fns were unaffected (they pass srid positionally, which binds to the renamed out_srid in the same slot, or omit it). 92 bench tests (incl. the failing runner test) green; flake8 clean. Co-authored-by: Isaac

Task-by-task plan for the VectorX-CRS spec. T1/T2 extract the CRS resolver to a tier-neutral home (light gbx.core.crs + pyrx shim; heavy neutral SpatialRefOps + rasterx forwarder) so pyvx shares one authority without duplicating the shipped CRS-100 resolver — regression gate is the existing CRS suites passing unchanged through the shims. T3/T4 build st_crs/st_setcrs/st_transformcrs light/heavy over the shared resolver + parse_geom/JTS I/O, driving output via the Q6 return matrix; st_setcrs uses to_authority() (EPSG+ESRI), authority-less raises. T5 = cross-tier parity + binding parity + docs (incl. the deferred VectorX rows in the CRS-page master table). Self-review flags the resolver-relocation import risk, the bytes-or-str UDF return-type nuance, EWKT round-trip, and heavy SR release. Co-authored-by: Isaac

…pyrx shim) Task 1 of the VectorX CRS family. Moved pyrx/core/crs.py verbatim (git mv, behavior-preserving) to the new tier-neutral databricks.labs.gbx.core package so the lightweight packages (pyrx AND the upcoming pyvx CRS family) share one CRS authority instead of forking it. pyrx/core/crs.py is now a re-export shim (public API + the privates the CRS-100 tests reference: _TRANSFORMER_CACHE_SIZE, _is_intlike, _transformer_cache, etc.), so all 20 existing importers keep working unchanged. 36 tests green (new neutral + shim test + full CRS-100 regression through the shim); all importers smoke-import; flake8 clean. Co-authored-by: Isaac

…arder) Move the real SpatialRefOps implementation from rasterx.operations to the new tier-neutral com.databricks.labs.gbx.operations package. VectorX expressions (Task 4) need CRS resolution without depending on the rasterx package; this gives them a clean import. rasterx.operations.SpatialRefOps becomes a thin forwarder — explicit def delegations for all six public members — so the ~12 existing rasterx importers (RST_Clip, RST_Sample, RST_Viewshed, RST_SetSrid, RST_SetCrs, RST_TransformCrs, RST_Crs, RST_H3_CellBBox, RST_GridFromPoints, VectorRasterBridge, GDAL_Batch, GDALRasterize) compile and pass unchanged. SpatialRefOpsTest extended with 10 dual-path assertions confirming the neutral path and the forwarder return identical results for resolveCrs/crsToCanonical/ getEPSGCode/fromEPSGCode/resolveSourceSR. All 22 tests pass; scalastyle clean.

Add INPUT_TILE to _PREAMBLE and pass input_tile=INPUT_TILE to both run_spark_path call sites in run_light (explain-only path + regular _todo path), so the generated notebook propagates the virtual vs materialized tile choice to the spark-path runner. Also sync ORDER with the two ResultRow fields Task 4 added (input_tile, output_disposition) so cluster.py no longer raises at import time. Co-authored-by: Isaac

Expose --input-tile {materialized,virtual} (default materialized) on gbx:bench:cluster so the virtual-tile spark-path leg can be exercised on-cluster without changing code. Parses via the existing _arg() helper, validates choices, and threads the value into cfg["input_tile"] for build_bench_notebook to consume. Co-authored-by: Isaac

Add test_spark_virtual_leg_smoke to exercise the full virtual-leg pipeline over Spark: disposition-sampling (Task 5) and creation micro-leg (Task 7) together. Asserts rst_slope→materialized, rst_setsrid→deferred, rst_avg→materialized, rst_fromfile→deferred, all per_tile_avg_s >= 0, and all rows tagged input_tile="virtual". Co-authored-by: Isaac

I1: gate spark-path input_tile="virtual" tag block on input_kind=="tile"; non-tile fns (agg/tile_array/bytes/path/geometry) no longer get falsely stamped virtual — they keep the truthful materialized/na defaults. I2: add test_pure_core_parity_full_tile_set covering 103 tile-input fns; all 103 match (0 divergences). Also fixes run_pure_core_parity to guard fingerprinting on fingerprint=False (was crashing on rst_boundingbox's WKB output being opened as raster bytes). Q1: replace _parity_fp (identical duplicate) with _fingerprint_for in run_pure_core_parity and delete the dead function. Q2: results.summarize virtual-tile section now counts all 3 disposition states (deferred/materialized/na) to match the per-fn table below. Co-authored-by: Isaac

A QA pass over the virtual-tile bench leg found five defects: - 5 tile-multiplying fns (maketiles/retile/tooverlappingtiles/separatebands/polygonize) were specced as scalars, but pyrx registers them as UDTFs, so spark-path errored -- flag udtf=True so they time via SQL LATERAL. - The disposition sampler called the scalar col_fn for every fn, so all UDTFs recorded "na" -- sample UDTFs via LATERAL and infer materialized for flat-output UDTFs. - rst_contour and rst_isempty read pixels but were classified "deferred" -- pin/allowlist them materialized. - rst_polygonize emits geometry (no tile to sample) -- pin virtual_disposition=materialized. - run_spark_path never routed the role=bng_gb tile to the 9 BNG gb_tile fns (pure-core and the Scala heavy tier do), so BNG spark-path binned an empty out-of-GB grid -- build a GB-tile DataFrame and route gb_tile fns to it. Adds regression + registry-wide guard tests (17 pass). Co-authored-by: Isaac

Part A: the Raster->grid table gains Light virtual/tile + Disposition columns; the 9 BNG rows are refreshed to real Great-Britain-cell numbers (the lightweight spark-path had been binning an empty out-of-GB grid), speedups recomputed, table re-sorted, and the prose corrected (rst_bng_tessellate is heavyweight-favored, not the empty-tile 1.81x). Part B: new "Virtual vs materialized input tiles" subsection (74 fns) documenting the deferred lazy-win -- set-CRS/SRID/nodata/band-select run 4.7-12.5x faster on a virtual tile (no pixel bytes shuffled, raster never opened), while pixel-reading ops pay a small open cost. Co-authored-by: Isaac

plot_raster/plot_file gained three optional kwargs; defaults reproduce the prior behavior byte-for-byte (a back-compat gate), so existing callers and tests are unchanged. - bands=: 1-based band select/reorder (matches rst_frombands order); 1 band renders viridis, >=3 composites RGB from the first three. - stretch="perband"(default)|"shared"|(lo,hi): an independent per-band percentile stretch desaturates strongly-correlated EO RGB to grey; "shared" applies one pooled stretch across the bands to preserve colour balance. - fill=: value masked as NoData for display (excluded from the stretch and drawn transparent) even when the tile has no nodata set. The enhanced RGB path normalizes uint8 (/255) and out-of-range float data to [0,1] before compositing so it is not hard-clipped to white; a 2-band or empty selection raises a clear ValueError. Adds render tests (51 total in vizx). Co-authored-by: Isaac

NB04 renders stacked/clipped tiles as true colour via plot_raster(bands=(1,2,3), stretch="shared", fill=0) with a matching narrative note; NB03 filters empty tiles before sampling so an all-NoData H3 cell no longer shows as a blank grey box. config_nb installs the wheel with --no-cache-dir so a same-version rebuild is actually picked up on Serverless. eo-series.mdx + README updated for the virtual-tile ingestion path. Co-authored-by: Isaac

Pure formatting (isort import order + black) to match CI; no functional change. Clears the isort drift that failed CI lint on these two bench test files. Co-authored-by: Isaac

PR #72's build failed on `black --check`. CI lints as `isort && black && flake8 src test`, so black's failure short-circuited before flake8 — hiding pre-existing flake8 debt that surfaces once black passes. This clears both in one pass: - black-reformat 7 drifted files (bench/{runner,spec}.py, test/bench/{cluster, results,spec}, test/pyrx/{core_agg,v2_tile_output_invariant}). - flake8: drop 3 unused imports (F401 in pyrx/_serde.py, test/rasterx/ test_udtf_error_row.py, test/vectorx/test_crs.py) and mark the bench helper run_spark_path with # noqa: C901 (matches the _render precedent). No functional change; none of these files were part of the vizx/eo-series work — this is latent tree-wide lint debt that was masked by the black short-circuit. Co-authored-by: Isaac

Michael Johns added 30 commits August 4, 2026 11:36

mjohns-databricks added 10 commits August 12, 2026 17:31

feat(bench): capture + tag virtual-tile output disposition

6f45d65

feat(bench): pure-core virtual==materialized parity sweep

15913ab

feat(bench): rst_fromfile virtual-tile creation micro-leg

cfacdab

feat(bench): summary shows disposition + QA anomalies

5527b35

mjohns-databricks temporarily deployed to runtime August 13, 2026 14:35 — with GitHub Actions Inactive

mjohns-databricks had a problem deploying to runtime August 13, 2026 15:02 — with GitHub Actions Failure

mjohns-databricks added 3 commits August 13, 2026 16:44

style(bench): isort/black reformat test files

d8f0496

Pure formatting (isort import order + black) to match CI; no functional change. Clears the isort drift that failed CI lint on these two bench test files. Co-authored-by: Isaac

mjohns-databricks temporarily deployed to runtime August 13, 2026 20:45 — with GitHub Actions Inactive

mjohns-databricks temporarily deployed to runtime August 13, 2026 20:46 — with GitHub Actions Inactive

mjohns-databricks temporarily deployed to runtime August 13, 2026 21:12 — with GitHub Actions Inactive

mjohns-databricks had a problem deploying to runtime August 13, 2026 21:21 — with GitHub Actions Failure

mjohns-databricks temporarily deployed to runtime August 13, 2026 22:26 — with GitHub Actions Inactive

mjohns-databricks temporarily deployed to runtime August 13, 2026 22:54 — with GitHub Actions Inactive

mjohns-databricks deployed to runtime August 13, 2026 22:55 — with GitHub Actions Active

Navigation Menu

Uh oh!

release: GeoBrix 0.5.0 — virtual tiles, v2 tile struct, CRS families - #72

release: GeoBrix 0.5.0 — virtual tiles, v2 tile struct, CRS families#72
mjohns-databricks wants to merge 501 commits into
mainfrom
beta/0.5.0

mjohns-databricks commented Aug 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

mjohns-databricks commented Aug 13, 2026

GeoBrix 0.5.0 — release PR (beta/0.5.0 → main)

Highlights

Pre-merge gate (requested)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

GeoBrix 0.5.0 — release PR (`beta/0.5.0` → `main`)