Streaming MEOS-parity harness and NebulaStream adapter#44
Draft
estebanzimanyi wants to merge 254 commits into
Draft
Streaming MEOS-parity harness and NebulaStream adapter#44estebanzimanyi wants to merge 254 commits into
estebanzimanyi wants to merge 254 commits into
Conversation
The PAIR_MEETING aggregation (added in MobilityDB#17) hardcoded the meeting-distance threshold at 200 m via a static constexpr DMEET_METRES, with the PR body noting parameterization as future work. This PR lands that future work: PAIR_MEETING now takes a fifth argument — a numeric constant in metres — and the physical operator uses it per-query. ## Surface PAIR_MEETING(lon, lat, ts, vehicle_id, dMeet) ^^^^^ new fifth arg (numeric constant, metres) The first four args remain FieldAccess (lon, lat, ts, vehicle_id); the fifth is pulled from the parser's constantBuilder as a numeric literal, parsed via std::stod, and threaded through the logical→physical lowering chain into the lower() lambda alongside the existing state pointers. ## Files (9, all stacked on MobilityDB#18 → MobilityDB#17 → MobilityDB#16 → MobilityDB#15) | Layer | File | |---|---| | Physical .hpp | PairMeetingAggregationPhysicalFunction.hpp — `DMEET_METRES` constexpr → `DEFAULT_DMEET_METRES` + instance field `dMeetMetres` | | Physical .cpp | PairMeetingAggregationPhysicalFunction.cpp — constructor takes dMeet; lower() passes it to the captureless lambda via `nautilus::val<double>` | | Logical .hpp | PairMeetingAggregationLogicalFunction.hpp — constructor + create() factory take dMeet; getter `getDMeetMetres()` | | Logical .cpp | PairMeetingAggregationLogicalFunction.cpp — initialize field; Registrar deserialize path uses DEFAULT_DMEET_METRES (see Serde caveat below) | | Parser | AntlrSQLQueryPlanCreator.cpp — both PAIR_MEETING dispatch sites (lexer-token case + funcName string-name case) extract the constant from constantBuilder, std::stod it, pass to create() | | Lowering | LowerToPhysicalWindowedAggregation.cpp — pmDescriptor->getDMeetMetres() flows to the physical constructor | | YAMLs (×3) | Queries/berlinmod/q5_continuous.yaml, q5_snapshot.yaml, q5_windowed.yaml — add `, 200.0` as the explicit fifth arg; comments updated to reflect the parameterization | ## Serde round-trip caveat (out of scope for this PR) `AggregationLogicalFunctionRegistryArguments` is strongly typed to `vector<FieldAccessLogicalFunction>` — there is no slot for a numeric constant in the existing Registrar interface, and `SerializableAggregationFunction` has no proto field for it either. As a result: - The parser path (live query execution) is FULLY parameterized — dMeet flows from SQL to physical correctly. - The Serde deserialize path falls back to DEFAULT_DMEET_METRES (preserves the 200 m scaffold behaviour). Round-trip fidelity for the dMeet value requires (a) adding a new field to SerializableAggregationFunction.proto, (b) extending AggregationLogicalFunctionRegistryArguments to carry it, and (c) threading both through Serialize/Register. That's an infrastructure change touching every registered aggregation; tracked as a follow-up. ## Build / test verification Cannot compile-verify locally — NebulaStream needs the full C++23 + vcpkg toolchain. Submitted for maintainer build verification (cc @marianaGarcez). Expected to compile cleanly; the only construction-time behaviour change is the constructor signature (5 params → 6 params for physical, 5 → 6 for logical create/ctor); the only runtime behaviour change is that dMeet is now read from the instance field instead of the class constexpr (the lambda receives it via the nautilus::val<double> extra arg). ## Mirrors the CROSS_DISTANCE shape CROSS_DISTANCE (also added by MobilityDB#17, hardcoded VID_A=100, VID_B=200) has the exact same parameterization pattern; a sibling PR can apply the same change with (lon, lat, ts, vid, vid_a, vid_b) — 6 args total instead of 5. Holding for separate PR.
… args Sibling to PAIR_MEETING.dMeet parameterization (PR MobilityDB#19) — applies the same 4-layer pattern to CROSS_DISTANCE. The aggregation (added in MobilityDB#17) hardcoded the target vehicle pair at (100, 200) via static constexpr VID_A / VID_B, with the PR body noting parameterization as future work. This PR lands that future work: CROSS_DISTANCE now takes two unsigned- integer constants as its fifth and sixth arguments, and the physical operator uses them per-query. ## Surface CROSS_DISTANCE(lon, lat, ts, vehicle_id, vidA, vidB) ^^^^ ^^^^ new constants (uint64) The first four args remain FieldAccess; vidA and vidB are pulled from the parser's constantBuilder (two unsigned-integer literals), std::stoull them, and threaded through the logical→physical lowering chain into the lower() lambda alongside the existing state pointer. ## Files (9, same shape as PR MobilityDB#19's PAIR_MEETING change) | Layer | File | |---|---| | Physical .hpp | CrossDistanceAggregationPhysicalFunction.hpp — `VID_A/B` constexpr → `DEFAULT_VID_A/B` + instance fields `vidA/B` | | Physical .cpp | CrossDistanceAggregationPhysicalFunction.cpp — constructor takes both; lift-time lambda gets them via `nautilus::val<uint64_t>` | | Logical .hpp | CrossDistanceAggregationLogicalFunction.hpp — constructor + create() factory + getters | | Logical .cpp | CrossDistanceAggregationLogicalFunction.cpp — initialize fields; Registrar deserialize falls back to defaults | | Parser | AntlrSQLQueryPlanCreator.cpp — both CROSS_DISTANCE dispatch sites extract two constants, std::stoull both, pass to create() | | Lowering | LowerToPhysicalWindowedAggregation.cpp — cdDescriptor->getVidA()/getVidB() flow to physical constructor | | YAMLs (×3) | Queries/berlinmod/q9_continuous.yaml, q9_snapshot.yaml, q9_windowed.yaml — add `, 100, 200` as explicit constants; comments updated | ## Serde round-trip caveat (same as PR MobilityDB#19) `AggregationLogicalFunctionRegistryArguments` is strongly typed to `vector<FieldAccessLogicalFunction>` — no slot for integer constants. `SerializableAggregationFunction.proto` has no field for them. So: - Parser path (live query execution) is FULLY parameterized. - Serde deserialize path falls back to `DEFAULT_VID_A` / `DEFAULT_VID_B` (preserves the 100, 200 scaffold defaults). Same infrastructure follow-up would close both round-trip gaps at once (PAIR_MEETING.dMeet and CROSS_DISTANCE.vidA/vidB). ## Build / test verification Same as PR MobilityDB#19 — submitted for maintainer build verification (@marianaGarcez). Constants now flow through std::stoull instead of std::stod; lambda gets two nautilus::val<uint64_t> args instead of one nautilus::val<double>. Pattern is structurally identical.
…codegen path Closes the Nebula structural parity gap with Flink/Kafka by shipping the codegen infrastructure for generating per-MEOS-function pipeline tuples (logical + physical + parser + lowering). No generated C++ committed in this PR — the maintainer (cc @marianaGarcez) runs the generator on a chosen MEOS-function batch, reviews output, ships operators in follow-up PRs at a controlled pace. Why no generated code in this PR: - Generator author cannot build NebulaStream (full C++23 + vcpkg toolchain not available in author's environment); shipping unverified generated code would risk batched-broken operators. - Per-function review value: maintainer iterates on templates with the first batch's build feedback before scaling up. - Template iteration cost: first-pass templates may need adjustment after first build; smaller blast radius if only the generator lands. What lands: - tools/codegen/codegen_nebula.py — Python generator with embedded C++ templates derived 1:1 from the hand-written TemporalEDWithinGeometry operator shape (logical/physical/.hpp/.cpp) - tools/codegen/codegen_input.example.json — first-wave input list (5 spatial-relation E/A predicates: EDisjoint, ATouches, ECovers, ACrosses, EOverlaps over tgeo_geo) - tools/codegen/README.md — full design proposal: why codegen, what the generator produces, recommended scaling-wave sequence (W1-W5), what the generator does NOT do (CMakeLists / parser / grammar remain manual paste for idempotence), compile-verification note Smoke-verified: the generator runs locally + emits 5 operators × 4 files = 20 well-formed C++ source files; templates produce syntactically-reasonable output matching the existing operator style. Scaling path (recommended sequence): - W1: 5 spatial-relation E/A predicates (the example input) — first follow-up PR - W2: All ever/always spatial-relation predicates over tgeo_geo (~18 functions) — second follow-up PR - W3: Distance functions over tgeo_geo and tgeo_tgeo (~30) — third - W4: Scalar accessors that decompose to per-event reads — template extension required - W5: Aggregations (windowed/cross-stream) — separate generator with the aggregation-specific 4-layer pattern Stacks on PR MobilityDB#20. Tools-only; touches no operator code, no CMakeLists, no parser/grammar.
Two adjacent compile-breakers found while validating the codegen output of PR MobilityDB#21 against the latest mariana/main: 1. SerializableAggregationFunction proto declares only {type, on_field, as_field}. The 5 MEOS aggregations landing in MobilityDB#16/MobilityDB#17 read additional fields out of the proto (vidA/vidB/dMeet/...), so they need the extra field. Adds: repeated SerializableFunction extra_fields = 4; Backwards-compatible (tag 4, new repeated). Aggregations whose extra fields are absent continue to deserialize unchanged. 2. CrossDistance/PairMeeting/TemporalLength aggregations carry an unused PipelineMemoryProvider& parameter on lower(). Werror=-Wunused-parameter turns that into a build failure. Annotates the parameter [[maybe_unused]] at the call site — no behavior change, intent stays visible to readers who later wire memory into the lowering. Verified locally on the mobilitynebula-v2 dev image (MEOS baked in): cmake --build build-w1 --target nes-physical-operators -j 4 → [110/111] Linking libnes-physical-operators-registry.a → [111/111] Linking libnes-physical-operators.a Stacks on MobilityDB#21 only because that is the active codegen branch where the breakage surfaced; the diff itself is independent of any codegen output.
…geom) First batch of MEOS operators generated by the PR MobilityDB#21 codegen, covering the spatial-relation family over (tgeo, geometry). Five operators landed, one per relation pattern: edisjoint_tgeo_geo → TemporalEDisjointGeometry atouches_tgeo_geo → TemporalATouchesGeometry ecovers_tgeo_geo → TemporalECoversGeometry acontains_tgeo_geo → TemporalAContainsGeometry etouches_tgeo_geo → TemporalETouchesGeometry Each operator is emitted at all four layers — logical .hpp/.cpp + physical .hpp/.cpp — same shape mariana's hand-written eContainsGeometry operator uses, so the runtime sees them as ordinary plugin operators with no special wiring. Generator tightenings landed alongside the output (kept inside tools/codegen so they remain re-runnable): * physical Registrar reads PhysicalFunctionRegistryArguments.childFunctions (the actual field name; the previous template used .children which only exists on the logical side). * VariableSizedData is accessed through .getContent() / .getContentSize() (the real API; the previous template used .getRawByteRef() / .size() which do not exist). * The MEOS spatial-rel signature is 2-arg (Temporal*, GSERIALIZED*) — no trailing atstart bool. The 3-arg distance form lives only on edwithin_tgeo_geo and edwithin_tgeo_tgeo and stays out of W1. * tools/codegen/codegen_input.example.json now references real MEOS symbols (etouches_tgeo_geo, acontains_tgeo_geo). The earlier eoverlaps_tgeo_geo / acrosses_tgeo_geo entries were placeholders and would not link. Verified locally on the mobilitynebula-v2 dev image (MEOS baked in): cmake --build build-w1 --target nes-logical-operators -j 4 cmake --build build-w1 --target nes-physical-operators -j 4 → both link clean. The 5 new operators compile and register at both layers. Stacks on PR-A (proto extra_fields + Werror unused-param) and PR MobilityDB#21 (the codegen itself). The same generator scales to W2 (e/a spatial-rels over tgeo × tgeo, ~10 ops) and W3 (distance functions over tgeo × geo + tgeo × tgeo, ~30 ops) with no further template work — that is the path the 9 BerlinMOD-query recipes open beyond the surface metric.
Adds the 2 remaining publicly-declared 2-arg spatial-rel ops over
(tgeo, geometry) not yet covered by W1 + mariana's seeds:
adisjoint_tgeo_geo → TemporalADisjointGeometry
eintersects_tgeo_geo → TemporalEIntersectsGeometry
Combined with the prior layers, the public-API _tgeo_geo spatial-rel
row is now complete for the 2-arg shape:
e: econtains ecovers edisjoint eintersects etouches (5/5)
a: acontains adisjoint aintersects atouches (4/4)
Provenance per layer:
- mariana seeds: TemporalEContainsGeometry, TemporalAIntersectsGeometry,
TemporalEDWithinGeometry (3-arg), TemporalIntersectsGeometry
- W1 (PR MobilityDB#23): edisjoint, atouches, ecovers, acontains, etouches
- W2 (this PR): adisjoint, eintersects
The 3-arg dwithin pair (edwithin / adwithin) is excluded from the
2-arg shape and stays out of this PR.
Note on acovers_tgeo_geo: the symbol exists in libmeos.so but has
no public declaration in meos_geo.h (libmeos-internal only), so it
is correctly out of scope for a binding-level PR.
Local verification on the mobilitynebula-v2 dev image:
cmake --build build-w1 --target nes-physical-operators -j 4
→ [161/161] Linking libnes-physical-operators.a
cmake --build build-w1 --target nes-logical-operators -j 4
→ [43/43] Linking libnes-logical-operators.a
Same generator, no template changes, just 2 more input rows — the
mechanical-scale path the 9 BerlinMOD-query recipes open.
… (9 ops)
Closes the public-API _tgeo_tgeo 2-arg spatial-relation row by emitting
all 9 publicly-declared ops as Nebula operators (one new op per relation,
per e/a quantifier). The MEOS signature is
`int fn(const Temporal*, const Temporal*)`, so each operator builds
TWO single-instant tgeompoints from event fields (lonA/latA/tsA +
lonB/latB/tsB) before invoking MEOS:
econtains_tgeo_tgeo → TemporalEContainsTGeometry
ecovers_tgeo_tgeo → TemporalECoversTGeometry
edisjoint_tgeo_tgeo → TemporalEDisjointTGeometry
eintersects_tgeo_tgeo → TemporalEIntersectsTGeometry
etouches_tgeo_tgeo → TemporalETouchesTGeometry
acontains_tgeo_tgeo → TemporalAContainsTGeometry
adisjoint_tgeo_tgeo → TemporalADisjointTGeometry
aintersects_tgeo_tgeo → TemporalAIntersectsTGeometry
atouches_tgeo_tgeo → TemporalATouchesTGeometry
The 3-arg dwithin pair (edwithin / adwithin) stays out — same as in
W1/W2, they belong to a separate distance-arg template branch.
Generator extension
-------------------
This is the first PR where the codegen ships a NEW template branch in
addition to new rows. Adds:
* PHYSICAL_CPP_TEMPLATE_TWO_TEMPORAL_POINTS — mirrors the one-temporal-point
template, but with two single-instant tgeompoints and no static
geometry argument.
* `build_two_temporal_points` boolean flag on operator descriptors,
dispatched alongside `build_temporal_point` in `emit_operator`.
No existing template paths change. Row totals:
| family | _tgeo_tgeo (2-arg) ops in meos_geo.h | shipped |
|--------|--------------------------------------|---------|
| e/* | econtains, ecovers, edisjoint, eintersects, etouches | 5/5 |
| a/* | acontains, adisjoint, aintersects, atouches | 4/4 |
Local verification on the mobilitynebula-v2 dev image:
cmake --build build-w1 --target nes-physical-operators -j 4
→ [38/38] Linking libnes-physical-operators.a
cmake --build build-w1 --target nes-logical-operators -j 4
→ [52/52] Linking libnes-logical-operators.a
Both targets link clean on the first attempt — the template extension
worked without iteration, validating the generator approach for the next
shape (distance functions, 3-arg signature).
…mplates)
Closes the public-API distance-function row over (tgeo, geo) and
(tgeo, tgeo). Two distinct measure types, both built from the same
event-field shape used by W1/W2/W3:
Scalar measure — `nad_*` (nearest-approach distance, double return):
nad_tgeo_geo → TemporalNADGeometry
nad_tgeo_tgeo → TemporalNADTGeometry
Thresholded test — `*dwithin_*` (3-arg, int return):
edwithin_tgeo_tgeo → TemporalEDWithinTGeometry
adwithin_tgeo_geo → TemporalADWithinGeometry
adwithin_tgeo_tgeo → TemporalADWithinTGeometry
`edwithin_tgeo_geo` is already shipped as mariana's `TemporalEDWithinGeometry`
seed, so the (e/a × tgeo_geo/tgeo_tgeo) dwithin square is now complete.
Row totals after this PR (publicly-declared in meos_geo.h):
| shape | covered |
|-----------------------|------------------------|
| nad_tgeo_geo | 1/1 ✅ |
| nad_tgeo_tgeo | 1/1 ✅ |
| edwithin_tgeo_geo | 1/1 (mariana seed) ✅ |
| edwithin_tgeo_tgeo | 1/1 ✅ |
| adwithin_tgeo_geo | 1/1 ✅ |
| adwithin_tgeo_tgeo | 1/1 ✅ |
Generator extension
-------------------
Two new template branches; existing branches untouched:
* PHYSICAL_CPP_TEMPLATE_TEMPORAL_POINT_WITH_DIST
— one-tgeo + static geometry + trailing `double dist` (5 args).
* PHYSICAL_CPP_TEMPLATE_TWO_TEMPORAL_POINTS_WITH_DIST
— two-tgeo + trailing `double dist` (7 args).
Dispatch in `emit_operator` extends the existing if/elif chain with
`build_temporal_point_with_dist` and `build_two_temporal_points_with_dist`
flags. NAD reuses the existing temporal-point / two-temporal-points
branches with no template change — only `return_type="double"` and
`nautilus_return="FLOAT64"` differ at the operator-descriptor level.
Local verification on the mobilitynebula-v2 dev image:
cmake --build build-w1 --target nes-physical-operators -j 4
→ [43/43] Linking libnes-physical-operators.a
cmake --build build-w1 --target nes-logical-operators -j 4
→ [57/57] Linking libnes-logical-operators.a
Both targets link clean on the first attempt.
…0 dispatch cases) Extends the codegen to back-fill the SQL-parser glue that the W1–W4 PRs (MobilityDB#23–MobilityDB#26) shipped without — so the 21 generated operators become SQL-invokable end-to-end instead of just runtime-registered plugins waiting for manual wiring. What the codegen now writes --------------------------- After emitting the .hpp/.cpp files, the codegen idempotently injects into the existing in-tree files: * nes-sql-parser/AntlrSQL.g4 - lexer-token entries (TOKEN: 'TOKEN' | 'token';) bracketed with /* BEGIN/END CODEGEN LEXER TOKENS */ marker - functionName: alternation list updated with new tokens * nes-sql-parser/src/AntlrSQLQueryPlanCreator.cpp - #include <Functions/Meos/XxxLogicalFunction.hpp> per op - case AntlrSQLLexer::TOKEN: { ... } dispatch block per op, bracketed with /* BEGIN/END CODEGEN PARSER GLUE: TOKEN */ * nes-{logical,physical}-operators/src/Functions/Meos/CMakeLists.txt - add_plugin(NebulaName {Logical,Physical}Function ...) per op Idempotency: every per-op injection skips when either the codegen marker is present OR a pre-existing hand-written case (no marker) is already in the file. Re-running the codegen on the same input is a no-op for the parser side; only the .hpp/.cpp emitters re-write deterministically. Two opt-out CLI flags: --no-parser-glue skip .g4 + parser .cpp injection --no-cmake-entries skip CMakeLists.txt injection Four dispatch-case templates by shape ------------------------------------- * one tgeo + static geom (4 args: lon, lat, ts, geom) * two tgeos (6 args: lonA, latA, tsA, lonB, latB, tsB) * one tgeo + static geom + dist (5 args: lon, lat, ts, geom, dist) * two tgeos + dist (7 args: lonA, latA, tsA, lonB, latB, tsB, dist) The constantBuilder→functionBuilder lift mirrors mariana's pattern from TGEO_AT_STBOX and EDWITHIN_TGEO_GEO (TRUE/FALSE → BOOLEAN, strtod-clean → FLOAT64, else → VARSIZED), so distance literals and WKT literals deserialize the same way the hand-written ops do. Back-fill: 20 new dispatch cases + 21 includes + 20 lexer tokens ---------------------------------------------------------------- Ran the codegen against the combined W1+W2+W3+W4 input (21 ops). One of the 21 (TEMPORAL_EINTERSECTS_GEOMETRY) was already wired manually by mariana so the codegen detected and skipped it; 20 cases injected clean. nes-sql-parser links green with the regenerated ANTLR lexer + parser stubs. Local verification on the mobilitynebula-v2 dev image: cmake --build build-w1 --target nes-sql-parser -j 4 → links clean cmake --build build-w1 --target nes-logical-operators -j 4 → up to date cmake --build build-w1 --target nes-physical-operators -j 4 → up to date What this unlocks ----------------- The 21 W1–W4 operators are now SQL-invokable end-to-end. From now on, every codegen PR ships parser glue in-PR by default (per the `--no-parser-glue` opt-out, which is OFF by default). The path past the spatial-rel surface (W5 tnumber scalar, W5b extended types, W7 aggregations) inherits the closed loop.
…stests)
First-batch tnumber-shape operators. The MEOS surface for nearest-approach
distance over tnumber types is small (4 publicly-declared ops in meos.h
beyond the TBox-arg variants, which are deferred):
nad_tfloat_float → TemporalNADFloatScalar (3 args: value, ts, scalar)
nad_tint_int → TemporalNADIntScalar (3 args: value, ts, scalar)
nad_tfloat_tfloat → TemporalNADTFloat (4 args: vA, tsA, vB, tsB)
nad_tint_tint → TemporalNADTInt (4 args: vA, tsA, vB, tsB)
Single-instant tnumber construction uses MEOS's text constructor
`tfloat_in`/`tint_in` over a per-event WKT string "value@ts", mirroring
the existing tgeompoint pattern (where the WKT is built per record from
event fields and parsed by `temporal_in`). The constructed Temporal* is
freed after the MEOS call.
Generator additions
-------------------
Two new physical-cpp template branches + two new parser-glue dispatch-case
templates, all plumbed through emit_operator's existing flag dispatch:
* PHYSICAL_CPP_TEMPLATE_TNUMBER_POINT_WITH_SCALAR
— flag: build_tnumber_point_with_scalar
* PHYSICAL_CPP_TEMPLATE_TWO_TNUMBER_POINTS
— flag: build_two_tnumber_points
* DISPATCH_CASE_TNUMBER_POINT_WITH_SCALAR (3-arg dispatch)
* DISPATCH_CASE_TWO_TNUMBER_POINTS (4-arg dispatch)
Per-op extras in the JSON descriptor parameterize tnumber type (FLOAT64
or INT32) and the MEOS `*_in` constructor:
"tnumber_value_cpp_type": "double" | "int32_t"
"scalar_cpp_type": "double" | "int32_t"
"tnumber_in_fn": "tfloat_in" | "tint_in"
"tnumber_wkt_format": "{}@{}" (consumed by fmt::format at runtime)
Codegen anchor fix
------------------
The parser-dispatch anchor regex tuned for the pre-W4.5 layout
(TGEO_AT_STBOX → default:) no longer matched after W4.5 injected 20
cases between the two. New logic: insert just after the LAST
`/* END CODEGEN PARSER GLUE: ... */` marker if any exist (so successive
codegen runs cluster their cases), else fall back to the original
TGEO_AT_STBOX→default anchor.
Per-shape systests
------------------
Two new .test files in Tests/Functions/ — one per dispatch shape:
* nad_tfloat_float.test (one-tnumber + scalar; 3 rows; expected distance)
* nad_tfloat_tfloat.test (two-tnumbers; 3 rows; expected distance)
Per the testing-cadence directive: every codegen PR ships at least one
systest per dispatch shape it introduces.
Local verification on the mobilitynebula-v2 dev image:
cmake --build build-w1 --target nes-physical-operators -j 4
→ [47/47] Linking libnes-physical-operators.a
cmake --build build-w1 --target nes-logical-operators -j 4
→ [61/61] Linking libnes-logical-operators.a
cmake --build build-w1 --target nes-sql-parser -j 4
→ [11/11] Linking libnes-sql-parser.a
All three targets link clean on the first attempt — both new template
branches worked without iteration, and the parser-anchor fix is in
the generator so subsequent W5b/W6/W7 inherit it.
…mplate + 1 systest)
First restriction-shape operators. MEOS signature is
`Temporal* fn(const Temporal*, const GSERIALIZED*)` — returns the
clipped Temporal* (non-null if input survives the restriction, null
if clipped to empty).
For per-event single-instant inputs (the codegen's current shape), the
restriction collapses to a filter predicate: 1 if the point survives,
0 if clipped. This mirrors mariana's TemporalAtStBox int-collapse
pattern exactly — see TemporalAtStBoxPhysicalFunction.cpp:90 for the
hand-written precedent (`clipped.get() != nullptr ? 1 : 0`).
Operators
---------
tgeo_at_geom → TemporalAtGeometry (4 args; survives if point inside the geom)
tgeo_minus_geom → TemporalMinusGeometry (4 args; survives if point outside the geom)
Honest semantic note
--------------------
Per-event single-instant TEMPORAL_AT_GEOMETRY is **semantically equivalent**
to TEMPORAL_ECONTAINS_GEOMETRY (PR MobilityDB#23), and TEMPORAL_MINUS_GEOMETRY ≡
TEMPORAL_EDISJOINT_GEOMETRY. The restriction ops only add genuinely new
SQL surface when the input tgeompoint is a *sequence* of multiple
instants (W7-territory — windowed aggregations), where clipping produces
a different sequence than the original. Shipped now because:
1. They round out the SQL surface PostGIS / MobilityDB users expect
(the `AT`/`MINUS` idiom is standard there).
2. They exercise the codegen's first restriction-shape template, which
W7 sequence-aggregated restriction will inherit.
3. The collapse-to-int return matches mariana's TemporalAtStBox so
downstream consumers see a consistent shape across at/minus ops.
Generator additions
-------------------
* PHYSICAL_CPP_TEMPLATE_TEMPORAL_POINT_RESTRICTION
— calls `Temporal* {meos_call}(...)`, checks non-null, frees, returns int.
Flag: `build_temporal_point_restriction`.
* dispatch_case_for() reuses the existing DISPATCH_CASE_ONE_TEMPORAL_POINT
template — same 4-arg parser shape (lon, lat, ts, geom), only the
physical-cpp body shape differs (`Temporal*` return vs `int` return).
Per-shape systest
-----------------
Tests/Functions/at_geometry.test exercises TEMPORAL_AT_GEOMETRY: one
point inside a polygon (expect 1), one outside (expect 0).
Local verification on the mobilitynebula-v2 dev image:
cmake --build build-w1 --target nes-physical-operators -j 4
→ [49/49] Linking libnes-physical-operators.a
cmake --build build-w1 --target nes-logical-operators -j 4
→ [63/63] Linking libnes-logical-operators.a
cmake --build build-w1 --target nes-sql-parser -j 4
→ [11/11] Linking libnes-sql-parser.a
All three targets link clean on the first attempt.
…n/codegen_aggregations.py + 2 systests)
Companion to codegen_nebula.py: a separate generator targeting the
windowed-aggregation surface — MEOS scalar functions of the shape
`<scalar> fn(const Temporal*)` where the Temporal* is a per-(window,
group) sequence assembled across multiple events.
Operators
---------
3 tgeo-shape aggregations (lift = (lon, lat, ts), lower = trajectory):
temporal_num_instants → TemporalNumInstants
temporal_num_sequences → TemporalNumSequences
temporal_num_timestamps → TemporalNumTimestamps
9 tnumber-shape aggregations (lift = (value, ts), lower = sequence):
tfloat_start_value → TemporalTFloatStartValue
tfloat_end_value → TemporalTFloatEndValue
tfloat_min_value → TemporalTFloatMinValue
tfloat_max_value → TemporalTFloatMaxValue
tnumber_integral → TemporalTNumberIntegral
tint_start_value → TemporalTIntStartValue
tint_end_value → TemporalTIntEndValue
tint_min_value → TemporalTIntMinValue
tint_max_value → TemporalTIntMaxValue
12 ops, at the 15-op-per-PR cap. Each op emits 4 layer files
(logical .hpp + .cpp, physical .hpp + .cpp) mirroring mariana's hand-written
TemporalLengthAggregation 1:1.
Why a separate generator
------------------------
Aggregations live in DIFFERENT directories from the per-event ops:
* nes-{logical,physical}-operators/.../Aggregation*/ (this generator)
* nes-{logical,physical}-operators/.../Functions/Meos/ (codegen_nebula.py)
They use a DIFFERENT base class (AggregationPhysicalFunction vs
PhysicalFunction), DIFFERENT parser dispatch (windowAggs accumulator
vs functionBuilder stack), and DIFFERENT registry. Keeping them in
separate generators preserves shape cohesion and matches the
in-tree directory split.
What the generator writes
-------------------------
Per op, 4 emitted code files (above), AND idempotent injection into 5
shared files:
* nes-sql-parser/AntlrSQL.g4
- lexer-token entries
- functionName: alternation list
* nes-sql-parser/src/AntlrSQLQueryPlanCreator.cpp
- case AntlrSQLLexer::TOKEN: dispatch (dedicated-token switch)
- else if (funcName == "TOKEN") dispatch (IDENTIFIER fallback chain)
* nes-query-optimizer/src/RewriteRules/LowerToPhysical/
LowerToPhysicalWindowedAggregation.cpp
- if (name == "Xxx") block lowering logical → physical descriptor
* nes-{logical,physical}-operators/.../Aggregation*/CMakeLists.txt
- add_plugin(...) per layer
All injections are bracketed with
`/* BEGIN CODEGEN AGGREGATION GLUE: TOKEN ... */` markers so re-runs are
no-ops; pre-existing hand-written cases (mariana's TemporalLength,
PairMeeting, CrossDistance) are detected by raw token match and skipped.
Two lift-shape branches selected by descriptor.input_shape:
* "tgeo" — 3 fields per event; lower builds {Point(lon lat)@ts, ...}
parsed via MEOS::Meos::parseTemporalPoint.
* "tnumber" — 2 fields per event; lower builds {value@ts, ...} parsed
via tfloat_in or tint_in per descriptor.
Codegen target-naming convention
--------------------------------
Mariana's CMakeLists target name is the SQL aggregation name (e.g.
`TemporalLength`), NOT the C++ class basename (`TemporalLengthAggregation`).
The registry-codegen appends "Aggregation<RegistryKind>" to the target name,
so a target ending in "Aggregation" would yield a double-Aggregation
function symbol (caught here on the first build by linker error "did you
mean RegisterTemporalXXXAggregationAggregationLogicalFunction"). The
generator follows the mariana convention exactly.
Per-shape systests
------------------
* Tests/Functions/temporal_num_instants.test — tgeo aggregation
* Tests/Functions/temporal_tfloat_max_value.test — tnumber aggregation
Per the testing-cadence directive: every codegen PR ships at least one
systest per dispatch shape it introduces.
Local verification on the mobilitynebula-v2 dev image:
cmake --build build-w1 --target nes-logical-operators -j 4
→ [53/53] Linking libnes-logical-operators.a
cmake --build build-w1 --target nes-physical-operators -j 4
→ up to date
cmake --build build-w1 --target nes-sql-parser -j 4
→ [11/11] Linking libnes-sql-parser.a
cmake --build build-w1 --target nes-query-optimizer -j 4
→ up to date
All four targets link clean. The aggregation generator scales to any
single-Temporal*→scalar MEOS function by adding rows to the descriptor
JSON; new lift shapes (tcbuffer, tnpoint, tpose, …) need new template
branches following the tgeo/tnumber pattern.
…nical row-add)
Three more tnumber-shape aggregations fitting the existing W7 generator
templates exactly — no template work, only new descriptor rows. Validates
that the W7 aggregation generator scales by JSON-row addition for any
new single-Temporal*->scalar MEOS function with no further code change.
tfloat_avg_value → TemporalTFloatAvgValue
tnumber_twavg → TemporalTNumberTwAvg (time-weighted average, tfloat input)
tnumber_avg_value → TemporalTIntAvgValue (any-numeric MEOS fn applied via tint_in lift)
Note: tnumber_avg_value accepts any numeric Temporal* (tfloat or tint).
Wrapped via the tint_in lift to round out the tint side of the average
family; the tfloat side uses the type-specific tfloat_avg_value.
Per-shape systest
-----------------
Tests/Functions/temporal_tnumber_twavg.test — exercises TwAvg with a
known weighted-mean computation across 3+2 events per group.
No new shape is introduced (this PR adds rows to the existing tnumber-
aggregation shape covered by W7's temporal_tfloat_max_value.test), so
the single twavg systest is supplementary rather than per-shape-required.
Local verification on the mobilitynebula-v2 dev image:
cmake --build build-w1 --target nes-logical-operators -j 4
→ [59/59] Linking libnes-logical-operators.a
cmake --build build-w1 --target nes-physical-operators -j 4
→ up to date
cmake --build build-w1 --target nes-sql-parser -j 4
→ [11/11] Linking libnes-sql-parser.a
cmake --build build-w1 --target nes-query-optimizer -j 4
→ up to date
All four targets link clean on the first build.
… ops; bool+int64)
Five more tgeo-shape aggregations on the existing W7 template, exercising
two RETURN types the generator had not yet emitted (bool and int64).
Validates that the generator handles all four MEOS scalar return types
(int32, double, int64, bool) with zero template change — only new
descriptor rows in the JSON.
temporal_start_timestamptz → TemporalStartTimestamp (int64, TimestampTz)
temporal_end_timestamptz → TemporalEndTimestamp (int64, TimestampTz)
temporal_lower_inc → TemporalLowerInc (bool)
temporal_upper_inc → TemporalUpperInc (bool)
tpoint_is_simple → TemporalTPointIsSimple (bool)
All five use the existing tgeo lift shape (lon, lat, ts). The bool
and int64 final-stamp types map directly to the Nautilus val<>
templated wrapper without any template modification.
Per-shape systest
-----------------
Tests/Functions/temporal_tpoint_is_simple.test — exercises the bool
return path with one simple trajectory (expect TRUE) and one self-
intersecting trajectory (expect FALSE).
No new lift/dispatch shape is introduced; the systest is added to
demonstrate the BOOLEAN return type actually executes correctly
(belt-and-suspenders for the first PR exercising it).
Local verification on the mobilitynebula-v2 dev image:
cmake --build build-w1 --target nes-logical-operators -j 4
→ [69/69] Linking libnes-logical-operators.a
cmake --build build-w1 --target nes-physical-operators -j 4
→ up to date
cmake --build build-w1 --target nes-sql-parser -j 4
→ [11/11] Linking libnes-sql-parser.a
cmake --build build-w1 --target nes-query-optimizer -j 4
→ up to date
All four targets link clean on the first build.
…plate + 1 systest)
First extended-type batch: tcbuffer (circular buffer = point + radius)
spatial-relations against a static geometry. New 5-arg lift shape
(lon, lat, radius, ts, geometry) extends the codegen to its third
primitive Temporal* family beyond tgeo and tnumber.
econtains/ecovers/edisjoint/eintersects/etouches _tcbuffer_geo (5 e-ops)
acontains/acovers/adisjoint/aintersects/atouches _tcbuffer_geo (5 a-ops)
Per-event tcbuffer is constructed via tcbuffer_in() with WKT format
`Cbuffer(Point(lon lat),radius)@ts` (format confirmed by probing the
MEOS library directly). The Temporal* is freed after the MEOS call.
Generator additions
-------------------
One new physical-cpp template branch + one new dispatch-case template;
existing branches untouched:
* PHYSICAL_CPP_TEMPLATE_TCBUFFER_POINT
— 5 args: lon, lat, radius, ts, geometry. Calls
`int {meos_call}(const Temporal*, const GSERIALIZED*)`.
* DISPATCH_CASE_TCBUFFER_POINT
— 5-arg parser dispatch with geometry lift as VARSIZED.
* `build_tcbuffer_point` flag dispatch in emit_operator + dispatch_case_for.
Coverage scope
--------------
W10 covers ONLY the tcbuffer × geo 2-arg spatial-rel row (5 e + 5 a = 10
ops). The publicly declared tcbuffer surface in meos_cbuffer.h includes
more variations (tcbuffer × cbuffer, tcbuffer × tcbuffer, plus the
3-arg dwithin family), each requiring its own template branch and lift
shape. Those follow as future PRs per the ≤15-ops-per-PR cap.
Extended-types coverage at this PR:
* tcbuffer × geo (2-arg): 10/10 ✅
* tcbuffer × cbuffer (2-arg): 0/10 (separate template)
* tcbuffer × tcbuffer (2-arg): 0/9 (separate template, 8-arg lift)
* tcbuffer dwithin (3-arg): 0/6 (separate template per shape)
Note on tnpoint / tpose
-----------------------
Probing meos_npoint.h and meos_pose.h showed those families have NO
publicly declared spatial-rel ops — their non-tcbuffer surface is
restriction (at/minus), distance (tdistance), and nad. Those are
follow-up PRs, not part of W10.
Per-shape systest
-----------------
Tests/Functions/econtains_tcbuffer_geo.test — one tcbuffer with
radius 10 covering its own center point (expect 1), one with
radius 0.0001 vs a far point (expect 0).
Local verification on the mobilitynebula-v2 dev image:
cmake --build build-w1 --target nes-physical-operators -j 4
→ [59/59] Linking libnes-physical-operators.a
cmake --build build-w1 --target nes-logical-operators -j 4
→ [73/73] Linking libnes-logical-operators.a
cmake --build build-w1 --target nes-sql-parser -j 4
→ [11/11] Linking libnes-sql-parser.a
All three targets link clean on the first build.
…ew template + 1 systest)
Second tcbuffer batch. The static second arg is now a Cbuffer literal
(parsed via cbuffer_in) instead of a geometry (parsed as WKT).
econtains/ecovers/edisjoint/eintersects/etouches _tcbuffer_cbuffer (5 e-ops)
acontains/acovers/adisjoint/aintersects/atouches _tcbuffer_cbuffer (5 a-ops)
The per-event tcbuffer construction is identical to W10 (5-arg lift:
lon, lat, radius, ts, blob). Only the blob-parser differs:
W10: cbuffer literal as VARSIZED WKT geometry → MEOS::Meos::StaticGeometry
W11: cbuffer literal as VARSIZED WKT cbuffer → cbuffer_in() → Cbuffer*
So the dispatch case (5-arg SQL parser shape) is REUSED — only the
physical-cpp body differs.
Generator additions
-------------------
* PHYSICAL_CPP_TEMPLATE_TCBUFFER_POINT_CBUFFER — new physical template
with cbuffer_in() second-arg parser and {meos_call}(Temporal*, Cbuffer*)
call signature.
* `build_tcbuffer_point_cbuffer` flag dispatch in emit_operator.
* dispatch_case_for collapses build_tcbuffer_point and
build_tcbuffer_point_cbuffer to the same DISPATCH_CASE_TCBUFFER_POINT
(identical 5-arg SQL shape, only the physical-cpp body differs).
PR-awareness correction
-----------------------
A prior version of W10's PR body (MobilityDB#33) incorrectly stated that tnpoint
and tpose have no spatial-rels in the public MEOS API. That claim was
made against the vcpkg-baked MEOS in this dev image, which lags upstream
MobilityDB master. The retraction is now visible in MobilityDB#33's body. The
upstream-master substrate for tnpoint / tpose spatial-rel parity is in
open MobilityDB PRs:
#987 Close tpose parity gap with spatial functions, analytics, and tile
via tgeompoint composition
#1082 Add the tnpoint typed value accessors to the MEOS public API
#1083 tcbuffer + tpose typed value constructors
#1084 tcbuffer/tnpoint/tpose from-base time constructors
#1085 Export tpose_from_mfjson to MEOS public API
Those substrates feed W12+ (tcbuffer × tcbuffer + tnpoint and tpose
batches).
Per-shape systest
-----------------
Tests/Functions/econtains_tcbuffer_cbuffer.test — one self-intersection
case (expect 1) and one far-apart case with tiny radii (expect 0).
Local verification on the mobilitynebula-v2 dev image:
cmake --build build-w1 --target nes-physical-operators -j 4
→ [69/69] Linking libnes-physical-operators.a
cmake --build build-w1 --target nes-logical-operators -j 4
→ [83/83] Linking libnes-logical-operators.a
cmake --build build-w1 --target nes-sql-parser -j 4
→ [11/11] Linking libnes-sql-parser.a
All three targets link clean on the first build.
…ps + 8-arg lift + 1 systest)
Third tcbuffer batch. Two per-event tcbuffer instants are built from
(lonA, latA, radiusA, tsA) and (lonB, latB, radiusB, tsB) and passed
to MEOS `int fn(const Temporal*, const Temporal*)`. New 8-arg lift
shape.
adisjoint/aintersects/atouches _tcbuffer_tcbuffer (3 a-ops)
ecovers/eintersects/etouches _tcbuffer_tcbuffer (3 e-ops)
Total 6 publicly-declared 2-arg ops. econtains/edisjoint/acovers/acontains
are NOT publicly declared on _tcbuffer_tcbuffer; covered by extended
coverage rows omitted.
Generator additions
-------------------
* PHYSICAL_CPP_TEMPLATE_TWO_TCBUFFER_POINTS
— 8 args. Two tcbuffer_in() per-event constructions.
* DISPATCH_CASE_TWO_TCBUFFER_POINTS
— 8-arg parser dispatch (no constants).
* `build_two_tcbuffer_points` flag dispatch in emit_operator + dispatch_case_for.
Per-shape systest
-----------------
Tests/Functions/eintersects_tcbuffer_tcbuffer.test — overlapping
tcbuffers (expect 1) vs non-overlapping (expect 0).
Local verification on the mobilitynebula-v2 dev image:
cmake --build build-w1 --target nes-physical-operators -j 4
→ [75/75] Linking libnes-physical-operators.a
cmake --build build-w1 --target nes-logical-operators -j 4
→ [89/89] Linking libnes-logical-operators.a
cmake --build build-w1 --target nes-sql-parser -j 4
→ [11/11] Linking libnes-sql-parser.a
All three targets link clean on the first build.
Coverage scope
--------------
Tcbuffer 2-arg spatial-rels coverage at this PR:
tcbuffer × geo (2-arg): 10/10 ✅ (W10)
tcbuffer × cbuffer (2-arg): 10/10 ✅ (W11)
tcbuffer × tcbuffer (2-arg): 6/6 ✅ (this PR — missing 4 ops are not declared)
tcbuffer × {geo, cbuffer, tcbuffer} dwithin (3-arg): 6 ops pending future PR
tnpoint and tpose spatial-rel coverage gated on upstream MobilityDB
PRs (#987, #1082-#1085) reaching this dev image's vcpkg-baked MEOS;
see MobilityDB#33's RETRACTION section for the substrate map.
…t templates + 1 systest)
Closes the in-image-MEOS tcbuffer surface. Adds the 3-arg dwithin
variants across all three tcbuffer × {geo, cbuffer, tcbuffer} sub-shapes,
each with a trailing double distance threshold:
edwithin_tcbuffer_geo → TemporalEDWithinTCbufferGeometry (6-arg)
adwithin_tcbuffer_geo → TemporalADWithinTCbufferGeometry (6-arg)
edwithin_tcbuffer_cbuffer → TemporalEDWithinTCbufferCbuffer (6-arg)
adwithin_tcbuffer_cbuffer → TemporalADWithinTCbufferCbuffer (6-arg)
edwithin_tcbuffer_tcbuffer → TemporalEDWithinTCbufferTCbuffer (9-arg)
adwithin_tcbuffer_tcbuffer → TemporalADWithinTCbufferTCbuffer (9-arg)
Generator additions
-------------------
Three new physical-cpp template branches (one per sub-shape) + two
new dispatch case templates (the with-dist 6-arg dispatch is shared
across geo and cbuffer because the parser shape is identical — only
the physical-cpp blob-parser differs):
* PHYSICAL_CPP_TEMPLATE_TCBUFFER_POINT_WITH_DIST
* PHYSICAL_CPP_TEMPLATE_TCBUFFER_POINT_CBUFFER_WITH_DIST
* PHYSICAL_CPP_TEMPLATE_TWO_TCBUFFER_POINTS_WITH_DIST
* DISPATCH_CASE_TCBUFFER_POINT_WITH_DIST (shared 6-arg)
* DISPATCH_CASE_TWO_TCBUFFER_POINTS_WITH_DIST (9-arg)
* `build_tcbuffer_point_with_dist`, `build_tcbuffer_point_cbuffer_with_dist`,
`build_two_tcbuffer_points_with_dist` flag dispatch.
Per-shape systest
-----------------
Tests/Functions/edwithin_tcbuffer_tcbuffer.test — overlapping pair
(expect 1) vs far-apart pair (expect 0) at threshold 2.0.
Local verification on the mobilitynebula-v2 dev image:
cmake --build build-w1 --target nes-physical-operators -j 4
→ [81/81] Linking libnes-physical-operators.a
cmake --build build-w1 --target nes-logical-operators -j 4
→ [95/95] Linking libnes-logical-operators.a
cmake --build build-w1 --target nes-sql-parser -j 4
→ [11/11] Linking libnes-sql-parser.a
All three targets link clean on the first build.
Coverage scope — tcbuffer surface closed for this dev image
-----------------------------------------------------------
After W13 the entire in-image-MEOS tcbuffer 2-arg-and-3-arg spatial-rel
surface is closed:
tcbuffer × geo (2-arg, W10): 10/10 ✅
tcbuffer × cbuffer (2-arg, W11): 10/10 ✅
tcbuffer × tcbuffer (2-arg, W12): 6/6 ✅ (4 ops not publicly declared)
tcbuffer × {geo, cbuffer, tcbuffer} dwithin (this PR): 6/6 ✅
Total tcbuffer ops shipped this session: 32.
tnpoint and tpose spatial-rel coverage remain gated on upstream
MobilityDB PRs (#987, #1082-#1085); see MobilityDB#33's RETRACTION section
and MobilityDB#34's PR-awareness note.
… ops + 1 template + 1 systest) Closes the tpose × geo spatial-rel parity gap at the Nebula binding layer using the SAME composition recipe MobilityDB PR #987 uses at the SQL layer: convert the temporal pose to a temporal geometry point, then apply the existing _tgeo_geo spatial-rel. Correcting the record: an earlier W10 PR body (MobilityDB#33) claimed tpose has no spatial-rels in the public MEOS API. That was wrong — checking open PRs (per the always-check-PRs rule) shows MobilityDB #987 closes tpose parity, and the composition primitive tpose_to_tpoint() is ALREADY in this dev image's MEOS public API. No dev-image rebuild was needed. econtains/ecovers/edisjoint/eintersects/etouches via tpose→tgeo (5 e-ops) acontains/adisjoint/aintersects/atouches via tpose→tgeo (4 a-ops) (acovers_tgeo_geo is not publicly declared, so ACovers is correctly out of scope — same gap noted in W2 for tgeo × geo.) Composition path (per event): Pose(Point(x y), theta)@ts --tpose_in--> Temporal* (tpose) --tpose_to_tpoint--> Temporal* (tgeompoint) --{meos_call}(tgeo, gs)--> int Both Temporal* freed after the call. Generator additions ------------------- * PHYSICAL_CPP_TEMPLATE_TPOSE_POINT_VIA_COMPOSITION — 5 args (x, y, theta, ts, geometry); builds tpose, converts via tpose_to_tpoint, calls the existing _tgeo_geo spatial-rel. * `build_tpose_point_via_composition` flag dispatch. * dispatch_case_for collapses tcbuffer-point / tcbuffer-cbuffer / tpose-composition to the same 5-arg DISPATCH_CASE_TCBUFFER_POINT (identical SQL shape: 3 doubles + ts + blob). Per-shape systest ----------------- Tests/Functions/econtains_tpose_geo.test — tpose at a point contained by an identical static point (expect 1) vs a far point (expect 0). Local verification on the mobilitynebula-v2 dev image: cmake --build build-w1 --target nes-physical-operators -j 4 → [90/90] Linking libnes-physical-operators.a cmake --build build-w1 --target nes-logical-operators -j 4 → [104/104] Linking libnes-logical-operators.a cmake --build build-w1 --target nes-sql-parser -j 4 → [11/11] Linking libnes-sql-parser.a All three targets link clean on the first build. This composition recipe generalizes: tpose × tpose, tnpoint × geo, tnpoint × tnpoint all follow the same convert-then-delegate pattern (tnpoint_to_tgeompoint is likewise already in the public API).
…(9 ops + 1 template + 1 systest) Completes the tpose family started in W14 (MobilityDB#37): pairs two single-instant tposes against each other (8 args) instead of one tpose against a static geometry (5 args). Each tpose is lifted to a tgeompoint via tpose_to_tpoint at run time, then the existing _tgeo_tgeo spatial-rel (shipped in W3, MobilityDB#25) is applied — no new MEOS symbols, no dev-image rebuild. 9 e/a operators: e{contains,covers,disjoint,intersects,touches} + a{contains,disjoint,intersects,touches}. acovers_tgeo_tgeo is not publicly declared, so ACovers is out of scope (same gap as W3/W14). Generator additions: - PHYSICAL_CPP_TEMPLATE_TWO_TPOSE_POINTS_VIA_COMPOSITION (8-arg two-tpose body) - build_two_tpose_points_via_composition flag dispatch - DISPATCH_CASE_TWO_TPOSE_POINTS (8-arg parser case) Systest: Tests/Functions/eintersects_tpose_tpose.test. Local verification (nes-development:mobilitynebula-v2): nes-physical-operators, nes-logical-operators, nes-sql-parser all link clean.
…s + 2 templates + 1 systest)
Unblocks the tnpoint family on NebulaStream. tnpoint composes per-event
exactly like tpose (W14/W15): tnpoint_in -> tnpoint_to_tgeompoint -> the
existing _tgeo_geo / _tgeo_tgeo spatial-rels (W2/W3). The route-geometry
lookup inside tnpoint_to_tgeompoint goes through MEOS's per-thread TLS
ways cache, so the operator carries no network state — the codegen shape
is identical to the other composition waves.
18 ops: 9 tnpoint x geo (_tgeo_geo) + 9 tnpoint x tnpoint (_tgeo_tgeo),
each the e/a set e{contains,covers,disjoint,intersects,touches} +
a{contains,disjoint,intersects,touches}. acovers is not publicly
declared (out of scope, as in W2/W3/W14/W15).
Generator additions:
- PHYSICAL_CPP_TEMPLATE_TNPOINT_POINT_VIA_COMPOSITION (4 args: rid,
fraction, ts, geometry) + build_tnpoint_point_via_composition
- PHYSICAL_CPP_TEMPLATE_TWO_TNPOINT_POINTS_VIA_COMPOSITION (6 args:
ridA, fractionA, tsA, ridB, fractionB, tsB) +
build_two_tnpoint_points_via_composition
- dispatch_case_for reuses the 4-arg / 6-arg parser cases by arity.
Systest: Tests/Functions/eintersects_tnpoint_tnpoint.test.
Runtime note: tnpoint_to_tgeompoint reads the ways network from
/usr/local/share/ways1000.csv (MEOS default path). That file must be
present to run tnpoint queries (a copy ships in MobilityDB at
meos/examples/data/ways1000.csv). tnpoint_to_tgeompoint yields a
tgeompoint in the network SRID, so tnpoint x static-geometry needs the
geometry in that SRID; tnpoint x tnpoint is unaffected.
Local verification (nes-development:mobilitynebula-v2): nes-physical-operators,
nes-logical-operators, nes-sql-parser all link clean.
…nce) (4 ops + 1 systest)
Adds nearest-approach distance for the tpose and tnpoint families,
completing their distance-measure surface alongside the spatial-rels
(W14/W15/W18). No new generator template: nad has the same
(Temporal*, ...) -> scalar shape as the spatial-rels, so the existing
composition templates are reused with a double (FLOAT64) return — exactly
as the tgeo nad ops (TemporalNADGeometry / TemporalNADTGeometry) already
do.
4 ops: TemporalNAD{TPoseGeometry,TPoseTPose,TNpointGeometry,TNpointTNpoint}
calling nad_tgeo_geo / nad_tgeo_tgeo. tpose resolves via tpose_to_tpoint,
tnpoint via tnpoint_to_tgeompoint (network SRID; needs the ways CSV at
run time, same as W18).
Systest: Tests/Functions/nad_tpose_tpose.test (identical tposes -> 0).
Local verification (nes-development:mobilitynebula-v2): nes-physical-operators,
nes-logical-operators, nes-sql-parser all link clean.
…s + 1 systest)
Completes the tpose and tnpoint distance surface (with nad in W19 and the
spatial-rels in W14/W15/W18). tpose/tnpoint resolve to tgeompoints via
tpose_to_tpoint / tnpoint_to_tgeompoint, then the existing 3-arg
edwithin/adwithin _tgeo_geo / _tgeo_tgeo calls run with the query-level
distance constant.
8 ops: Temporal{E,A}DWithin{TPoseGeometry,TPoseTPose,TNpointGeometry,TNpointTNpoint}.
Generator: 4 new with-dist composition templates (the W14/W15/W18 bodies
plus a trailing double dist forwarded to the MEOS call) + build_* flags.
No new parser dispatch — dispatch_case_for reuses the existing with-dist
dispatches by arity/constant pattern (distance is a lifted SQL constant;
tpose×geo/tpose×tpose match the 6-arg/9-arg tcbuffer-with-dist cases,
tnpoint the 5-arg/7-arg tgeo cases).
Systest: Tests/Functions/edwithin_tpose_tpose.test.
Local verification (nes-development:mobilitynebula-v2): nes-physical-operators,
nes-logical-operators, nes-sql-parser all link clean.
… ops + 1 systest) Rounds out the tcbuffer distance surface (spatial-rels W10-W12, dwithin W13). Like the tpose/tnpoint nad (W19), no new generator template: nad has the same (Temporal*, ...) -> scalar shape as the tcbuffer spatial-rels, so the existing tcbuffer templates are reused with a double (FLOAT64) return. 3 ops: TemporalNADTCbuffer (nad_tcbuffer_geo), TemporalNADTCbufferCbuffer (nad_tcbuffer_cbuffer), TemporalNADTCbufferTCbuffer (nad_tcbuffer_tcbuffer). nad_tcbuffer_stbox deferred with the other TBox-arg variants. Systest: Tests/Functions/nad_tcbuffer_tcbuffer.test (identical tcbuffers -> 0). Local verification (nes-development:mobilitynebula-v2): nes-physical-operators, nes-logical-operators, nes-sql-parser all link clean.
estebanzimanyi
force-pushed
the
feat/nebula-streaming-parity-harness
branch
6 times, most recently
from
7775ddf to
e760763
Compare
tnumber_tboxes (Temporal -> TBox[]): the whole-series value/time bounding
boxes of a temporal number, via the existing array-out span_val machinery
(tbox_out per element), mirroring tgeo_stboxes. Count out-param only, no
extras. Output carries '(' so hand-recorded from the results CSV.
Suite green (2-batch, zero regressions); tnumber_tboxes verified passing.
NebulaStream: 1691->1692 proven, gap 242->241.
tgeometry_to_tgeompoint, tgeometry_to_tgeography, tgeography_to_tgeometry, tgeography_to_tgeogpoint, tgeogpoint_to_tgeography — unary temporal->temporal conversions via the TEMPORAL_TO_TEMPORAL map (tspatial_as_text output). Add tgeography/tgeogpoint WKT-primary inputs (tgeography_in/tgeogpoint_in) and a WKT-primary cols branch. Fix: the tgeometry/tgeography/tgeogpoint builders now strip the systest VARSIZED field's surrounding quotes before tgeometry_in (else 'parse error - invalid geometry'); all probe-verified. Suite green: 1683/1683 (2-batch), zero regressions. NebulaStream: 1692->1697 proven (86.8%->87.1%), gap 241->236.
temporal_instant_n (Temporal,int->TInstant, fixed n=1) and tpoint_get_z (3D temporal point -> tfloat z track). New early TPOINT_OPS branch on a temporal-point primary, intercepted before the generic unary/arity-2 branches so the tpoint subtype is not mis-mapped to a tfloat instant; tpoint_get_z uses the tgeompoint3d (lon/lat/z) builder. tpoint_trajectory and tpoint_is_simple are deliberately excluded: both already carry an aggregation registration, so a windowless per-event query routes to the NebulaStream agg path and throws 9005 (the known 'Only TimeBasedWindowType' planner bug). Suite green: 1685/1685 (2-batch), zero regressions. NebulaStream: 1697->1699 proven (87.1%->87.2%), gap 236->234.
tstzspan_bins, tstzspanset_bins, datespanset_bins (Span/SpanSet + duration interval + time origin -> Span[] time bins) and tfloat_value_time_boxes, tint_value_time_boxes (tnumber + vsize + duration + vorigin + torigin -> TBox[]). All via the existing array-out span_val machinery with interval (_a3iv) and time-origin (_TSTZ0 / _a3st date) extras, mirroring datespan_bins. Suite green: 1690/1690 (2-batch), zero regressions. NebulaStream: 1699->1704 proven (87.2%->87.4%), gap 234->229.
The *_hash_extended visibility asymmetry (cbuffer/npoint/pose/stbox internal while set/span/spanset/tbox are public) and the 43 type-dispatch predicates (temporal_basetype family + MeosType enum) that MobilitySpark and the Nebula streaming grind need. All already exported in libmeos; this is a declaration- visibility change. Nebula implements them data-bound meanwhile.
set_hash_extended, span_hash_extended, spanset_hash_extended, tbox_hash_extended (container/box + fixed uint64 seed -> uint64 hash). New HASH_OPS branch; the seed rides as a fixed call arg, result is an INT64 field. These are the public data-input hash accessors; the cbuffer/npoint/pose/stbox siblings stay internal pending the meos.h visibility handoff. Suite green: 1694/1694 (2-batch), zero regressions. NebulaStream: 1704->1708 proven (87.4%->87.6%), gap 229->225.
cbuffer_hash_extended, npoint_hash_extended, pose_hash_extended, stbox_hash_extended (object/box + fixed seed -> uint64). Same HASH_OPS branch as the public hashes; these are declared in meos_internal.h (exported), so the operators include it via extra_headers pending the public-visibility handoff — the include drops once MEOS promotes them to meos.h, no regen churn. Suite green: 1698/1698 (2-batch), zero regressions. NebulaStream: 1708->1712 proven (87.6%->87.8%), gap 225->221.
The temporal_basetype family — temporal/tnumber/tgeo/tpoint/tgeometry/tspatial/ talpha type+basetype+spantype predicates, plus set/span/spanset/time variants (32 bool fns). These are streaming-runtime metadata: a window operator reads the value's type to dispatch deserialize/compare/aggregate (exactly why MobilitySpark needs temporal_basetype). Implemented data-bound via a new codegen type_field mode: build the value, pass its public type-field cast to MeosType — e.g. temporal_basetype((MeosType) temp->temptype). MeosType + the predicate decls are in meos_catalog.h (extra_headers); the type fields temptype/settype/spantype are public meos.h struct fields. Suite green: 1730/1730 (2-batch), zero regressions. NebulaStream: 1712->1744 proven (87.8%->89.5%), gap 221->189.
int32_cmp, int64_cmp (base-scalar + scalar -> int) and text_cmp (text + text -> int). New CMP_OPS branch: a base-scalar/text primary + a same-type second operand, returning the three-way comparison int. Suite green: 1733/1733 (2-batch), zero regressions. NebulaStream: 1744->1747 proven (89.5%->89.6%), gap 189->186.
Documents why one MEOS C symbol costs ~8 NebulaStream artifacts (logical+ physical operator, grammar token + functionName alternative, parser case, CMake, systest), contrasts with a generic library (JSON), gives the value_at_timestamptz 7-symbol fan-out example, and proposes a generic symbol-dispatch UDF + a grammar-free function registry + a typed external-value handle to make C-library onboarding descriptor-driven.
tboolinst_make (bool, ts -> tbool instant), ttextinst_make (text, ts -> ttext instant) via the MAKE_SPEC pattern. Adds a bool_base scalar primary to the codegen (BOOLEAN value -> bool), which also unblocks future bool-valued constructors. Suite green: 1735/1735 (2-batch), zero regressions. NebulaStream: 1747->1749 proven (89.6%->89.7%), gap 186->184.
nsegment_make (rid, pos1, pos2 -> Nsegment), interval_make (years..secs -> Interval), pose_make_3d and pose_make_point3d (3D pose constructors with a unit quaternion), npoint_to_nsegment, nsegment_round, and stbox_get_space (STBox space-only projection), via the MAKE_SPEC / ROUND_RET / unary-accessor generator patterns. Adds an Nsegment result serializer (nsegment_out) and the nsegment_value_out / stbox_text_out return kinds to the codegen. Suite green: 1742/1742 (2-batch), zero regressions. NebulaStream: 1749->1756 proven (89.7%->90.1%), gap 184->177.
tbool/tfloat/tint/tgeo/tpose/ttext_value_at_timestamptz: the value of a temporal at a given timestamp, returned through MEOS's bool f(temp, t, strict, T*result) out-parameter form. Adds a value_at_timestamptz branch to gen_algebra_wave that builds the per-event instant at a fixed timestamp and matches the query-ts argument to it (strict=false), reusing the existing codegen out_param path (scalar result for tint/tfloat/tbool, heap-object for tgeo/tpose/ttext). Suite green: 1748/1748 (2-batch), zero regressions. NebulaStream: 1756->1762 proven (90.1%->90.4%), gap 177->171.
tfloatbox/tintbox_shift_scale shift and scale a TBox's value span (the hasshift/haswidth flags fixed true) via the named-extras path. geom_azimuth and bearing_point_point return the azimuth/bearing angle between two geometries through MEOS's bool f(geom, geom, double*) out-parameter form, reusing the out_param codegen path with a second-geometry operand (new _geomaz3 arity gate). Suite green: 1752/1752 (2-batch), zero regressions. NebulaStream: 1762->1766 proven (90.4%->90.6%), gap 171->167.
geo/stbox/tspatial/cbuffer/pose_transform_pipeline reproject their input through an explicit PROJ pipeline. A fixed 4326->3857 web-mercator pipeline string + target SRID 3857 + forward flag ride as constant call arguments (the SRID-transform pattern, extended to arity 4 via a new _tpipe4 gate and a TRANSFORM_PIPELINE_OPS map). All five probe-verified. Suite green: 1757/1757 (2-batch), zero regressions. NebulaStream: 1766->1771 proven (90.6%->90.9%), gap 167->162.
stbox_get_time_tile (a timestamp's tile via the named-extras path), tgeo_scale (scale a temporal geometry about an origin), and stbox_get_space_tile (a 3D space tile of a point). The scale/origin geometries and tile sizes are constant operands passed inline as call arguments. All probe-verified. Suite green: 1760/1760 (2-batch), zero regressions. NebulaStream: 1771->1774 proven (90.9%->91.0%), gap 162->159.
temporal_split_n_spans / temporal_split_each_n_spans split a temporal's time extent into spans, and stbox_quad_split splits a box into its four quadrant boxes, all via the existing array-output machinery (Span/STBox struct array with a split-count scalar / a bare count out-param). Suite green: 1763/1763 (2-batch), zero regressions. NebulaStream: 1774->1777 proven (91.0%->91.2%), gap 159->156.
temporal_shift_time / temporal_scale_time (shift or rescale a temporal's time by an interval), temporal_at_timestamptz / temporal_minus_timestamptz (restrict to or remove a timestamp), and temporal_to_tsequence / temporal_to_tsequenceset (convert with a fixed step interpolation). New TEMPORAL_TIMEOP branch; the interpType argument is passed as a cast enum constant. Suite green: 1769/1769 (2-batch), zero regressions. NebulaStream: 1777->1783 proven (91.2%->91.5%), gap 156->150.
temporal_set_interp sets a temporal's interpolation, and temporal_simplify_min_tdelta drops samples closer in time than a minimum interval. Both extend the temporal time-op path. Suite green: 1771/1771 (2-batch), zero regressions. NebulaStream: 1783->1785 proven (91.5%->91.6%), gap 150->148.
tfloat_value_split / tint_value_split partition a temporal by value buckets and temporal_time_split by time buckets, each returning the temporal fragments. The array-output machinery gains an aux_out option to capture and free the auxiliary bin-boundary out-array these functions write alongside the fragment array. Suite green: 1774/1774 (2-batch), zero regressions. NebulaStream: 1785->1788 proven (91.6%->91.7%), gap 148->145.
tfloat_value_time_split / tint_value_time_split partition a temporal jointly by value and time buckets. The array-output machinery's aux_out gains an aux_out2 companion to capture and free the two bin-boundary out-arrays (value bins and time bins) these functions write alongside the fragment array. Suite green: 1776/1776 (2-batch), zero regressions. NebulaStream: 1788->1790 proven (91.7%->91.8%), gap 145->143.
tgeo_space_split / tgeo_space_time_split partition a 3D temporal point by space tiles (and time tiles) into fragments, with the space origin and bitmatrix/ border flags as fixed call arguments and the GSERIALIZED** / TimestampTz* bin-boundary out-arrays captured and freed via the array aux_out machinery. Suite green: 1778/1778 (2-batch), zero regressions. NebulaStream: 1790->1792 proven (91.8%->91.9%), gap 143->141.
cbufferarr_round / posearr_round / temparr_round round an array of objects. A new array_in_round codegen path wraps the per-event object in a one-element array, invokes the array-input MEOS function, and serializes the single rounded result — the first array-input function family wired in the per-event operators. Suite green: 1781/1781 (2-batch), zero regressions. NebulaStream: 1792->1795 proven (91.9%->92.1%), gap 141->138.
geo_cluster_intersecting / geo_cluster_within group geometries into clusters (by mutual intersection, or within a distance) and return the cluster geometry collections. The array-output machinery gains an array_in option that wraps the per-event geometry in a one-element input array for these array-input functions. Suite green: 1783/1783 (2-batch), zero regressions. NebulaStream: 1795->1797 proven (92.1%->92.2%), gap 138->136.
geo_cluster_kmeans / geo_cluster_dbscan assign each geometry a cluster id. A new array_in_scalar codegen path wraps the per-event geometry in a one-element input array and returns the single cluster id, completing the geo-clustering family. Suite green: 1785/1785 (2-batch), zero regressions. NebulaStream: 1797->1799 proven (92.2%->92.3%), gap 136->134.
The MEOS trgeo->trgeometry user-API rename moved the streamable surface to trgeometry_* names, but nebula.feed.tsv still carried 48 old trgeo_* / *_trgeo_* entries (40 marked proven, 8 wired). None of them appear in feeds/streamable.txt, so they were phantom rows inflating the proven count by 40 — exactly the over-claim the parity methodology guards against. Their trgeometry_* equivalents remain present and proven. Regenerate the feed from the full passing systest suite via tools/streaming_parity/adapters/nebula.py, which drops the ghosts and emits the feed in canonical (LC_ALL=C) sorted order.
stboxarr_round rounds an STBox to a given precision via the array-input MEOS function stboxarr_round(arr, count, maxdd) applied to a single-element array; codegen_nebula gains a "contig" array-in-round variant for functions that return a contiguous struct array (T*) rather than a pointer array (T**). tnpoint_positions returns the network segments a temporal network point occupies, as a set of NSegment values. Both operators are recorded against the streamable surface and proven by the systest suite (feed: 1799 -> 1801 proven), bringing object-output recording in line with the existing tgeo_stboxes/*_points convention.
estebanzimanyi
changed the title
Adapt the generated operators to the pin's MEOS signature changes and re-record the expected outputs shifted by its logic fixes. The full MEOS systest suite passes 1783/1783. - Codegen reads struct-return multi-output functions, passes the trailing int *count to array-returning functions, and uses the _geo argument name for the tdistance tnpoint/tpose distance operators. - Drop the operators no longer on the public MEOS surface: set_basetype, talphanum_type, temporal_basetype, tdistance_tnpoint_point and tdistance_tpose_point. - Re-record the tcbuffer intersects/disjoint and tnpoint always_ne/ever_eq expected blocks for the corrected intersects routing and the network-point epsilon. - Refresh the exported-symbol manifests to the pin.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The streaming-platform counterpart of MobilityDB's cross-type parity methodology (
doc/methodology/cross_type_parity.md) and audit harness (tools/parity_audit/). It measures, per streaming platform (NebulaStream, Flink, Kafka), how many streamable MEOS functions are backed by a passing test.A function counts only at the proven layer: exported in the pinned
libmeos(nm -D), wired by an operator that calls it, and exercised by a passing test. The reference surface is the streamable MEOS public API derived frommeos-idl.json; io-meta, sequence-only, ambiguous, and internal functions are reason-marked non-streamable.This branch is the integration-evidence and benchmark vehicle for the topical series #15–#43: it folds them together so the systest suite and the BerlinMOD benchmark run on the combined state before the individual PRs land. It is not itself a merge target; the individual PRs are the deliverable.