Register the public MEOS functions alone on the Spark surface - #136
Merged
estebanzimanyi merged 2 commits intoOct 3, 2026
Merged
estebanzimanyi merged 2 commits into
estebanzimanyi merged 2 commits into
Conversation
An argument of a C enum travels in SQL as the text its name is, and both JVM SQL surfaces turn that text into the enum through the catalog function returning that enum from one string. The rule is one: SqlModel._enum_parsers in codegen_jvm.py and ENUM_PARSER in codegen_spark_udfs.py both admit a public parser alone (null_handle_type_from_string, interptype_from_string, raquet_pixtype_from_string). In the Spark arm an enum argument is a string whose conversion the four call builders now apply, as they pass every other scalar argument through its expression. Witness. Against MobilityDB d29d4ab293 and the catalog of MEOS-API 5ba0cd8492, the Spark arm refuses tjsonb_to_tbigint, public since MobilityDB 21a15203ba, and every other function taking a nullHandleType or an interpType, as an argument it cannot read, which the gaps ledger records; the Flink arm reads an interpType through interptype_from_string, which the catalog stated internal until MobilityDB 671ed23326. Why. A binding calls the public API alone, and a public parser tests its argument at entry, so a null or unknown name is reported to the engine rather than read. Flink already passed an enum as its text; Spark refused every function taking one, so the two surfaces differed on the constructors and conversions that take an interpolation or a null handling. Measured. From the same catalog and jar, against the generator that answers every overload of a Spark name whose argument classes differ, the Flink surface this branch generates is byte-identical, and Spark registers 3,724 names where that generator registers 3,646: 78 added and none removed, among them tintSeq, tfloatSeqSet and the other <type>Seq and <type>SeqSet constructors, setInterp, tsample, appendInstant, the tjsonb_to_* conversions and the JSON functions taking a null handling; tnpoint gains its sequence constructors from a base value and a time span. The gaps ledger loses 33 functions and lists 711, none new; the two parsers leave it as the conversions the surface calls.
The Spark arm admits a function by the catalog's api field, as the Flink arm does: a function is registered only when api is public. It tested the name of the Doxygen group instead, and a function stating no group, which the catalog states internal, passed that test. Witness. Against MobilityDB 91efcc4721 and the catalog of MEOS-API 5ba0cd8492, the Spark surface JMEOS main generates calls 139 functions the catalog states internal and registers 137 of them under their C names, among them geo_clip_subject, geo_is_planar_areal, interptype_name, pc_compression_name and the pg_date_*, pg_interval_* and pg_numeric_* families, while the Flink surface calls none. Why. A binding projects the public API, the functions whose @InGroup is a public group; an internal function carries no contract a binding may rely on, asserting its preconditions where a public one tests them. MobilityDuck reads the same field for the same reason. Measured. From the same catalog and jar, Spark registers 3,587 names where main registers 3,724: 137 removed, none added, and no remaining name changes the functions behind it. The Flink surface is unchanged and the gaps ledger lists the same 711 functions. The one internal function the Spark surface still calls is numeric_out, the text writer the catalog's typeEncodings names for NumericData, through which the public function returning a numeric is serialized.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Read an enum argument from its text through the public parser
An argument of a C enum travels in SQL as the text its name is, and both
JVM SQL surfaces turn that text into the enum through the catalog function
returning that enum from one string. The rule is one: SqlModel._enum_parsers
in codegen_jvm.py and ENUM_PARSER in codegen_spark_udfs.py both admit a
public parser alone (null_handle_type_from_string, interptype_from_string,
raquet_pixtype_from_string). In the Spark arm an enum argument is a string
whose conversion the four call builders now apply, as they pass every other
scalar argument through its expression.
Witness. Against MobilityDB d29d4ab293 and the catalog of MEOS-API
5ba0cd8492, the Spark arm refuses tjsonb_to_tbigint, public since
MobilityDB 21a15203ba, and every other function taking a nullHandleType or
an interpType, as an argument it cannot read, which the gaps ledger records;
the Flink arm reads an interpType through interptype_from_string, which the
catalog stated internal until MobilityDB 671ed23326.
Why. A binding calls the public API alone, and a public parser tests its
argument at entry, so a null or unknown name is reported to the engine
rather than read. Flink already passed an enum as its text; Spark refused
every function taking one, so the two surfaces differed on the constructors
and conversions that take an interpolation or a null handling.
Measured. From the same catalog and jar, against the generator that
answers every overload of a Spark name whose argument classes differ, the
Flink surface this branch generates is byte-identical, and Spark registers
3,724 names where that generator registers 3,646: 78 added and none removed, among them tintSeq,
tfloatSeqSet and the other Seq and SeqSet constructors,
setInterp, tsample, appendInstant, the tjsonb_to_* conversions and the
JSON functions taking a null handling; tnpoint gains its sequence
constructors from a base value and a time span. The gaps ledger loses 33
functions and lists 711, none new; the two parsers leave it as the
conversions the surface calls.
Register the public MEOS functions alone on the Spark surface
The Spark arm admits a function by the catalog's api field, as the Flink
arm does: a function is registered only when api is public. It tested the
name of the Doxygen group instead, and a function stating no group, which
the catalog states internal, passed that test.
Witness. Against MobilityDB 91efcc4721 and the catalog of MEOS-API
5ba0cd8492, the Spark surface JMEOS main generates calls 139 functions the
catalog states internal and registers 137 of them under their C names,
among them geo_clip_subject, geo_is_planar_areal, interptype_name,
pc_compression_name and the pg_date_, pg_interval_ and pg_numeric_*
families, while the Flink surface calls none.
Why. A binding projects the public API, the functions whose @InGroup is a
public group; an internal function carries no contract a binding may rely
on, asserting its preconditions where a public one tests them. MobilityDuck
reads the same field for the same reason.
Measured. From the same catalog and jar, Spark registers 3,587 names where
main registers 3,724: 137 removed, none added, and no remaining name
changes the functions behind it. The Flink surface is unchanged and the
gaps ledger lists the same 711 functions. The one internal function the
Spark surface still calls is numeric_out, the text writer the catalog's
typeEncodings names for NumericData, through which the public function
returning a numeric is serialized.