substrait-csharp

Experimental C# bindings for substrait.

Development

The repository requires .NET SDK 10.0.403 or a newer stable .NET 10 SDK. The checked-in global.json selects a compatible installed SDK. Running the .NET 8 tests also requires .NET 8.0.33 or later.

Clone the repository:

git clone https://github.com/substrait-io/substrait-csharp.git

Restore, build, and test the solution from the repository root:

dotnet restore Substrait.sln
dotnet build Substrait.sln --configuration Release --no-restore
dotnet test --solution Substrait.sln --configuration Release --no-build

Create the NuGet package locally with:

dotnet pack src/Substrait/Substrait.csproj --configuration Release --output artifacts/packages

The package ID and managed assembly name are Substrait.Net, and public namespaces remain under Substrait.

Preview package validation

Pull-request and branch continuous integration creates a run-scoped prerelease package, validates its NuGet metadata and Source Link mappings, restores it into a standalone consumer, and generates an SPDX 2.2 SBOM. The .nupkg, .snupkg, and SBOM are uploaded as workflow artifacts for review. Tagged releases publish to NuGet.org through trusted publishing; see RELEASING.md.

The package consumer has no project reference to the library. To exercise it locally after packing a preview version, restore from the package output plus a public source and then run it:

dotnet restore tests/PackageSmokeTest/PackageSmokeTest.csproj -p:SmokeTestPackageVersion=0.1.0-preview.1 --configfile tests/PackageSmokeTest/NuGet.Config --no-cache
dotnet run --project tests/PackageSmokeTest/PackageSmokeTest.csproj --configuration Release --no-restore -p:SmokeTestPackageVersion=0.1.0-preview.1

When local network policy blocks public package downloads, the Windows and Linux CI jobs are the authoritative package restore and consumer validation.

See Preview package status for the API stability and compatibility scope of the package artifacts.

The package and assembly identity remain provisional until the first package preview.

The specification is supplied by Substrait.Net.Protobuf, Substrait.Net.Antlr, and Substrait.Net.Extensions NuGet packages, currently version 0.107.0. No submodule, protobuf compiler, or Java/ANTLR generation tool is needed. Versions are pinned in Directory.Packages.props; see Updating Substrait for upgrade instructions and compatibility notes for changes from 0.104.0 and the previously bundled v0.73.0 specification.

Conversion and serialization

Generated read-only metadata facades provide immutable snapshots of protobuf relation metadata, including opaque advanced extensions. Supported relations preserve common fields, operator extensions, and named-table extensions through conversion. Relation anchors are validated across all plan entries and subqueries.

Use ProtoToPlanConverter and PlanToProtoConverter to convert between generated protobuf plans and the immutable internal representation. Extension references can be resolved strictly or selectively with ExtensionsDictionary.StrictMode; non-strict conversion preserves unresolved function references but cannot attach their declarations.

FileUtils reads and writes protobuf binary and protobuf JSON plan files. Converting a plan does not add nondeterministic metadata, so repeated protobuf serialization of the same internal plan produces the same bytes.

User-defined types support ordered null, data-type, boolean, signed 64-bit integer, enum, and string parameters. Full-plan export emits the required type declarations and URNs, including nested dependencies. Standalone type conversion requires explicit extension contexts. See user-defined types for construction, programmatic registration, strict/permissive resolution, and declaration retention, and the remaining YAML-loading boundaries.

ParameterizedType.List and ParameterizedType.Map model nested collection types, including a struct nested arbitrarily deep inside either. See list and map types for construction, malformed- message error reporting, and depth-first named-struct field traversal.

ParameterizedType.AliasReference models a plan-local type-alias reference as a lossless anchor value; it does not resolve or follow the alias it points to. See type-alias references for construction, the distinction from alias-table resolution, and the variation-rejection behavior.

ParameterizedType.Func models a function's parameter and return types; it is the one parameterized type that cannot carry a type variation. See function types for construction, malformed-message error reporting, and the variation-rejection behavior.

UnboundType.Instance models Type.unbound, a placeholder for a type not yet bound to a concrete type in a partially bound plan or expression; unlike every other IType, its nullability throws rather than returning a value. See the unbound placeholder type for construction, the throwing-nullability behavior, and binding boundaries.

ProtoToPlanConverter.FromBytes, FromStream, and FromFile parse binary protobuf privately, avoiding redundant metadata copies; FromJson explicitly parses protobuf JSON and accepts a parser with registered Any payload types. Caller-provided streams remain open. The existing From(Protobuf.Plan) and standalone relation converters copy retained metadata, and protobuf exports remain detached. See metadata ownership and API migration for constructor, equality, anchor, and format details.

All plan import entry points resolve detached expression and relation subtrees, including those inside subqueries, without increasing protobuf parser limits. IPlan.IRelation.Metadata retains the original container-local tables so opaque extension payloads keep valid ordinal references. Preserve this metadata when copying an imported entry. Export preserves the tables and writes the logical tree inline by default. Use PlanToProtoConverter.From(plan, maxInlineRelExpressionDepth: 8) to generate detached subtrees with a combined inline relation/expression depth bound. Retained tables are unchanged and exempt from the bound; it is not a protobuf parser recursion limit. See detached plan imports for validation and ownership, and bounded export for the depth contract and compatibility details.

Custom ExtensionLeaf, ExtensionSingle, and ExtensionMulti relations preserve opaque detail payloads and common metadata. Supply an unmapped schema when constructing a relation, or an IExtensionRelationSchemaResolver when importing one. Unknown schemas remain unresolved rather than being guessed; schema-dependent operations fail explicitly. See extension schema contracts for examples, output mapping, and round-trip limitations.

IPlan.Metadata preserves plan-level advanced extensions, the ordered expected_type_urls, and imported extension URN/declaration tables. Export keeps all imported anchors, including aliases and apparently unused dependencies that opaque payloads may reference. Core references use one existing anchor per identity; new dependencies receive nonconflicting anchors. Use PlanBuilder.SetEnhancement, AddOptimization, and the related editing methods while composing a plan; earlier built plans remain immutable. Payload methods accept protobuf messages or read-only facades. Existing PlanMetadata can also be passed to the metadata-first plan constructor, Plan.FromRelations, or PlanBuilder. See plan extension metadata for construction, payload ownership, and compatibility details.

Plans can contain ordered mixtures of roots and reusable non-root relations. Use PlanBuilder to assign reference ordinals and safely share a registered subplan across outputs:

using Substrait.Core.Expression;
using Substrait.Core.Plan;
using Substrait.Core.Plan.Converters;
using Substrait.Core.Relation;
using Substrait.Core.Type;

var schema = new NamedStruct(
    ["order_id"], TypeFactory.REQUIRED.Struct([TypeFactory.REQUIRED.I64]));
var orders = new NamedTableRead(schema, ["orders"], filter: null);

var builder = new PlanBuilder();
Reference sharedOrders = builder.RegisterSubplan(orders); // Entry 0, not an output.
builder.AddRoot(sharedOrders, ["all_order_ids"]);         // Entry 1.
builder.AddRoot(
    new Project(sharedOrders, [new Literal.I64Literal(1)]),
    ["order_id", "marker"]);                             // Entry 2.

Plan plan = builder.Build();
Substrait.Protobuf.Plan protobuf = new PlanToProtoConverter().From(plan);
IPlan roundTrip = new ProtoToPlanConverter().From(protobuf);

IPlan.Relations is the authoritative ordered list, including non-root entries; Roots is only its root projection. References point at entry ordinals, not root indexes, and each registration creates a new entry without deduplication. Build() returns an immutable snapshot; later registrations do not change earlier plans. References returned by a builder belong to that builder.

Empty plans remain constructible and deserializable, but cannot be serialized. See multi-relation API migration for ordering, advanced composition, validation, and correlation limitations.

Contributing

Here are some ways you can contribute to the substrait-csharp project:

• Submit PRs to fix bugs or add new features.
• Review currently open PRs.
• Provide feedback and report bugs related to the software or the documentation.
• Enhance our design documents, examples, tutorials, and overall documentation.

To get started, read the contribution guide, then take a look at the issues and leave a comment if any of them interest you.

If you plan to make significant changes, open an issue to discuss them with the substrait-csharp community first. This helps ensure that your contributions align with the project's goals and avoids duplicating efforts.

Contributor License Agreement

Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

Code of Conduct

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

License

See the LICENSE file for more details.