Use provider C# reference map with pre-write internalization by live1206 · Pull Request #10976 · microsoft/typespec

live1206 · 2026-06-12T04:31:43Z

Summary

Replaces the C# generator's Roslyn reference-map/internalization path with a provider/expression-based reference map and pre-write provider accessibility updates while preserving generated-output correctness.

Latest changes

Builds generated-code reachability from TypeProvider metadata and provider expression/body dependency traversal instead of constructing the generated reference map through Roslyn.
Fully removes the generated-source Roslyn reference-map fallback for generated code; generated dependencies now come from provider metadata, expression trees, helper/body/signature dependency lists, and explicit provider relationships.
Applies internal/public accessibility before writing generated files by updating provider modifiers directly.
Skips the later Roslyn InternalizeAsync pass when pre-write accessibility has already been applied; removal and reduce still run through the existing post-processing pipeline.
Preserves XML docs for providers that are made internal before writing.
Removes model-factory methods for models that are internalized pre-write.
Handles custom-code roots through provider custom views plus customization-wide NamedTypeSymbolProviders, using lightweight metadata identity and syntax dependency extraction so arbitrary custom Roslyn symbols do not need full CSharpType/member materialization.
Keeps request-header extension helpers reachable from both generated expressions and custom-code syntax references.

How provider reference-map construction works

This PR no longer uses Roslyn to construct the generated-code reference map. Instead, generated type reachability is derived from the in-memory provider model before files are written.

Build all TypeProviders first, then run visitors, customized-member filtering, and back-compat processing so the analyzer sees the final provider shape.
Flatten generated providers into graph nodes, including top-level providers, nested providers, and serialization providers. Nodes are keyed by fully-qualified generated type name.
Add graph edges from provider metadata: provider type, base type, declaring type, implemented interfaces, nested/serialization providers, properties, fields, constructors, method signatures, attributes, generic arguments, arrays, collection definitions, helper/body/signature dependencies, and structured provider expression/body references.
Build both a public-signature graph and a full generated graph. The public graph decides what must remain public; the full graph decides what generated files/helpers are reachable for removal.
Seed reachability from known roots: public clients, additional configured roots, API-baseline generated types, custom-code references, generated public declarations, discriminator/derived model relationships, union variants when needed, and helper roots discovered after an initial reachability pass.
Compute candidates from reachability: unreachable public declarations become internalize candidates, internal/generated declarations needed by public API become publicize candidates, and unreachable full-graph declarations become remove candidates.
Apply accessibility before writing by updating provider modifiers directly. Model-factory methods for internalized/removed models are filtered for output, XML docs are preserved for types made internal, and Roslyn InternalizeAsync is skipped because the generated files already have the final accessibility.

Roslyn is still used by the existing post-processing pipeline for removal/reduce work and for reading custom-code symbols/syntax, but not for generated reference-map construction or applying internalization.

Latest benchmark data

Latest local full-generation benchmark compares Roslyn as the baseline against the current provider implementation with pre-write internalization and the latest custom-code/state-cleanup fixes. The values below are aggregated means across 3 BenchmarkDotNet runs with profiling disabled.

Benchmark	Roslyn baseline	Provider reference map	Improvement vs Roslyn
Full generation mean	855.0 ms / 68.43 MB	558.9 ms / 44.57 MB	34.6% faster, 34.9% less allocation

Per-run results:

Run	Roslyn baseline	Provider reference map	Improvement
Run 1	835.5 ms / 68.52 MB	582.3 ms / 44.56 MB	30.3% faster, 35.0% less allocation
Run 2	883.2 ms / 68.16 MB	542.4 ms / 44.49 MB	38.6% faster, 34.7% less allocation
Run 3	846.4 ms / 68.60 MB	552.0 ms / 44.67 MB	34.8% faster, 34.9% less allocation

Benchmark artifacts:

Run	Artifact
Latest no-profile full-generation run 1	`/tmp/typespec-provider-map-latest-20260701-0828-run1`
Latest no-profile full-generation run 2	`/tmp/typespec-provider-map-latest-20260701-0828-run2`
Latest no-profile full-generation run 3	`/tmp/typespec-provider-map-latest-20260701-0828-run3`
Previous no-profile full-generation run 1	`/tmp/typespec-provider-map-no-profile-20260630-1547-run1`
Previous no-profile full-generation run 2	`/tmp/typespec-provider-map-no-profile-20260630-1547-run2`
Previous no-profile full-generation run 3	`/tmp/typespec-provider-map-no-profile-20260630-1547-run3`
Previous provider reference-map reruns	`/tmp/typespec-hybrid-benchmark-reruns-20260629-065344`
Pre-write internalization	`/tmp/typespec-prewrite-internalize-benchmark-20260629-073203`

Azure SDK local project-reference benchmark

Measured Azure SDK for .NET management-plane regeneration using the same local project-reference setup in both runs: /workspaces/azure-sdk-for-net was wired to the sibling /workspaces/typespec checkout via .NET project references, and only the TypeSpec checkout changed between main and this PR branch.

SDK package	TypeSpec main baseline	This PR branch	Improvement
Azure.ResourceManager.Network	00:37:01.9	00:07:12.1	80.6% faster / 5.14x speedup
Azure.ResourceManager.DataFactory	00:13:43.4	00:03:33.9	74.0% faster / 3.85x speedup
Azure.ResourceManager.AppService	00:10:46.9	00:03:06.5	71.2% faster / 3.47x speedup

Validation

Latest Azure SDK full regen PR: [DO NOT MERGE] Preview Generator Version 1.0.0-alpha.20260625.1 (Azure data plane) Azure/azure-sdk-for-net#60254.
Local full Azure data-plane regen with the MTG RegenPreview.ps1 -Azure script regenerated 39 Azure-branded data-plane libraries with no SDK code diffs and no public API diffs.
- The full parallel run passed 38/39 libraries and hit a transient NuGet restore race in Azure.Data.AppConfiguration (*.nuget.g.props already existed`).
- Rerunning Azure.Data.AppConfiguration alone with the same local generator passed, leaving no SDK/API diffs.
Targeted local Azure data-plane regen for Azure.AI.Vision.ImageAnalysis passed after the custom-code stack-overflow fix.
- The remaining SDK changes are expected generated cleanup/reachability diffs: unused System/Azure usings are removed from the model factory, and internal error models are generated/marked buildable in the MRW context.
C# emitter npm run format passes.
Repo-root pnpm format passes; unrelated formatter changes outside http-client-csharp were discarded.
Full Microsoft.TypeSpec.Generator.ClientModel.Tests pass (1451 tests).
C# emitter npm run build -- --no-restore passes.
Focused Microsoft.TypeSpec.Generator.Tests customization/post-processing tests pass (ClientCustomizationTests, PostProcessorTests.RemovesInvalidUsings).
C# emitter npm run cop passes.

pkg-pr-new · 2026-06-12T04:34:05Z

Open in StackBlitz

npm i https://pkg.pr.new/@typespec/http-client-csharp@10976

commit: 9951f4b

github-actions · 2026-06-12T04:34:26Z

All changed packages have been documented.

✅ @typespec/http-client-csharp

Show changes

`@typespec/http-client-csharp` - internal ✏️

Improve generated C# reference-map analysis by deriving generated body references from provider expression trees.

This reverts commit ff55509.

live1206 · 2026-07-01T01:10:18Z

Follow-up on the remaining samples question from the latest review: the provider remove path only computes removal candidates from generated provider graph nodes, not from arbitrary project symbols. Sample files are generated documents but they are not provider nodes, so sample classes themselves are not selected as remove candidates by the provider graph. For generated types referenced only by samples, the full DPG regen/no-diff validation is the practical coverage; I do not see an additional code change needed here. The remaining static-state concern is still a known low-priority follow-up rather than a blocker for this PR.

JoshLove-msft · 2026-07-01T01:14:26Z

Design suggestion: derive body references from expression trees rather than declared deps + source re-parsing

Following up on the body-reference discussion — the current approach builds graph edges from provider metadata (signatures, properties, fields, attributes, etc.) and then compensates for body-level references three ways: manually-declared HelperDependencyNames / BodyDependencyTypes / IncludeGeneratedBodyReferences, AddGeneratedBodyReferences re-parsing generated source with Roslyn for a curated set of "candidate" providers, and hardcoded fallbacks (the SetDelimited scan, ChangeTracking helper roots, etc.).

The concern is soundness: a pure-metadata graph is an under-approximation of reachability, because a C# type reference can appear anywhere a method body can (new Helper(), SomeStatic.Call(), extension calls like request.Headers.SetDelimited(...), casts, typeof). For a removal/internalization pass, under-approximating is the dangerous direction — you conclude "nothing references X," internalize/remove it, and the break surfaces later as a downstream compile error or a silent public-API regression. Correctness then rests on provider authors remembering to declare every body dependency (a manually-maintained parallel list that will drift) plus the hand-curated candidate list.

Suggestion: walk the body expression trees instead. The bodies already exist as structured MethodBodyStatement / ValueExpression trees before they're rendered to source. Traversing those to collect CSharpType references (constructor targets, static member owners, type literals, generic args, cast targets) would be:

sound by construction for anything the generator itself emits — it closes the gap generically instead of per-provider declaration, and would subsume the SetDelimited special-case and most BodyDependencyTypes/HelperDependencyNames overrides;
likely cheaper than AddGeneratedBodyReferences re-parsing generated C# through a Roslyn semantic model.

The fact that the PR re-parses generated source for the client/serialization candidates (rather than walking expression trees) suggests the tree walk wasn't feasible for all bodies — presumably where bodies are raw TypeProvider source or interpolated string literals whose contents aren't structured ValueExpressions. If so, that's the honest residual limitation to document: expression-tree walking would handle the structured majority soundly, with a narrow, explicit fallback (or a CI superset-check against the existing full Roslyn map) for the unstructured remainder — instead of today's broader reliance on manual declarations + name-based special-cases.

Not a merge blocker, but I think it's the change that would turn this from "correct-by-testing" into "correct-by-construction." Would also be worth attributing how much of the ~21% comes from skipping body analysis vs. from pre-write internalization (skipping the Roslyn InternalizeAsync/reduce passes) — if it's mostly the latter, a sound full-body map may retain most of the win.

--generated by Copilot

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

… mtg-hybrid-reference-map # Conflicts: # packages/http-client-csharp/generator/Microsoft.TypeSpec.Generator.ClientModel/test/PostProcessing/ClientBodyDependencyPostProcessingTests.cs # packages/http-client-csharp/generator/Microsoft.TypeSpec.Generator/src/PostProcessing/ProviderReferenceMapAnalyzer.cs # packages/http-client-csharp/generator/Microsoft.TypeSpec.Generator/src/Providers/TypeProvider.cs

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

This reverts commit ec68ee3.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

jorgerangel-msft · 2026-07-01T16:35:53Z

@@ -0,0 +1,7 @@
+---


we should remove this change

jorgerangel-msft · 2026-07-01T16:36:09Z

 changelog: ["@chronus/github/changelog", { repo: "microsoft/typespec" }]

 additionalPackages:
+  - packages/http-client-csharp


was this intentional?

jorgerangel-msft · 2026-07-01T16:38:20Z

+            get
+            {
+                var ns = _namedTypeSymbol.ContainingNamespace.GetFullyQualifiedNameFromDisplayString();
+                return string.IsNullOrEmpty(ns) ? _namedTypeSymbol.Name : $"{ns}.{_namedTypeSymbol.Name}";


can we also cache this value please so it's not calculated each time?

jorgerangel-msft · 2026-07-01T16:44:24Z

+
+            foreach (var outputType in output.TypeProviders)
+            {
+                if (outputType is ModelFactoryProvider && outputType.Methods.Count == 0)


why is this only needed for model factories ?

jorgerangel-msft · 2026-07-01T16:46:15Z

+                    CodeModelGenerator.Instance.AdditionalRootTypes
+                        .Concat(CodeModelGenerator.Instance.NonRootTypes)
+                        .Select(GetFileNameForType))
+                .Distinct(StringComparer.Ordinal)


hmm, is the distinct necessary? The root types are a hashset, is it possible for there to be duplicates once we get the filename?

jorgerangel-msft · 2026-07-01T16:52:25Z

+                .ToArray();
+        }
+
+        private static string GetFileNameForType(string typeName)


Looking at this further, why do we need to explicitly do this now ? The types to keep were already being checked in the postprocess step. If we must do this here, should we try and find the corresponding type in the outputlibrary by name, and use it's RelativePath to get the name of the file instead?

jorgerangel-msft · 2026-07-01T16:58:52Z

+            var generatedPublicReachable = GetReachableTypes(internalizeRoots, internalizeReferences);
+            AddDerivedModelReferences(providers, publicGraph.Nodes, internalizeReferences, generatedPublicReachable, generatedDiscriminatorBaseNames);
+            internalizeRoots.UnionWith(customPublicRoots);
+            var internalizeReachableWithoutHelpers = GetReachableTypes(internalizeRoots, internalizeReferences);
+            AddDerivedModelReferences(providers, publicGraph.Nodes, internalizeReferences, internalizeReachableWithoutHelpers, generatedDiscriminatorBaseNames);
+            internalizeReachableWithoutHelpers = GetReachableTypes(internalizeRoots, internalizeReferences);
+            var publicizeRoots = internalizeRoots.ToHashSet(StringComparer.Ordinal);
+            var internalizeHelperRoots = GetHelperRootNames(generatedProviders, graph.Nodes, internalizeReachableWithoutHelpers);
+            internalizeRoots.UnionWith(internalizeHelperRoots);
+            var internalizeReachable = GetReachableTypes(internalizeRoots, internalizeReferences);
+            var internalizeDeclaredNodes = GetPostProcessorDeclaredNodes(generatedProviders, graph.Nodes, publicOnly: true);
+            var customInternalBoundaryNodes = graph.Nodes
+                .Where(name => publicGraph.References.TryGetValue(name, out var references) && references.Overlaps(customInternalDeclarations))
+                .ToHashSet(StringComparer.Ordinal);
+            var publicizeDeclaredNodes = GetPostProcessorDeclaredNodes(generatedProviders, graph.Nodes, publicOnly: false)
+                .Except(internalizeDeclaredNodes, StringComparer.Ordinal);
+            var generatedImplementationInternalDeclarations = GetGeneratedImplementationInternalTypeDeclarations(generatedInternalDeclarations).ToHashSet(StringComparer.Ordinal);
+            var publicApiTraversalNodes = internalizeDeclaredNodes
+                .Except(generatedInternalDeclarations, StringComparer.Ordinal)
+                .Concat(publicizeDeclaredNodes)
+                .Except(generatedImplementationInternalDeclarations, StringComparer.Ordinal)
+                .ToHashSet(StringComparer.Ordinal);


I will be honest and say this is quite difficult to follow ☹️

live1206 added 2 commits June 12, 2026 04:30

Profile C# post-processing benchmarks

ff55509

Add provider reference map shadow replacement

f022e78

microsoft-github-policy-service Bot added the emitter:client:csharp Issue for the C# client emitter: @typespec/http-client-csharp label Jun 12, 2026

live1206 added 25 commits June 12, 2026 06:31

Revert "Profile C# post-processing benchmarks"

f88a64a

This reverts commit ff55509.

Remove hybrid map diagnostics from generator

2d57265

Rename hybrid reference map components

dc145f4

Document hybrid reference map decisions

0cc3abf

Fix hybrid reference map CI failures

c2abe79

Track serialization providers in hybrid reference map

c463995

Handle model factory subclasses in reference map

9a1d1a1

Track generated body type references

2917111

Match Roslyn cleanup for body references

2cda184

Remove unused serialization providers

ce16922

Merge remote-tracking branch 'origin/main' into mtg-hybrid-reference-map

7a32814

Preserve public discriminator subtypes

5a8248f

Use explicit serialization helper dependencies

43529f3

Use explicit collection result dependencies

21ecc1a

Use explicit client body dependencies

f3259a4

Track rest client helper dependencies

b6e8aad

Skip stubbed client body dependencies

48885cb

Internalize models from final API reachability

c6539de

Update model root visibility test

f676243

Detect custom model factory providers

0a37522

Preserve file header when removing usings

582fa84

Limit derived public reachability to discriminators

74be26d

Ignore custom roots for derived public reachability

dc6f0f2

Propagate public discriminator reachability transitively

7c20ef2

Keep enum serialization helpers reachable

2e4af08

live1206 and others added 13 commits July 1, 2026 02:47

fix(csharp): avoid hardcoded helper keep files

e1a5d68

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

refactor(csharp): share provider accessibility candidate logic

62dff6f

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

fix(csharp): preserve custom request header helpers

f36c8c8

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

refactor(csharp): use provider markers in reference map

66c53ba

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

refactor(csharp): use typed helper dependencies

b7b1eec

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

refactor(csharp): root request header helpers from providers

3ab57ae

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Optimize C# reference map body analysis

9832e1a

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Fix custom-code reference analysis stack overflow

79903af

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Merge remote-tracking branch 'origin/main' into mtg-hybrid-reference-map

eb0b9ac

Update CSharp Spector regen baselines

ec68ee3

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Revert "Update CSharp Spector regen baselines"

6665e3a

This reverts commit ec68ee3.

Fix custom partial model reachability

ee53dab

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

live1206 requested review from bterlson, catalinaperalta, iscai-msft, markcowl, timotheeguerin and witemple-msft as code owners July 1, 2026 06:18

live1206 and others added 7 commits July 1, 2026 07:01

Fix CSharp customization test output path

fa098f7

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Fix provider reference map state cleanup

049dd2b

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Fix CSharp provider helper reachability

a792ee9

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Address CSharp provider review feedback

57c588f

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

fix(http-client-csharp): honor internal custom partials in provider map

5ffff14

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

refactor(http-client-csharp): remove redundant buildable attribute check

efabcde

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

refactor(http-client-csharp): simplify invalid attribute cleanup

9951f4b

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

live1206 marked this pull request as draft July 1, 2026 12:29

jorgerangel-msft reviewed Jul 1, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Use provider C# reference map with pre-write internalization#10976

Use provider C# reference map with pre-write internalization#10976
live1206 wants to merge 89 commits into
microsoft:mainfrom
live1206:mtg-hybrid-reference-map

live1206 commented Jun 12, 2026 •

edited

Loading

Uh oh!

pkg-pr-new Bot commented Jun 12, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 12, 2026 •

edited

Loading

`@typespec/http-client-csharp` - internal ✏️

Uh oh!

live1206 commented Jul 1, 2026

Uh oh!

JoshLove-msft commented Jul 1, 2026

Uh oh!

jorgerangel-msft Jul 1, 2026

Uh oh!

jorgerangel-msft Jul 1, 2026

Uh oh!

jorgerangel-msft Jul 1, 2026

Uh oh!

jorgerangel-msft Jul 1, 2026

Uh oh!

jorgerangel-msft Jul 1, 2026

Uh oh!

jorgerangel-msft Jul 1, 2026

Uh oh!

jorgerangel-msft Jul 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

live1206 commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Latest changes

How provider reference-map construction works

Latest benchmark data

Azure SDK local project-reference benchmark

Validation

Uh oh!

pkg-pr-new Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

@typespec/http-client-csharp - internal ✏️

Uh oh!

live1206 commented Jul 1, 2026

Uh oh!

JoshLove-msft commented Jul 1, 2026

Design suggestion: derive body references from expression trees rather than declared deps + source re-parsing

Uh oh!

jorgerangel-msft Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

jorgerangel-msft Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

jorgerangel-msft Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

jorgerangel-msft Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

jorgerangel-msft Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

jorgerangel-msft Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

jorgerangel-msft Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

live1206 commented Jun 12, 2026 •

edited

Loading

pkg-pr-new Bot commented Jun 12, 2026 •

edited

Loading

github-actions Bot commented Jun 12, 2026 •

edited

Loading

`@typespec/http-client-csharp` - internal ✏️