[Investigation] CI infrastructure failures — npm public-registry slowdown

[Mitch Denny (mitchdenny)]

Purpose

Whitespace-only commit to trigger CI so we can observe and diagnose current infrastructure failures.

Not for merge. Diagnostic vehicle only — no product code is changed.

Conclusion

This is an npm problem, not a NuGet problem.

npm install against the public registry (registry.npmjs.org) has become drastically slow from CI runners — ~2m21s vs the normal ~4s — so every Aspire path that shells out to npm install blows its 120s timeout. NuGet restore is healthy (1.1s in the same failing job) and there is no evidence of a NuGet network-isolation lockdown.

The decisive evidence is two npm install calls minutes apart inside the same CI container:

Invocation	Registry	Duration
Docker global install (explicit --registry)	dnceng dotnet-public-npm mirror	8s ✅
Guest AppHost install (default registry)	registry.npmjs.org	2m21s ⚠️
Next guest install (default registry)	registry.npmjs.org	killed at 120s, exit 143 ❌

Generated apps are created outside the repo (/tmp/...), so they never inherit the repo's dnceng .npmrc and fall back to the public registry — which is why in-repo npm work (extension build, markdownlint) is unaffected while every generated TypeScript AppHost stalls.

Ruled out: NuGet isolation lockdown, an npm.js status-page incident (the only recent one was publish-only and resolved ~4h earlier), the Copilot agent firewall doc, and any repo regression in the window. AzureSandboxes is a separate, unrelated failure.

Full detail, log excerpts and the regression window are in the comments below, along with a correction on mitigation: the CLI deliberately pins the public registry (NpmRunner.cs, see #19370) because the private mirror 401s on unmirrored transitive dependencies — so redirecting to the mirror is not a safe fix.

Verification

• Full deployment E2E run on main: 33834532243
• This PR touches a TypeScript polyglot fixture so job:polyglot exercises the failing TypeScript SDK Validation job directly.

[github-actions]

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 19928

Or

• Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 19928"

[Copilot]

Copilot review overview

🟡 Changes recommended

This diagnostic-only PR is explicitly marked as not for merge.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review tier: Balanced
Findings: None

What changed in this PR

Diagnostic-only PR adding whitespace to trigger CI; it is explicitly not intended for merge.

Changes:

• Adds a blank line to README.md.

File	Description
README.md	Adds trailing whitespace to trigger CI.

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[Mitch Denny (mitchdenny)]

Investigation findings

TL;DR — this is an npm problem, not a NuGet problem. npm install against the default public registry (registry.npmjs.org) has become drastically slow from CI runners (~2m21s vs the normal ~4s). Every Aspire code path that shells out to npm install then blows its 120s timeout. NuGet restore is healthy and there is no evidence of a NuGet network-isolation lockdown.

Regression window

Run	Time (UTC)	TypeScript deployment tests
33706458459	Sep 3 02:08	5/5 pass
33715044346	Sep 3 04:26	5/5 pass
33826035047	Sep 4 01:30	0/5 pass
33832301417	Sep 4 03:11	0/5 pass

CI on main shows the same break: last green was 33791618334 (Sep 3 18:37); everything from 33802459402 (Sep 3 20:28) onward is red.

Evidence 1 — the hang is exactly at npm install

From the aspire add CLI log in run 33832301417 (TypeScriptExpressDeploymentTests):

[03:27:43] Restored .../package-restore/2D6EE675D27CD7F4/obj/project.json (in 1.1 sec).   <-- NuGet fine
[03:27:45.781] [GuestAppHostProject] Generated 3 TypeScript files in .aspire/modules
[03:27:45.791] [GuestAppHostProject] Executing: /usr/local/bin/npm install
[03:27:45.792] [GuestAppHostProject] typescript/nodejs guest process 6107 started: /usr/local/bin/npm
<<< log ends here — test killed at 03:29:27 >>>

NuGet restore completed in 1.1 seconds. The log then dead-ends at npm install. Byte-identical behaviour in run 33826035047 (process 6077).

For contrast, the same step in the last green run (33715044346):

[04:39:45.781] Executing: /usr/local/bin/npm install
[04:39:53.876] added 112 packages, and audited 113 packages in 4s

4 seconds then, indefinite now.

Evidence 2 — the smoking gun (same job, two registries)

In CI job 100866611880 (Polyglot SDK Validation / TypeScript SDK Validation (Node 24.x)), two npm install invocations run minutes apart in the same container:

Invocation	Registry	Duration
Docker global install (explicit --registry)	pkgs.dev.azure.com/dnceng/.../dotnet-public-npm	8s ✅
Guest AppHost project install (default registry)	registry.npmjs.org	2m21s ⚠️
Next guest install (default registry)	registry.npmjs.org	killed at 120s, exit 143 ❌

#8 8.045 added 8 packages, and changed 1 package in 8s          <-- dnceng mirror

[00:33:30] Executing: /usr/local/bin/npm install
[00:35:51] typescript/nodejs guest process 491 exited with code 0   <-- 2m21s, default registry

[00:35:53] Executing: /usr/local/bin/npm install
[00:37:51] Cancellation requested while waiting for ... process 572
[00:37:51] typescript/nodejs guest process 572 exited with code 143
❌ Failed to install TypeScript (Node.js) dependencies.
❌ Timed out waiting 120s for AppHost to start.

The dnceng npm mirror is fast; the public registry is not. That is the whole bug.

Why only generated/TypeScript apps are affected

The repo pins the internal mirror at the root:

./.npmrc          registry=https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-public-npm/npm/registry/
./extension/.npmrc  (same)

But the templates ship no .npmrc, and generated apps are created outside the repo (/tmp/Aspire.Deployment.EndToEnd.Tests/Workspace/...), so they never inherit that config and fall back to registry.npmjs.org. That is precisely why in-repo npm work (extension build, markdownlint) is unaffected while every generated TypeScript AppHost stalls.

Hypotheses checked and ruled out

• NuGet network-isolation lockdown — no evidence. All sources in NuGet.config are dnceng (pkgs.dev.azure.com / dnceng.pkgs.visualstudio.com); there is no public api.nuget.org feed. Restores succeed, and observed NotFound responses are just normal package-source-mapping fallthrough before a successful install from dotnet-eng. Deployment-test NuGet restore completed in 1.1s.
• npm.js outage — not per the status page. status.npmjs.org reports All Systems Operational, no active incidents. The only recent incident (Intermittent Failures Impacting npm Publish, Sep 3 21:42→23:21Z) was publish-only and resolved ~4h before run 33832301417. So this is degradation/throttling not reflected on the status page, or egress filtering on the runners.
• docs/ci/copilot-agent-firewall.md (commit 9395ca9e63, Sep 3 19:20) — red herring. It documents a probe for the Copilot coding-agent shell, not GitHub Actions runners, and is docs-only.
• A repo regression — unlikely. No commit in the regression window touches the guest/TypeScript npm install path.
• AzureSandboxes failure — separate issue. It fails at WaitUntil("success prompt [5 OK] $") after 7m17s, unrelated to npm.

Registry reachability from a non-CI network is healthy for both (registry.npmjs.org ~0.08–2.9s, dnceng mirror ~1.9s), reinforcing that the slowness is specific to the CI egress path.

Suggested mitigation

Point generated AppHosts at the internal mirror instead of the public registry, e.g. ship an .npmrc with the TypeScript templates or set npm_config_registry in the environment the CLI passes to the guest runtime (GuestAppHostProject.InstallDependenciesAsync). That both fixes CI and removes a hard runtime dependency on npmjs.org — the same isolation posture already applied to NuGet. Raising the 120s timeout would only mask it.

Verification in flight

• Full deployment E2E run on main: 33834532243
• This PR touches a TypeScript polyglot fixture so job:polyglot runs the failing TypeScript SDK Validation job directly.

[Mitch Denny (mitchdenny)]

Correction / refinement on the suggested mitigation

My earlier suggestion — "point generated AppHosts at the internal dnceng mirror" — was too glib. There is a deliberate design decision that says the opposite, and it should not be reversed casually.

src/Aspire.Cli/Npm/NpmRunner.cs:

/// The canonical public npm registry URL. Commands that resolve, pack, or install
/// packages pass this explicitly via <c>--registry</c> so resolution and install use
/// the public feed and cannot inherit a project-level <c>.npmrc</c> that redirects to a
/// private feed (for example an Azure DevOps Artifacts feed). Such a private feed would
/// otherwise return 401 for packages (including transitive dependencies) it has not
/// mirrored, breaking <c>aspire agent init</c>.
/// See https://github.com/microsoft/aspire/issues/19370.
private const string PublicRegistry = "https://registry.npmjs.org/";

So the CLI intentionally forces registry.npmjs.org and explicitly refuses to inherit a private-feed .npmrc, because the dnceng mirror 401s on transitive dependencies it hasn't mirrored (#19370). Redirecting guest installs to the mirror would trade today's slowness for a different, previously-fixed class of failure.

Worth noting the two npm paths are distinct:

• NpmRunner — pins --registry https://registry.npmjs.org/ explicitly.
• GuestAppHostProject.InstallDependenciesAsync — the failing path, runs a bare npm install and so resolves via ambient config, which in a generated /tmp app is also registry.npmjs.org.

Both therefore land on the public registry. The dependency is by design, not by accident.

Revised options

1. Wait it out / confirm with GitHub. If this is transient npmjs degradation or Actions egress throttling, no code change is warranted. Given the status page is green while runners see a ~35x slowdown, this is worth raising with GitHub Actions support as an egress issue.
2. Add retry + a longer, configurable timeout on guest dependency install. The current 120s budget is a hard cliff, and an install that legitimately takes 2m21s fails even though it would have succeeded. This is the lowest-risk durable fix and helps real users on slow networks too.
3. A pull-through cache that falls back to the public registry (rather than a partial mirror) would give isolation without the 401 problem from npm registry hard-coded to internal dnceng feed — breaks aspire agent init, and ships as the customer default #19370 — the correct long-term fix, but a larger piece of work.

I'd suggest (1) + (2): confirm with GitHub whether runner egress to registry.npmjs.org is throttled, and independently make the guest install timeout configurable with retry so a slow-but-working registry degrades gracefully instead of failing the run.

I have not made any product change in this PR — it remains whitespace-only for diagnosis.

[github-actions]

Tests selector

0 / 99 PR test projects · 1 PR job · 0 advisory-only targets, from 1 changed file.

Selected PR test projects (0 / 99)

none — no PR-gated .NET test projects run for this change.

Selected PR jobs (1)

polyglot

Advisory workflow impact (0)

none

---

How these were chosen — grouped by what changed

Job reasons

Job	Triggered by
polyglot	tests/PolyglotAppHosts/Aspire.Hosting/TypeScript/apphost.mts

---

Selection computed for commit 33558ab.

[Copilot]

Copilot review overview

🟡 Changes recommended

The pull request is explicitly diagnostic and not intended for merge.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review tier: Balanced
Findings: None

[Mitch Denny (mitchdenny)]

Full deployment run — reproduced, plus an important refinement

Full deployment E2E run on main: 33834532243. TypeScript failed 5/5 again, same set as the nightly:

TypeScriptExpress
TypeScriptJavaScriptHosting
TypeScriptAzureContainerAppJob
TypeScriptVnetSqlServerInfra
AksAzureKubernetesEnvironmentCertManagerTypeScript
AzureSandboxes            <-- separate, unrelated failure (prompt timeout, not npm)

31 of 46 deploy jobs passed, 7 failed, 8 still running at time of writing. The failure set is stable and reproducible across three independent runs.

Refinement: the stall is intermittent, not a uniform slowdown

I need to correct an implication in my first comment. It is not the case that npm is uniformly slow — the same npm install is sometimes fast and sometimes stalls, within the same run:

TypeScriptExpress (run 33834532243):

[03:59:44.596] Executing: /usr/local/bin/npm install
[03:59:51.276] added 112 packages, and audited 113 packages in 4s     <-- FAST (112 packages!)

[04:00:13.554] Executing: /usr/local/bin/npm install
[04:00:13.555] typescript/nodejs guest process 5981 started
<<< no further output — killed by the 120s timeout >>>

The heavy install (112 packages, real network work) finished in 4s. The following install stalled indefinitely with zero npm output.

And it is not positional either — in TypeScriptJavaScriptHosting the first install (during aspire new, via ScaffoldingService) is the one that stalled:

[03:59:31.797] [ScaffoldingService] Executing: /usr/local/bin/npm install
<<< no completion >>>

So: any given npm install may stall, unpredictably. Observed durations for the same logical operation across runs: 4s → 709ms → 2m21s → indefinite.

What this means

This is consistent with intermittent network stalls to registry.npmjs.org from CI runners rather than a clean outage or a deterministic code bug — which also explains why npm's status page is green. The 120s timeout then converts a stalled connection into a hard test failure.

The core conclusions from my first comment stand:

• NuGet is healthy; restore completes in ~1.1s in the very same failing jobs. No NuGet network-isolation lockdown.
• Installs pinned to the dnceng npm mirror (--registry) have not been observed to stall; only default-registry installs stall.
• Generated apps live outside the repo and so don't inherit the repo .npmrc.

Still open

npm emits no output at all when it stalls, so I can't yet say which request hangs (tarball fetch, metadata, or the audit endpoint). Confirming that needs npm install --loglevel verbose (or --no-audit as an A/B) in the guest install path. I'd treat that as the next diagnostic step if this doesn't clear on its own.

Recommendation is unchanged from my previous comment: raise the runner-egress question with GitHub, and independently make the guest dependency install retry with a configurable timeout so a stalled connection degrades gracefully instead of failing the run outright.

[github-actions]

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

[Mitch Denny (mitchdenny)]

Reproduced on this PR — decisive evidence

The job:polyglot trigger worked and TypeScript SDK Validation (Node 24.x) failed on this PR (job 100906815325), on a branch whose only diff is a trailing newline in a fixture. That rules out anything in the change itself.

The log captures three npm install invocations in a single job, on a single runner, minutes apart:

#	Started	Caller	Duration
1	04:05:59	ScaffoldingService (aspire new)	3m 53s — succeeded
2	04:10:10	GuestAppHostProject (aspire add)	33s — succeeded
3	04:10:46	GuestAppHostProject	> 120s — killed

04:05:59 ScaffoldingService: Executing: /usr/local/bin/npm install
04:09:52   ...next activity (3m53s later)
04:10:10 GuestAppHostProject: Executing: /usr/local/bin/npm install
04:10:43   ...next activity (33s later)
04:10:46 GuestAppHostProject: Executing: /usr/local/bin/npm install
04:12:43 Stderr: Failed to install TypeScript (Node.js) dependencies.
04:12:43 Stderr: Timed out waiting 120s for AppHost to start.

Same command, same machine, same registry, same minutes: 3m53s / 33s / >120s. For reference, on the last green run the equivalent sequence was 4s / 709ms / 468ms.

That is the whole story. npm install against registry.npmjs.org from GitHub Actions runners is currently erratic and often two orders of magnitude slower than baseline, and Aspire's hard 120s AppHost-start timeout turns the slow tail into a hard failure.

Summary of the investigation

Confirmed

• npm-side degradation reaching the public registry from CI runners. Reproduced in 4 independent runs (2 nightly, 1 dispatched full run, 1 on this PR).
• Blast radius: generated TypeScript/guest AppHosts only. They are created in /tmp, outside the repo, so they don't inherit the repo .npmrc that pins the dnceng mirror. In-repo npm work (extension build, lint) is unaffected — as is every non-TypeScript deployment test.
• Full deployment run on main (33834532243): TypeScript 5/5 failed, 31 other deploy jobs passed.

Refuted

• ❌ No NuGet network-isolation lockdown. NuGet restore completes in ~1.1s inside the very same failing jobs. Root NuGet.config contains only dnceng feeds — no api.nuget.org — so a public-egress lockdown could not affect it anyway. Nothing in the regression window touches feed or isolation config.
• ❌ Not an Aspire regression — reproduces on a whitespace-only branch, and nothing in the Sep 3 04:26→20:28 UTC window touches the guest npm path.
• ❌ Not visible on npm's status page (green; the only recent incident was publish-only and resolved hours earlier).

Unrelated

• AzureSandboxes fails separately (RedeployProjectToAzureSandboxRetainsPreviousPublicUrl, prompt timeout at 7m17s). Not npm.

Suggested next steps

1. Raise runner egress to registry.npmjs.org with GitHub Actions support — this looks like an infrastructure/peering problem on their side, not ours.
2. Make the guest dependency install resilient — it currently has a hard 120s cliff with no retry. Retry with backoff plus a configurable timeout would ride out slow tails instead of failing the run. This is worth doing regardless of the current incident.
3. Add --loglevel verbose (or A/B --no-audit) to the guest install if we need to pin down which request stalls; npm currently emits nothing while hung.

⚠️ Do not simply repoint guest installs at the dnceng mirror. src/Aspire.Cli/Npm/NpmRunner.cs:17-26 deliberately forces the public registry because the ADO mirror 401s on unmirrored transitive dependencies (#19370). A mirror-based fix has to solve that first.

This PR is diagnostic only — no product code changed. Happy to open a separate PR for the retry/timeout hardening in (2) if that's wanted.

[Mitch Denny (mitchdenny)]

Final tally — full deployment run 33834532243

38 passed / 7 failed. Failures split cleanly into two buckets:

npm-related (5) — all TypeScript, all the same stall:

• TypeScriptExpress
• TypeScriptJavaScriptHosting
• TypeScriptAzureContainerAppJob
• TypeScriptVnetSqlServerInfra
• AksAzureKubernetesEnvironmentCertManagerTypeScript

Not npm-related (2) — Azure-side, pre-existing:

• AzureSandboxes — RedeployProjectToAzureSandboxRetainsPreviousPublicUrl, prompt timeout at 7m17s.
• VnetSqlServerConnectivity — Azure provisioning failure (❌ Pipeline failed). No npm install anywhere in the job; an ARM/provisioning issue, not network egress.

So the npm degradation accounts for exactly the 5 TypeScript deployment tests and nothing else. Every non-TypeScript deployment test that isn't hitting a separate Azure problem passes — further confirmation that neither NuGet nor general network isolation is implicated.

[github-actions]

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

[github-actions]

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.