Add terraform utility by Modrats · Pull Request #103 · microsoft/discovery

Terri Mod (Modrats) · 2026-07-14T07:55:40Z

Summary

A Terraform utility under utilities/terraform/ that provisions a complete Microsoft Discovery environment as an alternative to the Bicep path. It deploys the Discovery control plane (supercomputer with a node pool; workspace with a chat model deployment and project; storage container; optional bookshelf) together with the platform prerequisites it needs (VNets and subnets, four least-privilege managed identities with scoped RBAC, a storage account, and private networking).

Approach

Provider split. AzAPI drives every Microsoft.Discovery/* resource (not yet in the AzureRM catalog), pinned to GA API 2026-06-01; AzureRM handles all platform primitives for strongly-typed schemas and stable state.
Layered modules, single source of truth. modules/platform (network, identities, storage, RBAC, storage container) and modules/control-plane/* (supercomputer, workspace, bookshelf, tool). The root (main.tf) composes them into a single-apply end-to-end deployment — there is one definition of each resource, with no inlined Discovery or platform resources in the root.
Single full-stack path. The root builds the entire environment in one apply; it has no "bring your own" flag. Reusing an existing Supercomputer, Workspace, or Bookshelf is an advanced escape hatch: because the control-plane modules consume only IDs and never create platform resources, you call one directly and thread your existing resource in yourself (see the README "Reuse an existing resource" section and ADR 0002). Keeping the root single-branch also makes the supercomputer network unconditional, so its topology matches the Bicep by construction and cannot drift.
Tool module (extension). modules/control-plane/tool ships as a standalone control-plane module for provisioning Discovery Tools. It is intentionally not wired into the end-to-end root — it is an extension point for teams that provision Tools, callable directly with your own resource IDs.
Split-region managed resource groups. The control plane stays in one region while each resource's managed infrastructure — for example the GPU supercomputer — can run in another via the discovery.overridemrgregion tag, backed by a peered supercomputer VNet and cross-VNet private DNS.
Tagging. common_tags plus per-resource tag inputs merge so required platform tags are always preserved.
Tooling. A deterministic preflight.sh (RP registration, region and VM-SKU allowlists, compute-cores quota, pluggable checks under preflight-checks/) and an idempotent destroy.sh that handles Discovery-specific teardown ordering.

Type of change

Other (describe below)

Other: New IaC deployment utility under utilities/terraform/. Parallels the existing Bicep deployment path; no agent, starter-kit, schema, or workflow surface is touched.

Related issue / tracking

n/a

Schema impact

No schema changes

Validation checklist

terraform fmt and terraform validate pass for the root and every module.
Validated with a real end-to-end terraform apply into a fresh resource group (supercomputer + node pool, workspace + chat model + project, storage container).
Documentation is updated: README.md covers the quickstart, architecture, how to deploy, and reusing an existing resource; ADRs live under docs/adr/ (0001 network isolation posture, 0002 single full-stack path).
No OS/editor artifacts or secrets committed. State, .terraform/, *.tfplan, and real *.tfvars are gitignored.
No model-weight files added.
All Markdown links resolve.

Reviewer notes

Provider split is intentional — Microsoft.Discovery/* types are not in the AzureRM catalog yet. See the README intro for the rationale.
Modules are the single source of truth. The root has no inlined Discovery resources; the modules/control-plane/* modules are the only definition. Reusing an existing resource means calling those same modules directly, so there is no duplicate to drift.
Full stack is the only root-wired path. BYO variables were removed to keep the root single-branch and the supercomputer network unconditional (in parity with the Bicep). See ADR 0002.
Workspace children ordering. The project depends on a Succeeded chat model, which the reference graph does not otherwise express; the workspace module encodes this with depends_on between its children.
Preflight is deterministic-only by design — no capacity or throttling probes that could give false confidence. Add-on checks under preflight-checks/NN-*.sh are auto-sourced; the contract is documented in the README.
Supported regions: eastus, uksouth, swedencentral. (eastus2 is advertised by the Discovery RP but rejects new supercomputer creates, so it is excluded by the location validation and preflight.sh.)

github-actions · 2026-07-14T07:58:57Z

✅ Automated Check Results — All checks passed

All structural, schema, content, documentation, and secret scan checks passed.

This PR is ready for human review. The maintainers have been automatically requested.

github-actions · 2026-07-14T07:59:54Z

✅ Automated Check Results — All checks passed

All structural, schema, content, documentation, and secret scan checks passed.

This PR is ready for human review. The maintainers have been automatically requested.

github-actions · 2026-07-21T08:57:51Z

❌ Automated Check Results — 2 issue(s) found

All issues are listed below. Please address all of them before requesting human review.

Rule ID	File	Issue
`SEC-SCAN`	(repository)	Secret scan failed — potential credentials detected. Check the TruffleHog job for details.
`INTERNAL`	`(workflow)`	Validation job was skipped or did not produce results — check the Validate PR job logs for details.

Once all issues are resolved, push a new commit to re-trigger this check.
When the pr-review validator passes, the pr-validation-passed label will be applied and the maintainers will be auto-requested for review. Note: other status checks (unit-tests, schema regression, etc.) report separately — see the full status rollup at the bottom of this PR.

github-actions · 2026-07-21T09:41:33Z

❌ Automated Check Results — 2 issue(s) found

All issues are listed below. Please address all of them before requesting human review.

Rule ID	File	Issue
`SEC-SCAN`	(repository)	Secret scan failed — potential credentials detected. Check the TruffleHog job for details.
`INTERNAL`	`(workflow)`	Validation job was skipped or did not produce results — check the Validate PR job logs for details.

Once all issues are resolved, push a new commit to re-trigger this check.
When the pr-review validator passes, the pr-validation-passed label will be applied and the maintainers will be auto-requested for review. Note: other status checks (unit-tests, schema regression, etc.) report separately — see the full status rollup at the bottom of this PR.

github-actions · 2026-07-27T12:00:15Z

✅ Automated Check Results — All checks passed

All structural, schema, content, documentation, and secret scan checks passed.

This PR is ready for human review. The maintainers have been automatically requested.

Adds utilities/terraform/, a self-contained Terraform module that provisions a Microsoft Discovery services environment: * Virtual network with five subnets (workspaceSubnet + agentSubnet delegated to Microsoft.App/environments) * User-assigned managed identity + three role assignments (Storage Blob Data Contributor, Microsoft Discovery Platform Contributor, AcrPull) * Storage account (shared-key auth disabled, public network disabled, TLS1_2, Studio/VS Code CORS) with one AzureRM-managed blob container * Discovery Supercomputer + node pool * Discovery Workspace with one chat model deployment + project * Discovery StorageContainer bound to the AzureRM storage account Provider split: * hashicorp/azurerm ~> 4.20 for every platform primitive * Azure/azapi ~> 2.0 for all Microsoft.Discovery/* resources (pinned to @2026-02-01-preview) and for the blob container control-plane call so a data-plane RBAC grant is not required Also updates utilities/README.md inventory with the new entry.

New scripts in utilities/terraform/: - preflight.sh: deterministic pre-apply checks (RP registration, region allowlist, VM SKU allowlist, compute cores quota) with auto-sourced add-on modules under preflight-checks/ (one concern per file). - destroy.sh: idempotent teardown that pre-drains Discovery children serially, detaches AKS-orphaned NSGs, detects the unrecoverable legionservicelink SAL, and force-deletes surviving MRGs. Terraform changes: - discovery.tf: add explicit depends_on so project creation waits for the chat model deployment to reach Succeeded (RP-side ordering the reference graph does not express). - variables.tf: default location -> uksouth (matches current test region). - .gitignore: exclude bare 'tfplan' and local TESTING-PLAN.md. Docs: - utilities/terraform/README.md: Quickstart TL;DR and preflight callouts in the setup walkthrough. - README.md: trailing newline.

…oy.sh The utility now lets Azure surface the VNet/RG delete failure directly when an orphaned Microsoft.App/environments serviceAssociationLink (legionservicelink) is present. Orchestrating support-ticket workflows and maintaining a quarantined-RG tagging scheme is out of scope for this module. Removes: - --abandon-orphan-vnet CLI flag and ABANDON_ORPHAN_VNET variable. - Step 1.7 (detect_orphaned_sals, print_orphan_sal_guidance) and its ~170 lines of support-ticket template. - SKIP_RG_DELETE branch in step 2. - References to stuck-SAL deadlocks in the header comment block. Kept: - Step 1.5 serial pre-drain of workspaces/supercomputers (still prevents the race that creates orphaned SALs in the first place). - Step 1.6 AKS-orphaned NSG detachment (self-service, unrelated concern).

Bring the Terraform utility up to parity with uami.bicep through the search subnet: - Set NetworkIsolation, discovery.workbench.enableGhcpAiFeatures, and discovery.workbench.enableExtensions tags on the workspace (was version=v2 only). Fixes the broken-hybrid posture where subnet IDs were passed without NetworkIsolation=true, which disabled Cosmos public access with no private endpoint and deadlocked teardown. - Add network_isolation, enable_ghcp_ai_features, enable_extensions variables (default true) and search_subnet_prefix. - Add delegated searchSubnet (sixth subnet, 10.0.6.0/24) to match the bicep VNet. - Update README and terraform.tfvars.example for parity. Out of scope (post-search-subnet in bicep): four-identity least-privilege model and its seven role assignments remain a follow-up.

Port the per-identity least-privilege split from uami.bicep: - Add cluster, kubelet, and workload UAMIs (were all collapsed onto the single workspace identity). - Expand to seven scoped role assignments: workspace gets Discovery Platform Contributor (RG) + Storage Blob Data Contributor (storage); cluster gets Network Contributor (AKS subnet only); kubelet gets Managed Identity Operator (cluster identity) + AcrPull (RG) + Storage Blob Data Contributor (storage); workload gets Storage Blob Data Contributor (storage) only. - Wire the supercomputer clusterIdentity/kubeletIdentity/workloadIdentities to the dedicated identities and update create-ordering depends_on. - Add cluster/kubelet/workload identity name variables, locals, and outputs. - Update README (four identities, seven roles, 27-resource plan).

- Correct README plan summary: thirteen outputs (was twelve) after adding the cluster/kubelet/workload identity outputs. - Add ADR 0001 documenting the network isolation posture decision (default network_isolation=true, always provision the private topology) and link it from the README.

Patch isolationScope = "Regional" onto all four user-assigned managed identities via azapi_update_resource (azurerm 4.x does not expose the property), matching uami.bicep. Regional isolation restricts each identity to source resources in its own region, shrinking the blast radius on compromise and containing identity-plane failures to one region; all source resources here live in var.location. Update README plan summary (31 resources) and the identity section.

…twork - add existing_workspace_id and existing_bookshelf_id (skip their modules) - gate platform supercomputer network behind create_supercomputer_network - root skips the SC network when existing_supercomputer_id is set - resolve created-or-existing IDs for all three; document BYO knobs 🧩 - Generated by Copilot

github-actions · 2026-08-10T12:18:21Z

✅ Automated Check Results — All checks passed

All structural, schema, content, documentation, and secret scan checks passed.

This PR is ready for human review. The maintainers have been automatically requested.

- key chat_model_deployments and projects by a stable logical key - derive the Azure name from an optional name value (defaults to key) - fixes "Invalid for_each argument" on first plan when names derive from the random suffix 🐛 - Generated by Copilot

github-actions · 2026-08-11T11:55:56Z

✅ Automated Check Results — All checks passed

All structural, schema, content, documentation, and secret scan checks passed.

This PR is ready for human review. The maintainers have been automatically requested.

…elf_id - Drop eastus2 from location validation allowlist so terraform validate fails fast instead of misleading users into a ~30m apply failure; mirrors preflight KNOWN_BAD_REGIONS and 06-approved-regions.sh (RI-2) - Point providers.tf comment at the real ADR docs/adr/0001-network-isolation-posture.md (RI-1) - Explain why bookshelf_id uses a ternary instead of coalesce (RI-3) - Sync README region references with the tightened allowlist

github-actions · 2026-08-13T12:56:14Z

✅ Automated Check Results — All checks passed

All structural, schema, content, documentation, and secret scan checks passed.

This PR is ready for human review. The maintainers have been automatically requested.

github-actions · 2026-08-14T10:24:48Z

✅ Automated Check Results — All checks passed

All structural, schema, content, documentation, and secret scan checks passed.

This PR is ready for human review. The maintainers have been automatically requested.

Teardown is out of scope for this deployment utility; deletion is handled by the root delete-discovery-deployment utility.

github-actions · 2026-08-14T10:45:43Z

✅ Automated Check Results — All checks passed

All structural, schema, content, documentation, and secret scan checks passed.

This PR is ready for human review. The maintainers have been automatically requested.

Terri Mod (Modrats) · 2026-08-14T13:19:50Z

@microsoft-github-policy-service agree company="Microsoft"

github-actions · 2026-08-14T13:21:09Z

✅ Automated Check Results — All checks passed

All structural, schema, content, documentation, and secret scan checks passed.

This PR is ready for human review. The maintainers have been automatically requested.

github-actions · 2026-08-27T08:35:46Z

✅ Automated Check Results — All checks passed

All structural, schema, content, documentation, and secret scan checks passed.

This PR is ready for human review. The maintainers have been automatically requested.

Thomas Conté (tomconte) · 2026-09-04T08:25:18Z

+
+resource "azurerm_private_endpoint" "blob" {
+  name                = "pe-blob-${local.storage_account_name}"
+  location            = data.azurerm_resource_group.rg.location


Private endpoint region must follow its subnet's VNet, not the resource group

location resolves to the resource group's region, but subnet_id points at azurerm_subnet.private_endpoint, which lives in azurerm_virtual_network.workspace — created at line 44 with var.location. When the RG's region differs from var.location, apply fails:

Error: creating Private Endpoint (... Private Endpoint Name: "pe-blob-XXXX"): unexpected status 400 (400 Bad Request) with error: InvalidResourceReference: Resource .../virtualNetworks/VNET-XXXX referenced by resource .../privateEndpoints/pe-blob-XXXX was not found. Please make sure that the referenced resource exists, and that both resources are in the same region.

The message is misleading — the VNet does exist. The real constraint is the region mismatch. Reproduced with an RG in uksouth and location = "swedencentral"; every other resource applied cleanly and this one failed on its own.

Suggested change

location = data.azurerm_resource_group.rg.location

location = azurerm_virtual_network.workspace.location

Deriving from the VNet keeps this correct regardless of how var.location is later set.

Thomas Conté (tomconte) · 2026-09-04T08:25:18Z

+# -----------------------------------------------------------------------------
+resource "azurerm_storage_account" "outputs" {
+  name                = local.storage_account_name
+  location            = data.azurerm_resource_group.rg.location


Storage account lands in the RG's region instead of var.location

Every other regional resource in this module uses var.location (lines 44, 149, 155, 161, 167). This line and the private endpoint at 230 are the only two using the resource group's location. When they differ, the storage account is provisioned away from the workspace VNet it serves — cross-region latency and egress on all blob output traffic.

Suggested change

location = data.azurerm_resource_group.rg.location

location = var.location

Heads-up for anyone who has already applied: location forces replacement on azurerm_storage_account, so existing deployments need a data migration rather than a straight apply. Might be worth calling out in the README.

Terri Mod (Modrats) requested review from Ali Zaman (anzaman), charwen, karthik ananthakrishnan (karthikananth), leijgao, Mukesh Dua (mukesh-dua) and Suraj Mavilakandy (surajmb) as code owners July 14, 2026 07:55

github-actions Bot added update-branch PR branch is behind base — please update needs-human-review Awaiting human approval labels Jul 14, 2026

Terri Mod (Modrats) marked this pull request as draft July 14, 2026 07:56

github-actions Bot added the pr-validation-passed The pr-review workflow's validator passed. Other status checks report separately. label Jul 14, 2026

github-actions Bot added pr-validation-passed The pr-review workflow's validator passed. Other status checks report separately. and removed pr-validation-passed The pr-review workflow's validator passed. Other status checks report separately. labels Jul 14, 2026

github-actions Bot added pr-validation-failed The pr-review workflow's validator failed; see inline comments. and removed pr-validation-passed The pr-review workflow's validator passed. Other status checks report separately. labels Jul 21, 2026

github-actions Bot added pr-validation-failed The pr-review workflow's validator failed; see inline comments. and removed pr-validation-failed The pr-review workflow's validator failed; see inline comments. labels Jul 21, 2026

github-actions Bot added pr-validation-passed The pr-review workflow's validator passed. Other status checks report separately. and removed pr-validation-failed The pr-review workflow's validator failed; see inline comments. labels Jul 27, 2026

Terri Mod (Modrats) added 7 commits July 31, 2026 11:23

github-actions Bot added pr-validation-passed The pr-review workflow's validator passed. Other status checks report separately. and removed pr-validation-passed The pr-review workflow's validator passed. Other status checks report separately. labels Aug 10, 2026

github-actions Bot added pr-validation-passed The pr-review workflow's validator passed. Other status checks report separately. and removed pr-validation-passed The pr-review workflow's validator passed. Other status checks report separately. labels Aug 11, 2026

github-actions Bot added pr-validation-passed The pr-review workflow's validator passed. Other status checks report separately. and removed pr-validation-passed The pr-review workflow's validator passed. Other status checks report separately. labels Aug 13, 2026

update readme

752df18

github-actions Bot added pr-validation-passed The pr-review workflow's validator passed. Other status checks report separately. and removed pr-validation-passed The pr-review workflow's validator passed. Other status checks report separately. labels Aug 14, 2026

chore(terraform): remove destroy.sh from the utility

98bf04b

Teardown is out of scope for this deployment utility; deletion is handled by the root delete-discovery-deployment utility.

github-actions Bot added pr-validation-passed The pr-review workflow's validator passed. Other status checks report separately. and removed pr-validation-passed The pr-review workflow's validator passed. Other status checks report separately. labels Aug 14, 2026

Merge branch 'main' into add-terraform-utility

3d0d99c

github-actions Bot added pr-validation-passed The pr-review workflow's validator passed. Other status checks report separately. and removed pr-validation-passed The pr-review workflow's validator passed. Other status checks report separately. labels Aug 14, 2026

Bicep parity (except uamis)

507b84f

github-actions Bot added pr-validation-passed The pr-review workflow's validator passed. Other status checks report separately. and removed pr-validation-passed The pr-review workflow's validator passed. Other status checks report separately. labels Aug 27, 2026

Thomas Conté (tomconte) reviewed Sep 4, 2026

View reviewed changes

	location = data.azurerm_resource_group.rg.location
	location = azurerm_virtual_network.workspace.location

	location = data.azurerm_resource_group.rg.location
	location = var.location

Uh oh!

Conversation

Terri Mod (Modrats) commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Approach

Type of change

Related issue / tracking

Schema impact

Validation checklist

Reviewer notes

Uh oh!

github-actions Bot commented Jul 14, 2026

✅ Automated Check Results — All checks passed

Uh oh!

github-actions Bot commented Jul 14, 2026

✅ Automated Check Results — All checks passed

Uh oh!

github-actions Bot commented Jul 21, 2026

❌ Automated Check Results — 2 issue(s) found

Uh oh!

github-actions Bot commented Jul 21, 2026

❌ Automated Check Results — 2 issue(s) found

Uh oh!

github-actions Bot commented Jul 27, 2026

✅ Automated Check Results — All checks passed

Uh oh!

github-actions Bot commented Aug 10, 2026

✅ Automated Check Results — All checks passed

Uh oh!

github-actions Bot commented Aug 11, 2026

✅ Automated Check Results — All checks passed

Uh oh!

github-actions Bot commented Aug 13, 2026

✅ Automated Check Results — All checks passed

Uh oh!

github-actions Bot commented Aug 14, 2026

✅ Automated Check Results — All checks passed

Uh oh!

github-actions Bot commented Aug 14, 2026

✅ Automated Check Results — All checks passed

Uh oh!

Terri Mod (Modrats) commented Aug 14, 2026

Uh oh!

github-actions Bot commented Aug 14, 2026

✅ Automated Check Results — All checks passed

Uh oh!

github-actions Bot commented Aug 27, 2026

✅ Automated Check Results — All checks passed

Uh oh!

Thomas Conté (tomconte) Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

Thomas Conté (tomconte) Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Terri Mod (Modrats) commented Jul 14, 2026 •

edited

Loading