GKE generated AICR GB300 recipies

[rawsocket]

Summary

Recipes for GB300.

Motivation / Context

Updating the AICR repo with validate versions.

Type of Change

• New feature (non-breaking change that adds functionality)
• Refactoring (no functional changes)

Component(s) Affected

• Recipe engine / data (pkg/recipe)
• Other: kwok/

Implementation Notes

GB300 recipes generated with up to date component versions.

Testing

# Commands run (prefer `make qualify` for non-trivial changes)
make qualify

Risk Assessment

• Low — Isolated change, well-tested, easy to revert

Checklist

• Tests pass locally (make test with -race)
• Linter passes (make lint)
• I did not skip/disable tests to make CI green
• Commits are cryptographically signed (git commit -S) — GPG signing info

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

Welcome to AICR, @rawsocket! Thanks for your first pull request.

Before review, please ensure:

• All commits are signed off per the DCO
• CI checks pass (tests, lint, security scan)
• The PR description explains the why behind your changes

A maintainer will review this soon.

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds GB300 support for GKE COS inference and training. The change adds KWOK node profiles, tuning metadata, NCCL RoCE values, base recipes, Dynamo and Kubeflow overlays, and generated recipe coverage expectations.

Estimated code review effort: 4 (Complex) | ~45 minutes

Suggested labels: area/tests, area/infra, theme/recipes

Suggested reviewers: dims

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies generated GKE AICR recipes for GB300, which matches the main changes.
Description check	✅ Passed	The description explains that the pull request adds generated GB300 recipes and updates validated component versions, matching the changeset.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 3

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

recipes/components/gke-nccl-roce/values.yaml (1)

1-41: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

Register and wire gke-nccl-roce, or remove it.

recipes/components/gke-nccl-roce/values.yaml has no registry entry or componentRefs consumer. GB300 overlays explicitly bypass standalone networking installers in favor of GKE Managed RDMA.

If this component is retained, add its registry and recipe wiring. Then run make bom-docs and commit docs/user/container-images.md.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@recipes/components/gke-nccl-roce/values.yaml` around lines 1 - 41, Either
remove the unused gke-nccl-roce component, or register it in the component
registry and wire it through a recipe consumer using componentRefs. If retained,
update the generated container image documentation by running make bom-docs and
include the resulting docs/user/container-images.md changes.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@kwok/profiles/gke/a4x-maxgpu-4g-metal.yaml`:
- Around line 25-31: Update the gpu.memory value in the NVIDIA-GB300 profile
from 245760 to 285696 MiB per GPU, keeping the existing product, count, and
driver settings unchanged.

In `@recipes/overlays/gb300-gke-cos-inference-dynamo.yaml`:
- Around line 77-79: Update the Deployment.gpu-operator.version constraint in
the overlay’s constraints section from the outdated v24.6.0 floor to the
required >= v25.10.0 floor.

In `@recipes/overlays/gb300-gke-cos-inference.yaml`:
- Around line 54-64: Change the intent override in the nodewright-customizations
block from multiNodeTraining to inference so it matches this recipe’s
criteria.intent and selects the correct inference tuning profile; leave the
service, accelerator, and dependencyRefs values unchanged.

---

Outside diff comments:
In `@recipes/components/gke-nccl-roce/values.yaml`:
- Around line 1-41: Either remove the unused gke-nccl-roce component, or
register it in the component registry and wire it through a recipe consumer
using componentRefs. If retained, update the generated container image
documentation by running make bom-docs and include the resulting
docs/user/container-images.md changes.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 1cfd0cc4-f2e2-40ba-a596-fc5819707e01

📥 Commits

Reviewing files that changed from the base of the PR and between 0752ea1 and be8b984.

📒 Files selected for processing (10)

• docs/integrator/components/nodewright.md
• kwok/profiles/gke/a4x-maxgpu-4g-metal.yaml
• kwok/profiles/gke/system-n2.yaml
• pkg/recipe/testdata/coverage_golden.yaml
• pkg/tuning/compute_test.go
• recipes/components/gke-nccl-roce/values.yaml
• recipes/overlays/gb300-gke-cos-inference-dynamo.yaml
• recipes/overlays/gb300-gke-cos-inference.yaml
• recipes/overlays/gb300-gke-cos-training-kubeflow.yaml
• recipes/overlays/gb300-gke-cos-training.yaml

[github-actions]

Recipe evidence check

Other affected recipes without evidence yet: 4

These recipes are affected by this PR but carry no committed evidence pointer, so there is
nothing to verify. This is expected — evidence is hardware-gated and added over time.

• gb300-gke-cos-inference-dynamo
• gb300-gke-cos-inference
• gb300-gke-cos-training-kubeflow
• gb300-gke-cos-training

This gate is warning-only and never blocks merge. See ADR-007 for the trust model.

[mchmarny]

Welcome to AICR. The mechanical parts of adding an accelerator to this repo are easy to get wrong and you got them right. coverage_golden.yaml is correctly regenerated, pkg/tuning/compute_test.go and docs/integrator/components/nodewright.md are updated in step, and the inheritance chain (gb300-gke-cos-{training,inference} -> -kubeflow/-dynamo) matches how the b200 family is laid out. go test ./pkg/recipe/... ./pkg/tuning/... passes clean at this head.

The three blockers are all in the same category — config that ships to users and looks right but does the wrong thing — rather than anything structural. Each is a small fix.

Scope validated: all 10 changed files read at be8b984; recipe resolution run locally for the gb300 dynamo/inference/training paths; pkg/recipe + pkg/tuning tests run; apply-nodes.sh profile selection traced against this PR's own Tier 2 CI logs.

CI: Tier 1 / oke-ol-training (argocd-git) and the dependent KWOK Test Summary are red, but that is infrastructure, not your change — the log shows curl: (35) Recv failure: Connection reset by peer followed by Error: repo argo not found while adding the Argo CD Helm repo, in an OKE lane this PR does not touch. A re-run should clear it. The PR is also currently BEHIND main and will need a rebase.

Findings inline. Happy to re-review as soon as you have pushed.

[yuanchen8911]

Reviewed at be8b984 with recipe resolution run locally, the pinned tuning package pulled from ghcr.io, and image manifests verified against registry APIs.

Change requests

1. NodeWright tuning cannot work as configured: nvidia-tuning-gke:0.1.2 contains only gb200/ and h100/ profiles, and its prepare script rejects accelerator: gb300 before intent is read, failing Skyhook tuning on every GB300 node. Either ship with enabled: false or reuse the gb200 profile — maintainer question in the nodewright thread.

2. gb300-gke-cos-inference-dynamo sets Deployment.gpu-operator.version to >= v24.6.0, which replaces (not tightens) the parent's >= v25.10.0 on merge. GB300 support shipped in v25.10 — drop the constraint or set >= v25.10.0.

3. The new GKE KWOK profiles are never selected: apply-nodes.sh still maps gke/gb300 to eks/p5-h100, and this PR's Tier 2 CI logs show the GB300 lanes ran on amd64 H100 nodes. Wire them in or remove them. Also set gpu.memory to 285696 MiB (Google lists 279 GB per GPU; 245760 encodes only 240 GB).

4. The Kubeflow runtime.image/runtime.env overrides are consumed by nothing, the proposed 25.01-py3-arm64 tag does not exist on nvcr.io (404), and the manifest's hardcoded pytorch/pytorch digest is amd64-only — it cannot run on A4X Max's arm64 hosts. Parameterize the manifest image and pin a multi-arch CUDA 13 digest (NVIDIA PyTorch 25.08+).

Cleanup (config that reads as functional but is not)

• Drop gdrcopy.enabled: true (inert without an operator-managed driver; values-gke-cos.yaml documents gdrcopy as unsupported on COS), the dead driver.version pin, and driver.nvidiaDriverRoot (not a gpu-operator key).
• Drop the dynamo leaf's engine.cudaVersion/engine.arch (chart 1.2.1 does not consume them); document instead that GB300 requires the CUDA 13 runtime variants (vllm-runtime:1.2.1-cuda13 is published multi-arch).
• Correct the Managed RDMA comments: A4X Max additionally requires the asapd-lite DaemonSet, mrdma.google.com resource claims, and ComputeDomain/IMEX setup — document these as external provisioning prerequisites.
• Remove unreachable gke-nccl-roce/values.yaml (no registry entry, no consumer); add the Apache header to the four files missing it.

Branch is behind main with a merge commit — please rebase onto origin/main and squash to a single signed/DCO commit.

[mchmarny]

Still blocked on three functional issues at 2d01524:

1. Restore the Dynamo deployment constraint to Deployment.gpu-operator.version >= v25.10.0; the leaf currently replaces the parent floor with >= v24.6.0, producing a false pass. Thread
2. Point both GB300 NodeWright overrides at the supported gb200 tuning profile. Keep intent: inference for inference and intent: multiNodeTraining for training, then regenerate the tuning docs/test expectation. Package 0.1.2 has no gb300 profile and currently exits before applying tuning. Thread
3. Make the Kubeflow runtime.image and runtime.env overrides feed the emitted ClusterTrainingRuntime, and pin a valid arm64 or multi-arch image. The current manifest still hardcodes an amd64 image for these arm64 nodes. Thread

The previous KWOK profile-selection blocker is resolved by current main: label-based discovery now selects gke/system-n2.yaml and gke/a4x-maxgpu-4g-metal.yaml. The unused gke-nccl-roce values file and missing license headers remain non-blocking cleanup.

Focused local recipe/tuning/profile tests pass. Current-head CI is now running.

[rawsocket]

Will come back to it at a later time. All the changes suggested here are implemented.