[elliot-barn]

Problem

rayapp test can only ever test a template's AWS compute config, on the org's default cloud:

• setupEmptyWorkspace hardcodes c.template.ComputeConfig["AWS"], so the GCP and K8S entries in BUILD.yaml are never exercised.
• No template config sets a cloud: key, so the compute config is created in anyscale cloud get-default (rayapp injects it for the legacy format; the SDK falls back to it for the new format), and workspace_v2 create --compute-config inherits that cloud.

rayapp probe has the same limit: it sends the default cloud's ID to from_template, and the backend picks the entry matching that cloud. So testing a template on GCP or a Kubernetes cloud means changing the org default, which isn't an option on shared staging.

Change

Flag	On	Default	Effect
--cloud NAME	test, probe	CLI default cloud	Run on this Anyscale cloud
--compute-config KEY	test	AWS	Which compute_config entry to test with (GCP, K8S, …)

With neither flag set, behaviour is unchanged (covered by a test).

test with --cloud:

• The cloud is written as cloud: into a temp copy of the compute config, in both formats. A legacy file's existing cloud: is replaced, and a new-format file would otherwise land in the default cloud. The source file is not modified.
• The cloud is passed to workspace_v2 create --cloud.
• The compute config name gets a slugified cloud suffix (basic-single-node-k8s-<cloud>). Compute config names are shared across clouds, so without the suffix, a config already created from the same file in another cloud would be found by CreateComputeConfig's existence check, reused, and fail to launch.

test with --compute-config: a key the template doesn't have is an error, not a silent fallback to the workspace's default compute config, which would report success while testing the wrong thing.

probe with --cloud: the cloud is resolved with anyscale cloud get --name and its ID is sent as cloud_id. probe has no --compute-config because the backend picks the entry that matches the cloud.

GetCloud: anyscale cloud get logs Cloud not found. and exits 0 with empty stdout for an unknown name, so an empty ID is treated as not found. Otherwise a typo would surface later as an opaque launch error.

The options travel as a TestTarget{Cloud, ComputeConfig} struct through RunTemplateTest / RunAllTemplateTests / RunProbe, rather than as more positional strings.

Note: flags must come before the template name (Go flag stops at the first positional argument):

rayapp test  --cloud <gcp-cloud> --compute-config GCP <template>
rayapp test  --cloud <k8s-cloud> --compute-config K8S workspace-intro
rayapp probe --cloud <gcp-cloud> <template>

On Kubernetes clouds, a passing test is only trustworthy once #496 (detect remote command failure) lands.

Testing

• go test ./rayapp/... and go vet pass. Pre-commit/pre-push hooks pass (gofmt, golines, raycilint).
• New tests:
  • TestGetCloud: success, not found (empty stdout, exit 0), CLI failure.
  • TestCreateComputeConfig_CloudOverride: a new-format file gains cloud:; a legacy file's cloud: is replaced; the source file is untouched.
  • TestRunTemplateTestsWithTarget: GCP key and named cloud give the right file, the suffixed name, cloud: in the temp copy, and --cloud / --compute-config on workspace create.
  • TestRunTemplateTestsWithTarget_DefaultsToAWS: zero value matches today's behaviour (AWS config, no --cloud).
  • TestRunTemplateTestsWithTarget_MissingComputeConfig: an explicit missing key errors before any workspace is created.
  • TestProbe_WithCloud: default cloud, named cloud, and unknown cloud each produce the expected cloud_id or error.
• Not yet run end to end against a live non-default cloud.

🤖 Generated with Claude Code

[github-actions]

⚠️ PR Size Warning

This PR exceeds the size limits defined in .buildkite/raycilint.yaml.

• additions: 522 (max allowed: 500)

Break this into smaller PRs, or apply the skip-prcheck label to bypass.

[gemini-code-assist]

Code Review

This pull request introduces the ability to specify a target cloud and compute configuration when running template tests and probes. It adds a new GetCloud method to retrieve cloud information, updates CreateComputeConfig to support cloud overrides via temporary configuration files, and introduces a TestTarget struct to propagate these options. Additionally, command-line flags for --cloud and --compute-config have been added to the CLI. One review comment was provided regarding a potential temporary file leak in CreateComputeConfig if tmpFile.Close() fails before defer os.Remove(tmpPath) is registered, suggesting that the defer should be registered immediately after the temporary file is successfully created.

[gemini-code-assist]

The defer os.Remove(tmpPath) is registered after tmpFile.Close(). If tmpFile.Close() fails and returns an error, the function will exit early without registering the defer, resulting in a temporary file leak. To ensure the temporary file is always cleaned up, register the defer os.Remove(tmpPath) immediately after os.CreateTemp succeeds.

Suggested change

	tmpFile, err := os.CreateTemp("", "compute-config-*.yaml")
	if err != nil {
	return fmt.Errorf("failed to create temp file: %w", err)
	}
	tmpPath := tmpFile.Name()
	if err := tmpFile.Close(); err != nil {
	return fmt.Errorf("failed to close temp file: %w", err)
	}
	defer os.Remove(tmpPath)
	tmpFile, err := os.CreateTemp("", "compute-config-*.yaml")
	if err != nil {
	return fmt.Errorf("failed to create temp file: %w", err)
	}
	tmpPath := tmpFile.Name()
	defer os.Remove(tmpPath)
	if err := tmpFile.Close(); err != nil {
	return fmt.Errorf("failed to close temp file: %w", err)
	}