DefaultCopilotVersion drifts past compat.json max-agent, forcing a network install on every job and disabling the toolcache path

[Evangelink]

Summary

DefaultCopilotVersion is hard-pinned into every compiled .lock.yml as an explicit argument to install_copilot_cli.sh. That explicit argument short-circuits compat-matrix resolution, which is the only code path that can match a cached Copilot CLI in the runner toolcache by range. Because DefaultCopilotVersion has drifted above the max-agent published in compat.json, the toolcache entry on GitHub-hosted runners can never match, so every agentic job downloads the CLI from github.com — twice (once in agent, once in detection).

This turns a transient github.com release-CDN blip into a hard job failure, and the exposure is permanent rather than decaying as runner images catch up.

Versions

• gh-aw: v0.83.1 (also reproduces conceptually on main, see below)
• Runner: ubuntu-latest (GitHub-hosted)
• Toolcache contents: /opt/hostedtoolcache/copilot-cli/1.0.56/x64/bin/copilot

The version mismatch

github/gh-aw-actions → .github/aw/compat.json (on main):

"agent-compat-v1": {
  "cache-ttl-days": 14,
  "copilot": [
    { "min-gh-aw": "0.72.0", "max-gh-aw": "*", "min-agent": "1.0.21", "max-agent": "1.0.56", "open": true },
    { "min-gh-aw": "0.0.1", "max-gh-aw": "0.71.99", "min-agent": "0.0.0", "max-agent": "1.0.21" }
  ]
}

github/gh-aw → pkg/constants/version_constants.go:

const DefaultCopilotVersion Version = "1.0.75"   // main; v0.83.1 shipped 1.0.73

For any gh-aw ≥ 0.72.0 the published compatible window is 1.0.21 … 1.0.56, but the compiler pins 1.0.73 / 1.0.75 — outside its own declared window. The runner toolcache ships exactly 1.0.56, i.e. precisely the max-agent that the matrix says should be preferred.

Why the pin defeats the toolcache

actions/setup/sh/install_copilot_cli.sh:

if [ -z "$VERSION" ]; then
  echo "No explicit Copilot CLI version requested. Attempting compat-driven version resolution..."
  ...   # yields COMPAT_MATCHED_MIN_AGENT / COMPAT_MATCHED_MAX_AGENT
else
  echo "Explicit Copilot CLI version argument provided (${VERSION}); skipping compat matrix resolution."
fi

if CACHED_COPILOT_BIN="$(find_cached_copilot_bin "$REQUESTED_VERSION" \
    "${COMPAT_MATCHED_MIN_AGENT}" "${COMPAT_MATCHED_MAX_AGENT}" "${COMPAT_CACHE_TTL_DAYS}")"; then

With an explicit version, find_cached_copilot_bin takes the exact-match branch and rejects everything else:

if [ -n "$requested_version_normalized" ]; then
  if [ "$candidate_version_normalized" = "$requested_version_normalized" ]; then
    echo "  Exact version match found: ${candidate}" >&2
    printf '%s\n' "$candidate"; return 0
  fi
  echo "  Skipping candidate (version mismatch: want ${requested_version_normalized}, got ${candidate_version_normalized})" >&2
  continue
fi

The min_version / max_version range comparisons below it are unreachable whenever a version is passed — and pkg/workflow/copilot_installer.go always passes one:

if version == "" {
    version = string(constants.DefaultCopilotVersion)
}

So the compat/range path is effectively dead code for every compiled workflow.

Observed on every run (including green ones)

Installing GitHub Copilot CLI version 1.0.73 (os: Linux, arch: x86_64)...
Explicit Copilot CLI version argument provided (1.0.73); skipping compat matrix resolution.
Searching toolcache for GitHub Copilot CLI (requested: 1.0.73, arch: x64, range: none..none)...
  Scanning toolcache root: /opt/hostedtoolcache/copilot-cli
  Found candidate: /opt/hostedtoolcache/copilot-cli/1.0.56/x64/bin/copilot (version: 1.0.56, arch: x64)
  Skipping candidate (version mismatch: want 1.0.73, got 1.0.56)
  No compatible toolcache entry found
Downloading checksums from https://github.com/github/copilot-cli/releases/download/v1.0.73/SHA256SUMS.txt...

Note range: none..none — the range that would have accepted 1.0.56 was never populated.

Resulting failure

curl -fsSL --retry 3 --retry-delay 5 gives 4 attempts over roughly 26 s. A ~30 s 504 window on the release CDN is enough to fail the whole job:

Downloading checksums from https://github.com/github/copilot-cli/releases/download/v1.0.73/SHA256SUMS.txt...
curl: (22) The requested URL returned error: 504
curl: (22) The requested URL returned error: 504
curl: (22) The requested URL returned error: 504
curl: (22) The requested URL returned error: 504
##[error]Process completed with exit code 22.

When this lands in the detection job the blast radius is larger than expected: the job dies at install, before the model runs, so parse_threat_detection_results.cjs reports ERR_PARSE: No THREAT_DETECTION_RESULT found and safe_outputs is skipped entirely. safe-outputs.threat-detection.continue-on-error: true does not help, because it only covers a detection run that completed and failed to parse — the workflow silently produces no output instead of degrading gracefully.

We have seen this recur across two separate [aw] Detection Runs trackers in microsoft/testfx.

Reproduction

1. Compile any Copilot-engine workflow with gh-aw v0.83.1.
2. Run it on ubuntu-latest.
3. Observe the Install GitHub Copilot CLI step log: the cached 1.0.56 is rejected and a download always occurs.

engine.version is documented but ignored (separate bug)

The engines reference documents pinning:

engine:
  id: copilot
  version: "0.0.422"

For the Copilot engine this is a silent no-op. pkg/workflow/copilot_engine_installation.go:

// Copilot CLI is pinned to the default version constant.
copilotVersion := string(constants.DefaultCopilotVersion)
if workflowData.EngineConfig != nil {
    if workflowData.EngineConfig.Version != "" {
        copilotInstallLog.Printf("Ignoring pinned engine.version (%s): Copilot CLI install version is pinned to %s",
            workflowData.EngineConfig.Version, copilotVersion)
    }
    workflowData.EngineConfig.Version = copilotVersion
}

Verified empirically: setting version: "1.0.56" and running gh aw compile --strict reports 0 errors and 0 warnings, yet the generated lock still contains install_copilot_cli.sh 1.0.73. The "Ignoring pinned engine.version" message only surfaces at debug log level, so users get no signal that their pin was dropped.

This also means consumers have no supported way to work around the issue above from their own repo.

Suggested fixes

1. Keep DefaultCopilotVersion within the compat.json window, or drive it from the matrix, so the pinned version and the runner toolcache can converge. A CI check asserting min-agent ≤ DefaultCopilotVersion ≤ max-agent would prevent the drift from recurring.
2. Let the toolcache satisfy an explicit pin via the compat range. When compat resolution is available, pass min-agent/max-agent to find_cached_copilot_bin even for an explicit version, so a cached in-window binary is preferred over a network download. Alternatively resolve compat first and only treat the pin as a hard floor.
3. Surface the ignored engine.version as a compile-time warning (it is currently debug-only), and fix the engines reference so it does not advertise a knob the Copilot engine drops.
4. Harden the download — raise --retry / --retry-delay and add --retry-all-errors for the checksum and tarball fetches, which are the only two hard network dependencies in the step.

Item 1 or 2 alone would remove the network hop from the common path entirely and make this class of failure disappear rather than merely become rarer.