fix(hub): govern runner capabilities so Cursor reopen soft-fails on skew

[heavygee]

Summary

After #1037, Cursor reopen hard-failed when the hub was newer than a remote runner: missing cursor-chat-store-status was treated as deleted chat data even when store.db still existed.

This PR makes hub↔runner generation governance explicit:

• Soft-fail Cursor reopen when the chat-store probe errors (missing handler / skew / unknown). Definitive onDisk: false still blocks.
• Runners advertise version + capability set on machine metadata; hub declares required capabilities (starting with Cursor chat-store probe).
• Unmissable persistent web banner on session/machines UI listing upgrade/restart steps for skewed hosts.
• Ensure v1: if a newer CLI binary is already on disk (installedCliMtimeMs ≠ startedCliMtimeMs), hub may call stop-runner so systemd/handoff loads the new generation; otherwise the banner stays.
• Multi-machine install note pointing operators at the banner.
• Machine selector shows CLI version + UPDATE REQUIRED when skewed.

Test plan

• bun typecheck
• bun run test (shared capability registry, soft-fail reopen, runner ensure, web reopen gate, skew banner)
• Peer hub: live banner copy verified (Runner out of date on proxmox + upgrade/restart instructions)
• Operator dogfood: upgrade hub first, leave remote runner old, confirm reopen is allowed with honest hint + banner; upgrade remote CLI and confirm banner clears

Issues

Fixes #1084

[github-actions]

Findings

• [Major] Refresh full runner capability metadata for existing machines — /cli/machines returns an existing machine row unchanged (hub/src/store/machines.ts:44), so the new buildMachineMetadata({ ... }) payload passed by a restarted runner (cli/src/runner/run.ts:773) is ignored for every already-known machine. The new heartbeat refresh only writes installedCliMtimeMs (cli/src/api/apiMachine.ts:556), leaving old/missing capabilities and stale startedCliMtimeMs in place. Result: an upgraded runner can keep looking skewed forever; with stale mtime drift retained, the hub can repeatedly call stop-runner against the fresh runner (hub/src/sync/syncEngine.ts:474).
  Suggested fix:

  const installedCliMtimeMs = getInstalledCliMtimeMs()
  const nextCapabilities = [...CURRENT_MACHINE_CAPABILITIES]
  const needsCapabilityRefresh = !arraysEqual(
      this.machine.metadata?.capabilities ?? [],
      nextCapabilities,
  )
  const needsMtimeRefresh = typeof installedCliMtimeMs === 'number'
      && this.machine.metadata?.installedCliMtimeMs !== installedCliMtimeMs
  
  if (this.machine.metadata && (needsCapabilityRefresh || needsMtimeRefresh)) {
      void this.updateMachineMetadata((current) => ({
          ...(current ?? this.machine.metadata!),
          capabilities: nextCapabilities,
          ...(typeof installedCliMtimeMs === 'number' ? { installedCliMtimeMs } : {}),
          // Also refresh startedCliMtimeMs from the runner start snapshot, not the old DB row.
      }))
  }

• [Major] Preserve the runner handoff opt-out before hub-initiated stops — the existing runner explicitly skips mtime/version restarts when HAPI_DISABLE_VERSION_HANDOFF=1 is set (cli/src/runner/run.ts:893) and persists that opt-out for other callers (cli/src/runner/run.ts:731). The new hub ensure path does not receive or check that state, so once installedCliMtimeMs !== startedCliMtimeMs, it can still send stop-runner (hub/src/sync/syncEngine.ts:474) and force a clean shutdown that the operator opted out of.
  Suggested fix:

  // MachineMetadataSchema/buildMachineMetadata
  versionHandoffDisabled: process.env.HAPI_DISABLE_VERSION_HANDOFF === '1'
  
  // maybeEnsureRunnerGeneration
  if (machine.metadata.versionHandoffDisabled === true) {
      return
  }
  await this.rpcGateway.stopRunner(machineId)

Questions

• None.

Summary

• Review mode: initial
• Two Major issues found in runner generation governance. The existing tests cover skew detection and the stop trigger, but they do not cover an already-existing machine row being refreshed by a new runner, or a runner with version handoff disabled.

Testing

• Not run (automation; review only).

HAPI Bot

[heavygee]

Holding merge readiness pending insight from the operator's tooling/meta-bot session (dogfood-before-upstream order). Will follow up after that context lands — do not treat this as ready for review yet.

[github-actions]

Findings

• [Major] Build hub-artifacts for the target runner, not the hub host — upgradeMachineRunner() prepares hub-artifact offers with process.platform/process.arch, so a source hub on linux/x64 will send a linux/x64 binary to a connected darwin/arm64 or win32 runner. The CLI then pins its download query to offer.artifact.platform/arch, so the remote can install an incompatible executable and stay skewed. Evidence hub/src/sync/syncEngine.ts:562.
  Suggested fix:

  // MachineMetadataSchema / buildMachineMetadata
  arch: process.arch,
  
  // upgradeMachineRunner
  const platform = machine.metadata.platform
  const arch = machine.metadata.arch
  if (!platform || !arch) {
      return { type: 'error', message: 'Machine platform/arch unavailable', code: 'upgrade_unavailable' }
  }
  offer = await this.prepareArtifactOffer(offer, platform, arch)

• [Major] Relaunch the verified artifact executable directly — scheduleRunnerRelaunch() puts HAPI_CLI_EXECUTABLE only in options.env, but spawnHappyCLI() resolves the command from the current process env before merging options.env and, in compiled mode, writes the old command back into the child env. A verified hub-artifact can report started while relaunching the old binary. Evidence cli/src/upgrade/selfUpgrade.ts:146.
  Suggested fix:

  const env = { ...process.env, ...(cliExecutable ? { HAPI_CLI_EXECUTABLE: cliExecutable } : {}) }
  const child = cliExecutable
      ? spawn(cliExecutable, ['runner', 'start-sync'], { detached: true, stdio: 'ignore', env })
      : spawnHappyCLI(['runner', 'start-sync'], { detached: true, stdio: 'ignore', env })
  child.unref()

• [Major] Serialize artifact builds before mutating embeddedAssets.generated.ts — every first-time artifact build uses the same .fleet-upgrade.bak path while replacing a source file. Two concurrent auto/manual upgrade requests can make the second build back up the first build's stub over the real manifest, leaving the working tree with the empty embedded-assets file after restore. Evidence hub/src/upgrade/cliArtifact.ts:151.
  Suggested fix:

  let artifactBuildLock: Promise<void> = Promise.resolve()
  
  async function withArtifactBuildLock<T>(run: () => Promise<T>): Promise<T> {
      const previous = artifactBuildLock
      let release!: () => void
      artifactBuildLock = new Promise<void>((resolve) => { release = resolve })
      await previous
      try {
          return await run()
      } finally {
          release()
      }
  }
  
  return await withArtifactBuildLock(async () => {
      // re-check cached artifact, then call withStubEmbeddedAssets/build/writeMeta
  })

Questions

• None.

Summary

• Review mode: follow-up after new commits
• Three Major issues remain in the new hub-artifact fleet-upgrade path. The previous bot findings around metadata refresh and handoff opt-out appear addressed. Residual test gaps: no heterogeneous runner platform/arch case, no assertion that artifact relaunch uses the downloaded binary, and no concurrent artifact build coverage.

Testing

• Not run (automation; review only).

HAPI Bot

[heavygee]

Babysit update @ ead199947:

Addressed (threads resolved):

1. hub-artifact prepared for runner platform/arch (not hub host)
2. artifact relaunch spawn()s downloaded binary directly
3. artifact compile serialized + unique stub backups

CI: test green. pr-review failing on Codex Cloud 503 (供应商暂时不可用) — no new review posted against this tip; unresolved threads 0.

Will re-poll when the review action recovers; not holding further code changes for that infra flake.

[github-actions]

Findings

• [Major] Validate artifact identifiers before using them in paths — /cli/upgrade/cli-artifact accepts query version/platform/arch, then artifactFileName() joins those raw values into dataDir/upgrade-artifacts. Because decoded query strings can contain /../, any holder of a CLI/namespace token can make the hub compile/write the artifact and .json metadata outside the artifact directory, or build the current source under arbitrary fake versions. Evidence hub/src/upgrade/cliArtifact.ts:43, hub/src/web/routes/upgrade.ts:65.
  Suggested fix:

  const ARTIFACT_TOKEN_RE = /^[A-Za-z0-9._+-]+$/
  
  function artifactToken(name: string, value: string): string {
      if (!ARTIFACT_TOKEN_RE.test(value)) {
          throw new Error(`Invalid artifact ${name}`)
      }
      return value
  }
  
  export function artifactFileName(version: string, platform: string, arch: string): string {
      return `hapi-${artifactToken('version', version)}-${artifactToken('platform', platform)}-${artifactToken('arch', arch)}`
  }
  
  if (targetVersion !== baseOffer.targetVersion) {
      return c.json({ error: 'Unsupported artifact version' }, 400)
  }

Questions

• None.

Summary

• Review mode: follow-up after new commits
• One Major issue remains in the new hub-artifact download/build path. The prior bot findings around target arch, relaunching the downloaded binary, and concurrent artifact builds appear addressed in the follow-up commit. Residual risk: artifact route behavior is reviewed statically only.

Testing

• Not run (automation; review only).

HAPI Bot

[heavygee]

Superseded by same-repo mirror #1108 (tiann:fix/hub-runner-version-governance) so HAPI Bot can review while fork PR checkouts are blocked (see #1107 / GitHub 2026-07-20 checkout backport).

Commits and file set are identical; closing this fork-head PR to avoid duplicate review noise.

[heavygee]

Closed in favor of same-repo #1108.