You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
1. Update B10 β add PR #6342/6356 as an additional root cause and fix
The current B10 entry attributes the ~30 s per-directory timeout in fixArtifactPermissionsForRootless() solely to compound tag@digest image refs causing registry manifest verification (fixed by PR #6025). PRs #6342 and #6356 reveal a second, independent root cause: even after the image ref was corrected, docker run was invoked without --entrypoint sh, so AWF's own entrypoint.sh ran instead of the chown/chmod repair command. entrypoint.sh waits up to 30 s for the iptables-init container (which never starts in the repair context), then exits non-zero β producing the same ~30 s Γ 3 observable.
Proposed addition to the B10 "Fix" cell (append after the existing PR #6025 paragraph):
Additional fix (PRs #6342 / #6356, merged 2026-07-18): Even with a tag-only image ref, docker run was missing --entrypoint sh, causing AWF's entrypoint.sh to execute in place of the repair command. entrypoint.sh waits ~30 s for an iptables-init container that never starts in this context, then fails. The fix adds --entrypoint sh and passes the chown/chmod command via -c; stdout is now captured alongside stderr so timeout messages surfaced on stdout are not silently swallowed.
Proposed addition to the B10 "Citations" cell: github/gh-aw-firewall#6342, github/gh-aw-firewall#6356
2. New row B13 β topology MCP peers and difcProxyHost denied through Squid in network-isolation mode
Field
Value
ID
B13
Signal
In --network-isolation mode, MCP tool calls to topology-attached peers (e.g. awmg-mcpg) or to a difcProxyHost configured independently of topologyAttach return connection errors or are logged as TCP_DENIED in Squid's access log; the agent can reach the DIFC proxy host during startup (B5/B12) but individual in-session MCP/HTTP calls to those same hosts fail or produce Squid 403 noise
Root cause
buildProxyEnvironment() assembled NO_PROXY from a static list of AWF-internal hosts but did not include (a) config.topologyAttach peer hostnames or (b) config.difcProxyHost. Proxy-aware clients (Node undici, curl) honoured HTTPS_PROXY for those hosts; Squid denied them because they are not in the domain allowlist
Fix / flag
Fixed in AWF (PR #6189, merged 2026-07-20):config.topologyAttach peer hostnames are now added to NO_PROXY/no_proxy. Fixed in AWF (PR #6438, merged 2026-07-20):config.difcProxyHost (stripped of any :port suffix) is also added to NO_PROXY. Upgrade AWF to version including both PRs.
Probe
Inspect Squid access.log for TCP_DENIED entries whose target host is a topology peer or the DIFC proxy host; inside the agent container check echo $NO_PROXY β the topology peer and DIFC proxy hostname should appear
3. New row B14 β Copilot CLI silent crash (exit 1, ~0.5 s, no output) after ~/.copilot/config.json is masked as a credential
Field
Value
ID
B14
Signal
After upgrading to AWF including the centralized mount-policy refactor (PR #6339), the Copilot CLI agent exits immediately (exit code 1, ~0.5 s, zero stdout/stderr); no network calls are made; the issue appears on Docker and gVisor runtimes but not sbx (which moves files aside rather than overlay-masking them)
Root cause
PR #6339 added ~/.copilot/config.json to the credential deny list with reason "Copilot CLI persisted auth token". This is incorrect: config.json contains only experiment flags, first-launch timestamps, and plugin metadata β no token. AWF masks denied files with a read-only/dev/null bind-mount overlay. The Copilot CLI reads and atomically rewritesconfig.json via temp-file + rename at startup; a rename cannot target a bind-mounted path and the overlay is read-only, so the CLI crashes silently.
Fix / flag
Fixed in AWF (PR #6374, merged 2026-07-18):~/.copilot/config.json is removed from the credential deny list; the file is bind-mounted read-write as part of the .copilot tool subdir (pre-#6339 behaviour). Upgrade to the AWF version including #6374. Workaround (older AWF): pass --keep-containers and inspect the agent container for EROFS or rename errors; downgrade AWF past #6339 if unpatched.
Probe
awf ... copilot agent --version β if it exits 1 with no output in ~0.5 s, confirm AWF version includes #6374; inspect agent container logs for EROFS/rename errors on ~/.copilot/config.json
4. New row D10 β Rootless Copilot CLI (~/.local/bin/copilot) not found in sbx microVM PATH
Field
Value
ID
D10
Signal
On --container-runtime sbx, Copilot CLI installed via install_copilot_cli.sh --rootless (redirects install to ~/.local/bin) is not found even though ~/.local is mounted into the microVM; copilot: command not found or ENOENT at agent startup
Root cause
sbx microVMs do not run AWF's entrypoint.sh (which prepends $HOME/.local/bin to PATH). buildCoreEnvironment() injected only /usr/local/sbin:...:/bin as PATH. The binary is present in the VM but unreachable by name. Docker/gVisor compose runtimes were unaffected because entrypoint.sh fixed PATH for them.
Fix / flag
Fixed in AWF (PR #6407, merged 2026-07-19):buildCoreEnvironment() now prepends ${HOME}/.local/bin to the injected PATH, fixing sbx without affecting compose runtimes. Upgrade to the AWF version including #6407. Workaround (older AWF): use the non-rootless install (requires sudo on the runner) or pass --env PATH=/home/runner/.local/bin:$PATH explicitly.
Probe
Inside the sbx agent: echo $PATH β ~/.local/bin should appear first; which copilot or ls ~/.local/bin/copilot confirms binary presence vs PATH visibility
Copilot CLI exits immediately (exit 1, ~0.5 s, zero stdout/stderr) after AWF upgrade; works in sbx but not Docker/gVisor
B14 (.copilot/config.json masked as credential; fixed in #6374)
copilot: command not found inside sbx microVM when binary is at ~/.local/bin/copilot
D10 (~/.local/bin absent from sbx injected PATH; fixed in #6407)
TCP_DENIED in Squid access log for topology peer or DIFC proxy host during agent run; MCP tool calls silently fail or return connection errors in network-isolation mode
B13 (topology peers and difcProxyHost missing from NO_PROXY; fixed in #6189 and #6438)
Apply the same B10 update, B13/B14/D10 new rows, and three new error-string lookup entries as described above. The playbook ordering (Category A β B β C β D) remains unchanged; B13/B14 are appended after B12, D10 after D9.
Proposed portable agent changes
File: .github/agents/self-hosted-runner-doctor.md
Apply identical changes to the embedded failure-mode catalog and error-string lookup table in this file so the portable agent stays in sync:
Summary
Proposed knowledge-base changes
File:
.github/workflows/shared/self-hosted-failure-modes.md1. Update B10 β add PR #6342/6356 as an additional root cause and fix
The current B10 entry attributes the ~30 s per-directory timeout in
fixArtifactPermissionsForRootless()solely to compoundtag@digestimage refs causing registry manifest verification (fixed by PR #6025). PRs #6342 and #6356 reveal a second, independent root cause: even after the image ref was corrected,docker runwas invoked without--entrypoint sh, so AWF's ownentrypoint.shran instead of thechown/chmodrepair command.entrypoint.shwaits up to 30 s for the iptables-init container (which never starts in the repair context), then exits non-zero β producing the same ~30 s Γ 3 observable.Proposed addition to the B10 "Fix" cell (append after the existing PR #6025 paragraph):
Proposed addition to the B10 "Citations" cell:
github/gh-aw-firewall#6342, github/gh-aw-firewall#63562. New row B13 β topology MCP peers and
difcProxyHostdenied through Squid in network-isolation mode--network-isolationmode, MCP tool calls to topology-attached peers (e.g.awmg-mcpg) or to adifcProxyHostconfigured independently oftopologyAttachreturn connection errors or are logged asTCP_DENIEDin Squid's access log; the agent can reach the DIFC proxy host during startup (B5/B12) but individual in-session MCP/HTTP calls to those same hosts fail or produce Squid 403 noisebuildProxyEnvironment()assembledNO_PROXYfrom a static list of AWF-internal hosts but did not include (a)config.topologyAttachpeer hostnames or (b)config.difcProxyHost. Proxy-aware clients (Nodeundici, curl) honouredHTTPS_PROXYfor those hosts; Squid denied them because they are not in the domain allowlistconfig.topologyAttachpeer hostnames are now added toNO_PROXY/no_proxy. Fixed in AWF (PR #6438, merged 2026-07-20):config.difcProxyHost(stripped of any:portsuffix) is also added toNO_PROXY. Upgrade AWF to version including both PRs.access.logforTCP_DENIEDentries whose target host is a topology peer or the DIFC proxy host; inside the agent container checkecho $NO_PROXYβ the topology peer and DIFC proxy hostname should appear3. New row B14 β Copilot CLI silent crash (exit 1, ~0.5 s, no output) after
~/.copilot/config.jsonis masked as a credential~/.copilot/config.jsonto the credential deny list with reason "Copilot CLI persisted auth token". This is incorrect:config.jsoncontains only experiment flags, first-launch timestamps, and plugin metadata β no token. AWF masks denied files with a read-only/dev/nullbind-mount overlay. The Copilot CLI reads and atomically rewritesconfig.jsonvia temp-file + rename at startup; a rename cannot target a bind-mounted path and the overlay is read-only, so the CLI crashes silently.~/.copilot/config.jsonis removed from the credential deny list; the file is bind-mounted read-write as part of the.copilottool subdir (pre-#6339 behaviour). Upgrade to the AWF version including #6374. Workaround (older AWF): pass--keep-containersand inspect the agent container forEROFSor rename errors; downgrade AWF past #6339 if unpatched.awf ... copilot agent --versionβ if it exits 1 with no output in ~0.5 s, confirm AWF version includes #6374; inspect agent container logs forEROFS/renameerrors on~/.copilot/config.json4. New row D10 β Rootless Copilot CLI (
~/.local/bin/copilot) not found in sbx microVM PATH--container-runtime sbx, Copilot CLI installed viainstall_copilot_cli.sh --rootless(redirects install to~/.local/bin) is not found even though~/.localis mounted into the microVM;copilot: command not foundorENOENTat agent startupentrypoint.sh(which prepends$HOME/.local/binto PATH).buildCoreEnvironment()injected only/usr/local/sbin:...:/binasPATH. The binary is present in the VM but unreachable by name. Docker/gVisor compose runtimes were unaffected becauseentrypoint.shfixed PATH for them.buildCoreEnvironment()now prepends${HOME}/.local/binto the injected PATH, fixing sbx without affecting compose runtimes. Upgrade to the AWF version including #6407. Workaround (older AWF): use the non-rootless install (requires sudo on the runner) or pass--env PATH=/home/runner/.local/bin:$PATHexplicitly.echo $PATHβ~/.local/binshould appear first;which copilotorls ~/.local/bin/copilotconfirms binary presence vs PATH visibilityError-string quick-lookup additions (same file)
Append to the lookup table:
.copilot/config.jsonmasked as credential; fixed in #6374)copilot: command not foundinside sbx microVM when binary is at~/.local/bin/copilot~/.local/binabsent from sbx injected PATH; fixed in #6407)TCP_DENIEDin Squid access log for topology peer or DIFC proxy host during agent run; MCP tool calls silently fail or return connection errors in network-isolation modedifcProxyHostmissing fromNO_PROXY; fixed in #6189 and #6438)Proposed doctor changes
File:
.github/workflows/self-hosted-runner-doctor.mdApply the same B10 update, B13/B14/D10 new rows, and three new error-string lookup entries as described above. The playbook ordering (Category A β B β C β D) remains unchanged; B13/B14 are appended after B12, D10 after D9.
Proposed portable agent changes
File:
.github/agents/self-hosted-runner-doctor.mdApply identical changes to the embedded failure-mode catalog and error-string lookup table in this file so the portable agent stays in sync:
Source issues and PRs