Update Runner Doctor catalog for B13 Squid ACL gap and B9 RHEL/Amazon Linux CA mount fix#6485
Conversation
There was a problem hiding this comment.
Pull request overview
Updates Runner Doctor guidance for recent B9 CA-mount and B13 Squid ACL fixes.
Changes:
- Adds PR #6460 and #6473 remediation guidance.
- Synchronizes shared and portable catalogs.
- Extends catalog parity assertions.
Show a summary per file
| File | Description |
|---|---|
.github/workflows/shared/self-hosted-failure-modes.md |
Updates B9/B13 catalog entries. |
.github/agents/self-hosted-runner-doctor.md |
Mirrors catalog updates. |
scripts/ci/self-hosted-runner-doctor-workflow.test.ts |
Locks updated catalog text. |
Review details
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 3/3 changed files
- Comments generated: 3
- Review effort level: Medium
| | B11 | AWF exits with code 1 and logs show `[WARN] Rootless artifact permission repair failed ... (exit 1)` plus cleanup warnings, but no actionable stderr detail from the repair container | `fixArtifactPermissionsForRootless()` previously discarded repair-container stderr, so the warning hid the real failure context. The non-zero exit code comes from `runAgentCommand()` before cleanup; cleanup warnings do not override it. | **Improved in AWF (PR github/gh-aw-firewall#6072, merged 2026-07-10)**: stderr from the repair container is now captured in the warning, and chroot-home cleanup noise is reduced by downgrading that log from `warn` to `debug`. Treat this mode as diagnostic-opacity around a pre-existing non-zero command exit, not cleanup-driven exit-code mutation. | Compare agent-command logs with `Command completed with exit code: <n>`; on affected versions, repair warnings lack stderr context. On AWF including github/gh-aw-firewall#6072, the warning includes stderr detail for root-cause triage. | github/gh-aw-firewall#6070, github/gh-aw-firewall#6072 | | ||
| | B12 | `getaddrinfo EAI_AGAIN <awmg-cli-proxy>` or `ENOTFOUND <awmg-cli-proxy>` → `awf-cli-proxy could not connect to the external DIFC proxy` → `The agent was never invoked` on ARC/DinD, and DinD containers cannot resolve the Kubernetes Service name even after the ordering fix | ARC/DinD DNS isolation: the DinD Docker network does not forward DNS to the cluster resolver, so Kubernetes Service names such as `awmg-cli-proxy` are unreachable from AWF containers even though the service exists. | **Diagnosed in AWF (PR github/gh-aw-firewall#6328, merged 2026-07-17):** `detectDnsResolutionFailure()` scans cli-proxy logs for `EAI_AGAIN`/`ENOTFOUND`, extracts the unresolved hostname, and augments the startup error with concrete remediation. Address the DIFC proxy by IP, or configure `dockerd --dns <cluster-dns-ip>` so DinD containers can reach the cluster DNS server. | `docker run --rm alpine nslookup awmg-cli-proxy` — `NXDOMAIN`/`SERVFAIL` confirms the Kubernetes DNS is not reachable from DinD containers; compare with the augmented startup error from `detectDnsResolutionFailure()` | github/gh-aw-firewall#6326, github/gh-aw-firewall#6328 | | ||
| | B13 | In `--network-isolation` mode, MCP tool calls to topology-attached peers (e.g. `awmg-mcpg`) or to a `difcProxyHost` configured independently of `topologyAttach` return connection errors or are logged as `TCP_DENIED` in Squid's access log; the agent can reach the DIFC proxy host during startup (B5/B12) but individual in-session MCP/HTTP calls to those same hosts fail or produce Squid 403 noise | `buildProxyEnvironment()` assembled `NO_PROXY` from a static list of AWF-internal hosts but did not include (a) `config.topologyAttach` peer hostnames or (b) `config.difcProxyHost`. Proxy-aware clients (Node `undici`, curl) honoured `HTTPS_PROXY` for those hosts; Squid denied them because they are not in the domain allowlist. | **Fixed in AWF (PR github/gh-aw-firewall#6189, merged 2026-07-20):** `config.topologyAttach` peer hostnames are now added to `NO_PROXY`/`no_proxy`. **Fixed in AWF (PR github/gh-aw-firewall#6438, merged 2026-07-20):** `config.difcProxyHost` (stripped of any `:port` suffix) is also added to `NO_PROXY`. Upgrade AWF to version including both PRs. | Inspect Squid `access.log` for `TCP_DENIED` entries whose target host is a topology peer or the DIFC proxy host; inside the agent container check `echo $NO_PROXY` — the topology peer and DIFC proxy hostname should appear | github/gh-aw-firewall#6189, github/gh-aw-firewall#6438 | | ||
| | B13 | In `--network-isolation` mode, MCP tool calls to topology-attached peers (e.g. `awmg-mcpg`) or to a `difcProxyHost` configured independently of `topologyAttach` return connection errors or are logged as `TCP_DENIED` in Squid's access log; the agent can reach the DIFC proxy host during startup (B5/B12) but individual in-session MCP/HTTP calls to those same hosts fail or produce Squid 403 noise | `buildProxyEnvironment()` assembled `NO_PROXY` from a static list of AWF-internal hosts but did not include (a) `config.topologyAttach` peer hostnames or (b) `config.difcProxyHost`. Proxy-aware clients (Node `undici`, curl) honoured `HTTPS_PROXY` for those hosts; Squid denied them because they are not in the domain allowlist. | **Fixed in AWF (PR github/gh-aw-firewall#6189, merged 2026-07-20):** `config.topologyAttach` peer hostnames are now added to `NO_PROXY`/`no_proxy`. **Fixed in AWF (PR github/gh-aw-firewall#6438, merged 2026-07-20):** `config.difcProxyHost` (stripped of any `:port` suffix) is also added to `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6473, merged 2026-07-21):** Topology peer hostnames and `difcProxyHost` are also auto-added to the Squid ACL allowlist (via `resolveAllowedDomains()`) so that proxy-aware clients that honour `HTTP(S)_PROXY` but ignore `NO_PROXY` are still permitted by Squid. The gating mirrors `buildProxyEnvironment()`: only active when `networkIsolation` is true and the runtime is compose-based (`runtimeUsesComposeAgent()`); microVM backends (sbx) are excluded. Upgrade AWF to the version including github/gh-aw-firewall#6473. | Inspect Squid `access.log` for `TCP_DENIED` entries whose target host is a topology peer or the DIFC proxy host; inside the agent container check `echo $NO_PROXY` — the topology peer and DIFC proxy hostname should appear; inspect Squid `access.log` for `TCP_DENIED` entries for topology peer or DIFC proxy host even after upgrading to AWF with github/gh-aw-firewall#6189/#6438 — present on AWF missing github/gh-aw-firewall#6473; confirm topology hostname appears in generated `allowedDomains` in compose YAML after github/gh-aw-firewall#6473 | github/gh-aw-firewall#6189, github/gh-aw-firewall#6438, github/gh-aw-firewall#6473 | |
| | B11 | AWF exits with code 1 and logs show `[WARN] Rootless artifact permission repair failed ... (exit 1)` plus cleanup warnings, but no actionable stderr detail from the repair container | `fixArtifactPermissionsForRootless()` previously discarded repair-container stderr, so the warning hid the real failure context. The non-zero exit code comes from `runAgentCommand()` before cleanup; cleanup warnings do not override it. | **Improved in AWF (PR github/gh-aw-firewall#6072, merged 2026-07-10)**: stderr from the repair container is now captured in the warning, and chroot-home cleanup noise is reduced by downgrading that log from `warn` to `debug`. Treat this mode as diagnostic-opacity around a pre-existing non-zero command exit, not cleanup-driven exit-code mutation. | Compare agent-command logs with `Command completed with exit code: <n>`; on affected versions, repair warnings lack stderr context. On AWF including github/gh-aw-firewall#6072, the warning includes stderr detail for root-cause triage. | github/gh-aw-firewall#6070, github/gh-aw-firewall#6072 | | ||
| | B12 | `getaddrinfo EAI_AGAIN <awmg-cli-proxy>` or `ENOTFOUND <awmg-cli-proxy>` → `awf-cli-proxy could not connect to the external DIFC proxy` → `The agent was never invoked` on ARC/DinD, and DinD containers cannot resolve the Kubernetes Service name even after the ordering fix | ARC/DinD DNS isolation: the DinD Docker network does not forward DNS to the cluster resolver, so Kubernetes Service names such as `awmg-cli-proxy` are unreachable from AWF containers even though the service exists. | **Diagnosed in AWF (PR github/gh-aw-firewall#6328, merged 2026-07-17):** `detectDnsResolutionFailure()` scans cli-proxy logs for `EAI_AGAIN`/`ENOTFOUND`, extracts the unresolved hostname, and augments the startup error with concrete remediation. Address the DIFC proxy by IP, or configure `dockerd --dns <cluster-dns-ip>` so DinD containers can reach the cluster DNS server. | `docker run --rm alpine nslookup awmg-cli-proxy` — `NXDOMAIN`/`SERVFAIL` confirms the Kubernetes DNS is not reachable from DinD containers; compare with the augmented startup error from `detectDnsResolutionFailure()` | github/gh-aw-firewall#6326, github/gh-aw-firewall#6328 | | ||
| | B13 | In `--network-isolation` mode, MCP tool calls to topology-attached peers (e.g. `awmg-mcpg`) or to a `difcProxyHost` configured independently of `topologyAttach` return connection errors or are logged as `TCP_DENIED` in Squid's access log; the agent can reach the DIFC proxy host during startup (B5/B12) but individual in-session MCP/HTTP calls to those same hosts fail or produce Squid 403 noise | `buildProxyEnvironment()` assembled `NO_PROXY` from a static list of AWF-internal hosts but did not include (a) `config.topologyAttach` peer hostnames or (b) `config.difcProxyHost`. Proxy-aware clients (Node `undici`, curl) honoured `HTTPS_PROXY` for those hosts; Squid denied them because they are not in the domain allowlist. | **Fixed in AWF (PR github/gh-aw-firewall#6189, merged 2026-07-20):** `config.topologyAttach` peer hostnames are now added to `NO_PROXY`/`no_proxy`. **Fixed in AWF (PR github/gh-aw-firewall#6438, merged 2026-07-20):** `config.difcProxyHost` (stripped of any `:port` suffix) is also added to `NO_PROXY`. Upgrade AWF to version including both PRs. | Inspect Squid `access.log` for `TCP_DENIED` entries whose target host is a topology peer or the DIFC proxy host; inside the agent container check `echo $NO_PROXY` — the topology peer and DIFC proxy hostname should appear | github/gh-aw-firewall#6189, github/gh-aw-firewall#6438 | | ||
| | B13 | In `--network-isolation` mode, MCP tool calls to topology-attached peers (e.g. `awmg-mcpg`) or to a `difcProxyHost` configured independently of `topologyAttach` return connection errors or are logged as `TCP_DENIED` in Squid's access log; the agent can reach the DIFC proxy host during startup (B5/B12) but individual in-session MCP/HTTP calls to those same hosts fail or produce Squid 403 noise | `buildProxyEnvironment()` assembled `NO_PROXY` from a static list of AWF-internal hosts but did not include (a) `config.topologyAttach` peer hostnames or (b) `config.difcProxyHost`. Proxy-aware clients (Node `undici`, curl) honoured `HTTPS_PROXY` for those hosts; Squid denied them because they are not in the domain allowlist. | **Fixed in AWF (PR github/gh-aw-firewall#6189, merged 2026-07-20):** `config.topologyAttach` peer hostnames are now added to `NO_PROXY`/`no_proxy`. **Fixed in AWF (PR github/gh-aw-firewall#6438, merged 2026-07-20):** `config.difcProxyHost` (stripped of any `:port` suffix) is also added to `NO_PROXY`. **Fixed in AWF (PR github/gh-aw-firewall#6473, merged 2026-07-21):** Topology peer hostnames and `difcProxyHost` are also auto-added to the Squid ACL allowlist (via `resolveAllowedDomains()`) so that proxy-aware clients that honour `HTTP(S)_PROXY` but ignore `NO_PROXY` are still permitted by Squid. The gating mirrors `buildProxyEnvironment()`: only active when `networkIsolation` is true and the runtime is compose-based (`runtimeUsesComposeAgent()`); microVM backends (sbx) are excluded. Upgrade AWF to the version including github/gh-aw-firewall#6473. | Inspect Squid `access.log` for `TCP_DENIED` entries whose target host is a topology peer or the DIFC proxy host; inside the agent container check `echo $NO_PROXY` — the topology peer and DIFC proxy hostname should appear; inspect Squid `access.log` for `TCP_DENIED` entries for topology peer or DIFC proxy host even after upgrading to AWF with github/gh-aw-firewall#6189/#6438 — present on AWF missing github/gh-aw-firewall#6473; confirm topology hostname appears in generated `allowedDomains` in compose YAML after github/gh-aw-firewall#6473 | github/gh-aw-firewall#6189, github/gh-aw-firewall#6438, github/gh-aw-firewall#6473 | |
| expect(content).toContain('**Further fixed in AWF (PR github/gh-aw-firewall#6460, merged 2026-07-21):** `/etc/pki/ca-trust` and `/etc/pki/tls` are now included in the chroot mount policy'); | ||
| expect(content).toContain('github/gh-aw-firewall#5733, github/gh-aw-firewall#5783, github/gh-aw-firewall#6460'); | ||
| expect(content).toContain('**Fixed in AWF (PR github/gh-aw-firewall#6473, merged 2026-07-21):** Topology peer hostnames and `difcProxyHost` are also auto-added to the Squid ACL allowlist'); | ||
| expect(content).toContain('confirm topology hostname appears in generated `allowedDomains` in compose YAML after github/gh-aw-firewall#6473'); |
|
@copilot address review feedback. don't update .lock.yml files directly; only recompile them if the corresponding .md changed. |
…ains in Compose YAML
Fixed in commit The B13 probe in both catalogs (shared + portable) now directs operators to inspect |
|
✅ Copilot review passed with no inline comments. @copilot Add the |
|
📰 VERDICT: Smoke Docker Sbx has concluded. All systems operational. This is a developing story. 🎤 |
|
🔑 Smoke Copilot PAT PAT auth validated. All systems operational. ✅ |
|
🚀 Security Guard has started processing this pull request |
|
✅ Smoke Copilot BYOK AOAI (api-key) completed. Copilot AOAI BYOK (api-key) mode operational. 🔓 |
|
📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅ |
|
✅ Smoke Claude passed |
|
✅ Smoke Gemini completed. All facets verified. 💎 |
|
✅ Build Test Suite completed successfully! |
|
✅ Contribution Check completed successfully! Contribution check complete: the PR updates the shared failure-mode catalog and the mirrored tests in the expected locations, and the description references the related issue. No CONTRIBUTING.md violations found. |
|
✅ Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓 |
|
📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤 |
|
🔌 Smoke Services — All services reachable! ✅ |
|
🛡️ Smoke Copilot Network Isolation confirmed the egress allowlist is enforced. ✅ |
✅ Coverage Check PassedOverall Coverage
📁 Per-file Coverage Changes (1 files)
Coverage comparison generated by |
Smoke Test: Claude Engine Validation
Overall Result: PASS ✅
|
Smoke Test: Copilot BYOK (Direct Mode) ✅Test Results:
Status: PASS All smoke tests passed successfully.
|
🔍 Smoke Test Results
Overall: PARTIAL — MCP connectivity confirmed; pre-computed step outputs were not interpolated into the prompt (template vars arrived as literals).
|
🔥 Smoke Test: Copilot Network Isolation Egress EnforcementEGRESS_RESULT allow=pass deny=pass
Overall: PASS — Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "example.com"See Network Configuration for more information.
|
github-actions
Bot
added
the
smoke-copilot-network-isolation
Smoke Test: Copilot PAT Auth
Overall: PARTIAL — MCP connectivity confirmed; pre-computed step outputs were not injected (template variables unresolved). Auth mode: PAT (COPILOT_GITHUB_TOKEN)
|
Smoke Test Results — FAIL
Overall: FAIL —
|
🔍 Smoke Test: API Proxy OpenTelemetry Tracing
All 5 scenarios pass. OTEL tracing integration is functional with graceful degradation when endpoints are unconfigured.
|
|
|
|
Smoke test results:
|
Smoke Test Results (Gemini)
Overall status: FAIL Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "localhost"See Network Configuration for more information.
|
🏗️ Build Test Suite Results
Overall: 8/8 ecosystems passed — ✅ PASS
|
🔬 Smoke Test Results — Docker Sbx
Overall: INDETERMINATE — pre-computed smoke-data outputs were not substituted by the workflow runner. The
|
Runner Doctor’s 2026-07-20/21 scan surfaced two new catalog deltas: B13 needed the final Squid ACL closure beyond NO_PROXY-only fixes, and B9 needed the newer RHEL/Amazon Linux CA mount-policy citation/update. This PR updates the shared failure-mode catalog and the portable embedded catalog so triage guidance stays aligned.
B13: complete the topology/DIFC proxy fix chain
github/gh-aw-firewall#6473:difcProxyHostnow auto-added to Squid ACL allowlist viaresolveAllowedDomains()networkIsolation=true+ compose runtime (runtimeUsesComposeAgent()), excluding sbxTCP_DENIEDand generatedallowedDomains.#6473to B13 citations.B9: include mount-policy hardening for RHEL-family CA paths
github/gh-aw-firewall#6460:/etc/pki/ca-trustand/etc/pki/tlsare now in chroot mount policy, making/host/etc/pkiCA paths directly readable.#6460to B9 citations.Catalog parity guardrails
.github/workflows/shared/self-hosted-failure-modes.md.github/agents/self-hosted-runner-doctor.mdscripts/ci/self-hosted-runner-doctor-workflow.test.tsassertions to lock the new B9/B13 text/citations across shared + portable copies.