fix(runner): agent mounts return EACCES for ~15 min after STS credentials expire (never refreshed)

[mmabrouk]

Summary

Agent sandbox mounts are backed by geesefs (FUSE → S3/SeaweedFS). The runner signs
900-second STS credentials at mount time and passes them to the geesefs child as static
AWS_* env vars. Nothing ever refreshes them, and the daemon outlives them.

Fifteen minutes after any mount, SeaweedFS starts answering that daemon with 403.
geesefs maps 403 → EACCES, so every path under the mount returns "Permission denied" —
not ENOENT, not EIO — until something remounts it. The mount looks alive the whole time
(mountpoint -q passes, the daemon is running, no data is lost), which is why this reads as
a mysterious intermittent outage rather than an expiry.

Observed on the OSS deployment on 2026-07-25: 16 denial windows in one day, the longest
14m10s. Each one interrupted an in-flight writer and left stale index.lock / config.lock
behind. Any agent turn that runs longer than 15 minutes is guaranteed to break mid-flight.

Symptom inside the sandbox:

$ ls .git
ls: cannot open directory '.git': Permission denied
$ git status
fatal: not a git repository (or any of the parent directories): .git

Impact

• Long agent turns break mid-flight, deterministically, at the 15-minute mark.
• Interrupted writers leave stale git lock files, which wedge the next turn too.
• Recovery is incidental — it only happens because the next turn boundary remounts. There is
  no detection and no self-heal.
• Hits the durable per-agent volume hardest: it is long-lived and shared across every
  concurrent session of an agent, whereas session scratch dirs are re-signed at each turn
  boundary and so mostly dodge the window.

Root cause

api/oss/src/core/store/storage.py:193 signs with the 15-minute default:

async def sign_temp_credentials(
    self, *, bucket: str, prefix: str,
    duration_seconds: int = 900,
) -> Credentials:

services/runner/src/engines/sandbox_agent/mount.ts:238 hands them to geesefs as static env:

const env: Record<string, string> = {
  AWS_ACCESS_KEY_ID: creds.accessKey,
  AWS_SECRET_ACCESS_KEY: creds.secretKey,
};
if (creds.sessionToken) env.AWS_SESSION_TOKEN = creds.sessionToken;

There is no refresh, renew, or re-sign path anywhere in mount.ts / agent-mount.ts —
the only setTimeouts are mount-poll delays.

The code already anticipates the hard version of this failure at mount.ts:249:

 * `mountpoint -q` returns true for a STALE geesefs node — one whose daemon died (expired
 * STS creds, SeaweedFS restart) leaving the kernel entry behind. That node answers every
 * file op with ENOTCONN ("Transport endpoint is not connected") […]
 * Probe with a real access (`ls -A`) and treat ENOTCONN as NOT-mounted so the caller remounts.

It only handles ENOTCONN — the daemon dying. The soft death, where the daemon keeps
running with dead credentials and returns EACCES, passes the liveness probe and is never
detected.

Proposed fixes

1. Detect the soft death. Extend the isMounted probe at mount.ts:249 to treat EACCES
   the same as ENOTCONN — not-mounted, so the caller remounts. This is the highest-value fix:
   it turns multi-minute outages into sub-second blips and removes the stale-lock damage,
   because writers stop getting interrupted mid-operation.
2. Refresh before expiry. Re-sign and remount at ~80% of TTL, or drive geesefs from a
   credential-process / shared-config file the runner rewrites, so no daemon ever outlives its
   credentials. This is the actual fix.
3. Raise duration_seconds past the maximum expected turn length. One-line mitigation,
   buys time, does not fix anything.
4. Give each mount its own RoleSessionName. storage.py:238 uses a constant
   (STORE_SUBJECT = "agenta-store") for every mount, every session, every tenant, so they all
   share one STS session identity. Worth fixing on isolation grounds regardless of this bug.

---

Appendix — investigation

Environment

• Container: agenta-oss-team-runner-1, image agenta-allharness-sidecar:latest
• Backend: geesefs → SeaweedFS (chrislusf/seaweedfs:4.37), bucket agenta-store
• Affected mount: the durable per-agent volume, …/<session-id>-agent

geesefs --endpoint https://<store-endpoint> --region us-east-1 --no-detect
        --fsync-on-close -f -o allow_other
        agenta-store:mounts/<org-id>/<agent-id>
        /tmp/agenta/mounts/<org-id>/<session-id>-agent

Timeline of one window (16:05:42 → 16:08:34 UTC)

Mount created, credentials good for exactly 15 minutes:

15:50:34.435 [sandbox-agent] mountStorage begin cwd=…<session>-agent
             bucket=agenta-store prefix=mounts/<org>/<agent>
             sts=yes expiresAt=2026-07-25T16:05:31
15:50:35.181 [sandbox-agent] geesefs mount argv: …
15:52:25.948 [sandbox-agent] mounted … -> …-agent (verified alive)

Denials begin 11 seconds after that expiry, on both metadata and data ops — which is
exactly why .git cannot be stat'd:

16:05:42 geesefs stderr: fuse.ERROR *fuseops.LookUpInodeOp error: permission denied
16:05:42 geesefs stderr: fuse.ERROR *fuseops.ReadFileOp  error: permission denied
…269 more…

Recovery is the remount at the next turn boundary, not a self-heal:

16:08:34.871 [sandbox-agent] unmounted …<session>-…            (confirmed gone)
16:08:35.597 [sandbox-agent] unmounted …<session>-…-agent      (confirmed gone)
16:08:35.597 [sandbox-agent] geesefs mount argv: … -agent
16:08:36.226 [sandbox-agent] mounted … -agent (verified alive)   ← errors stop here
             mountStorage begin … sts=yes expiresAt=2026-07-25T16:23:34

Correlation across all 16 windows that day

Every EACCES burst and every credential expiry, extracted from the full-day runner log:

window start	duration	nearest expiry	Δ
12:03:59	0:50	12:03:59	+0s
13:37:51	0:49	13:37:50	+1s
15:19:20	5:18	15:19:19	+1s
14:18:41	14:10	14:18:33	+9s
16:05:42	2:52	16:05:31	+11s
14:08:34	0:00	14:08:22	+13s
12:22:38	2:42	12:22:08	+31s
16:01:38	0:27	16:01:01	+37s
11:16:23	0:00	11:15:36	+47s
15:55:49	0:11	15:53:56	+113s
15:31:25	0:00	15:28:12	+194s
13:54:50	0:01	13:50:50	+240s
12:12:10	0:14	12:07:43	+268s
10:32:48	0:26	10:28:07	+281s
13:26:14	0:13	13:37:49	−694s
15:58:35	1:49	16:01:01	−146s

10 of 16 windows begin within 0–180s after a credential expiry. The wider offsets have a
benign explanation: a mount stays broken from its expiry until its next remount, but errors are
only logged when the agent actually touches the filesystem — so the burst starts when I/O
resumes, not when the credential died. The two negative deltas are windows matched against a
later expiry belonging to a different concurrent session's mount.

The 14-minute and 10-minute outages are simply turns that ran longer than the credential
lifetime.

Ruled out

• docker inspect: RestartCount 0, OOMKilled false on both the runner and SeaweedFS;
  runner up since 2026-07-22, SeaweedFS since 2026-07-14. Neither restarted or was killed.
• docker events --since 15:45 --until 16:15: empty.
• dmesg -T | grep -iE 'fuse|oom|killed process|transport endpoint': no matches.
• SeaweedFS logs: no AccessDenied / ExpiredToken / rate-limit entries — though note S3
  request logging is off there (49 lines total across the 30-minute window), so this is absence
  of evidence, not evidence of absence.

Caveats

• The runner multiplexes all geesefs stderr into a single stream with no mountpoint tag, so
  an individual permission denied line cannot be attributed to a specific mount. The
  attribution above rests on the expiry timeline, which lines up cleanly. Adding a mount tag to
  that log prefix would make this class of bug far cheaper to diagnose.
• The 15:50:34 mount printed the same expiresAt=16:05:31 for both the scratch and the
  -agent mount, so in principle both died at 16:05:31. The scratch dir most likely stayed
  usable because its small, freshly-written working set was served from geesefs's cache without
  an S3 round-trip, while the larger, colder durable volume missed cache and hit the 403.
  This part is inference, not proven by the logs.
• A direct proof that a second AssumeRoleWithWebIdentity supersedes the first was not run
  (blocked in the investigation environment). It is not needed for the expiry finding, which
  stands on the timeline above, but it remains the open question behind fix add a simple design and remove flowbite #4.

[linear-code]

AGE-4000