Skip to content

A server-rejected fallback token is invisible until it expires locally — 34h silent outage #118

Description

@iceteaSA

One of my fallback accounts stopped working for 34 hours and nothing in the plugin said so. The sidebar showed 31-hour-stale quota bars and /openai-quota reported fetch failed — Refresh to retry.

The trigger was external: the provider started returning HTTP 401 from POST {CODEX_ISSUER}/oauth/token for that account, invalidating it server-side. Nothing here caused that, and /openai-account remove + add cured it immediately. I'm not asking for the 401 to be prevented.

What's worth fixing is that a recoverable external event became a multi-day silent outage.

The part I did not expect

refreshAccountNow returns early when the token isn't near expiry (core/accounts.ts:2365-2371):

if (
  latestAccount &&
  !options.force &&
  !tokenNeedsRefresh(latestAccount, latestStorage, this.now())
) {
  updateStoredAccount(storage, latestAccount)
  return latestAccount
}

tokenNeedsRefresh consults local expiry only. So a token the server has already rejected, but which still has hours of local validity left, is invisible: no refresh attempted, no log line, no throw. The rejected token goes straight to wham.

In my case that window was ~25 hours. The first fallback oauth refresh request start for the account carries:

expiresInMs: 14379442

Positive — about four hours remaining. The account had been failing since the previous morning; it simply hadn't drifted into the 4-hour pre-expiry window yet. Roughly 2,900 wham calls carried a server-rejected token before the refresh channel logged anything at all.

Local expiry is being used as a proxy for server validity, and a 401 from the resource endpoint never feeds back into that decision.

Then the second act

Once the token did enter the refresh window, ~690 refresh attempts ran, every one of them a 401, and core/refresh-all-quota.ts:264 threw the reason away:

try {
  refreshed = await deps.fallbackManager.refreshAccount(acct, storage)
} catch {
  refreshed = acct
}

Across three days of rotated logs, exactly one copy of Token refresh failed: 401 survives — and it came from the background path, which handles the identical error correctly (core/accounts.ts:2175-2181): it logs, then calls recordRefreshError, which arms a 24h backoff.

So one half of the plugin diagnosed the account as permanently broken and recorded it. The other half retried 690 times, because it never reads that. refresh-all-quota.ts contains no reference to lastRefreshError or refreshBackoffActive, and the command path's call site passes no respectBackoff at all (index.ts:1750-1766) — so it consults neither backoff store.

Worth flagging for whoever fixes this: adding a lastRefreshError check inside refresh-all-quota.ts alone would be dead code. The caller wiring (or the default) has to change too.

Two smaller ones in the same path:

  • :268 guards with !refreshed.access — presence, not validity. An expired token is a non-empty string, so it passes and reaches wham. The main-account path 100 lines above already does (auth.expires ?? 0) < deps.now().
  • The resulting error surfaces as wham usage check failed: 401, which points at the quota endpoint. The actual failure is token refresh. I spent a while suspecting the wrong component because of this.

Reproduction

An isolated probe drives the real refreshAllQuota with the real FallbackAccountManager, stubbing only codexRefreshFn and whamFn at the network boundary, wired as index.ts:1750-1766 wires it. 9/9 assertions.

Phase 1 — token locally valid (8h remaining), server rejects it:

  • zero fallback oauth refresh request start lines
  • codexRefreshFn never called
  • whamFn receives the stale token
  • outcome {ok:false, error:"wham usage check failed: 401"}
  • lastRefreshError not set
  • second call behaves identically — no backoff

Phase 2 — token expired:

  • request start logged, codexRefreshFn called
  • its error swallowed; the surfaced error is still the wham 401, not Token refresh failed: 401
  • lastRefreshError still not set

Happy to attach the probe if useful — it's the natural regression test for a fix, and it needs OPENCODE_OPENAI_AUTH_FILE, _STATE_FILE and _SIDEBAR_STATE_FILE isolated to temp paths to run safely.

Fix shapes

Ranked by how much each would have shortened the outage, not by effort:

  1. Feed server rejection back into the refresh decision. If the last quota fetch saw a 401 for this token fingerprint, force a refresh regardless of local expiry. This collapses the 25-hour silent window to roughly zero and is the only item that addresses the class rather than a symptom.
  2. Log and record in that catch. It needs to stay non-fatal for a transient network blip, but recordRefreshError already distinguishes transient from non-transient — the background path proves the machinery exists.
  3. Let the command path see the refresh backoff. Requires the call-site change noted above.
  4. Check expiry, not presence, at :268 — mirror the main path.
  5. Render the per-account error and a staleness indicator rather than a generic fetch failed next to day-old bars.

I don't know why the provider invalidated that account. It's a ChatGPT Work tenant and the plan-tier changes are rolling out, but that's speculation and I can't evidence it. The point is that the plugin shouldn't need that answer to notice.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions