core-service panic when key refresh is impossible

[the-glu]

Describe the bug
The core-service panics (and therefore shuts down the service) when it is unable to refresh keys on the jwks_endpoint.

To Reproduce
Start the core-service with a jwks_endpoint. Prevent the DSS from accessing this endpoint (e.g. shut down the server).

Monitoring: disabled
Debug Mode: off
2026-08-12T14:16:13.656Z        info    core-service/main.go:255        version {"address": ":8082", "version": "0.0.0+3b904b48-dirty"}
2026-08-12T14:16:13.656Z        info    core-service/main.go:256        build   {"address": ":8082", "description": {"Time":"2026-08-07.16:21:39","Commit":"3b904b48-dirty","Host":"buildkitsandbox"}}
2026-08-12T14:16:13.656Z        info    core-service/main.go:257        config  {"address": ":8082", "scd": true}
2026-08-12T14:16:13.656Z        info    core-service/main.go:259        config  {"address": ":8082", "scdGlobalLock": false}
2026-08-12T14:16:13.656Z        info    core-service/main.go:260        config  {"address": ":8082", "scdHashLock": false}
2026-08-12T14:16:13.656Z        info    core-service/main.go:261        config  {"address": ":8082", "timeBasedNotificationIndex": false}
2026-08-12T14:16:13.787Z        info    core-service/main.go:409        Starting DSS HTTP server        {"address": ":8082"}
2026-08-12T14:16:18.679Z        info    logging/http.go:109     GET /healthy HTTP/1.1   {"address": ":8082", "req_dump": "", "resp_dump": "ok", "req_headers": {"Accept":["*/*"],"Connection":["close"],"User-Agent":["Wget"]}, "resp_status_code": 200, "resp_status_text": "OK", "peer_address": "[::1]:44456", "start_time": "2026-08-12T14:16:18.679Z", "duration_ms": 0.1}

(http://172.17.0.1:8000/oauth2/jwks.json on my machine service a key set stopped)

2026-08-12T14:16:33.779Z        panic   auth/auth.go:161        failed to refresh key   {"error": "Error retrieving JWKS at http://172.17.0.1:8000/oauth2/jwks.json\n --- at /app/pkg/auth/auth.go:94 (JWKSResolver.ResolveKeys) ---\nCaused by: Get \"http://172.17.0.1:8000/oauth2/jwks.json\": dial tcp 172.17.0.1:8000: connect: connection refused"}
github.com/interuss/dss/pkg/auth.NewRSAAuthorizer.func1
        /app/pkg/auth/auth.go:161
panic: failed to refresh key

goroutine 117 [running]:
go.uber.org/zap/zapcore.CheckWriteAction.OnWrite(0x1?, 0x1?, {0x0?, 0x0?, 0x29448e524220?})
        /go/pkg/mod/go.uber.org/zap@v1.27.0/zapcore/entry.go:196 +0x54
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0x29448e24e1a0, {0x29448e246400, 0x1, 0x1})
        /go/pkg/mod/go.uber.org/zap@v1.27.0/zapcore/entry.go:262 +0x26e
go.uber.org/zap.(*Logger).Panic(0x29448deda4e0?, {0x141dd50?, 0x29448df16500?}, {0x29448e246400, 0x1, 0x1})
        /go/pkg/mod/go.uber.org/zap@v1.27.0/logger.go:285 +0x4b
github.com/interuss/dss/pkg/auth.NewRSAAuthorizer.func1()
        /app/pkg/auth/auth.go:161 +0x2b5
created by github.com/interuss/dss/pkg/auth.NewRSAAuthorizer in goroutine 1
        /app/pkg/auth/auth.go:152 +0x245

Expected behavior
I would expect the DSS to cache the keys at least for a while. A number of retries / a delay could be used to still prevent usage of outdated keys, but a single request failure should probably not stop the DSS server.

[BenjaminPelletier]

I agree: a single retrieval failure when the public key is already cached shouldn't panic. As a default/starting point, it seems like aligning validation behavior with token characteristics would be a reasonable starting point. I would suggest panicking on public key retrieval if:

1. No public key is available, or
2. It has been at least 1 hour since a successful public key retrieval and at least N attempts have been unsuccessful (maybe set N=3)

We might consider automatically adopting the security stance selected by the authority by changing the 1-hour limit according to values in the tokens received -- specifically, set the public key panic timeout to the maximum duration between when a token was received and that token's expiration, but not longer than 1 hour. For instance, we could start out with a 1 hour timeout. Then, if we received a token 27 minutes before its expiration, we would set the timeout to 27 minutes. If we received a token 23 minutes before its expiration, we would keep the timeout at 27 minutes. Then, if we received a token 29 minutes before its expiration, we would set the timeout to 29 minutes. Even 0 minutes (the first and only token received was about to expire) should be ok since we wouldn't panic until N unsuccessful attempts to retrieve the public key. This could probably be on a per-instance (or even per-thread) basis as we wouldn't want to lock workers to a shared resource storing this dynamic timeout.

[the-glu]

I agree: a single retrieval failure when the public key is already cached shouldn't panic. As a default/starting point, it seems like aligning validation behavior with token characteristics would be a reasonable starting point. I would suggest panicking on public key retrieval if:

1. No public key is available, or

2. It has been at least [1 hour](https://github.com/astm-utm/Protocol/blob/F3548-21/utm.yaml#L92) since a successful public key retrieval and at least N attempts have been unsuccessful (maybe set N=3)

I implemented the basic version in #1634, with a small change: since the number of attempts and the time since the last retrieval are directly linked (via keyRefreshTimeout which defines the interval), having both seems confusing and redundant. However, this can be added if there is a case I missed where both are needed.