Clarify the relationship between `HttpOptions.timeout()` (X-Server-Timeout header) and OkHttp `callTimeout`

[LukeYu22]

Description

I'm using google-genai:1.32.0 with Vertex AI for image generation (models.generateContent with responseModalities=IMAGE). I need help understanding the interaction between two timeout mechanisms in the SDK.

Two timeout paths in the SDK

1. Client-level HttpOptions.timeout() → OkHttp callTimeout

Set during client initialization:

HttpOptions httpOptions = HttpOptions.builder()
        .timeout(300000)  // 300s
        .retryOptions(HttpRetryOptions.builder().attempts(2).build())
        .build();

Client client = Client.builder()
        .project("my-project")
        .location("global")
        .httpOptions(httpOptions)
        .build();

From source code (ApiClient.createHttpClient), this sets OkHttpClient.Builder.callTimeout(Duration.ofMillis(timeout)), which covers the entire call lifecycle including all retry attempts.

2. Per-request HttpOptions.timeout() → X-Server-Timeout HTTP header

Set on GenerateContentConfig:

GenerateContentConfig config = GenerateContentConfig.builder()
        .responseModalities("IMAGE")
        .httpOptions(HttpOptions.builder()
                .timeout(180000)  // 180s
                .build())
        .build();

client.models.generateContent(model, prompt, config);

From source code (ApiClient.getTimeoutHeader), this converts timeout / 1000 and sends as X-Server-Timeout: 180 HTTP header. It does NOT override the OkHttp callTimeout.

My questions

Q1: What does X-Server-Timeout do on the server side?

Does the Vertex AI server respect this header? If so:

• Does it abort image generation processing after the specified seconds?
• What HTTP status code / error does it return when server-side timeout is reached?
• Is this behavior the same for all model types (text, image, video)?

Q2: How do these two timeouts interact with RetryInterceptor?

The callTimeout covers all retry attempts combined (not per-attempt). With attempts=2:

Scenario: X-Server-Timeout: 180, callTimeout: 300s

t=0s     attempt 1 starts
t=180s   server returns timeout error (respecting X-Server-Timeout: 180)
t=182s   attempt 2 starts (after ~2s backoff)
         BUT only 118s remains in callTimeout budget
t=300s   callTimeout fires → InterruptedIOException: timeout
         attempt 2 is killed (never had a chance to complete)

Is this the expected behavior? It seems like the second attempt can never succeed if the first attempt exhausts most of the callTimeout budget.

Q3: What is the recommended configuration?

For long-running operations like image generation (which can take 30-120s), what is the recommended way to configure timeouts?

Specifically:

• Should we set per-request timeout at all, or just rely on client-level callTimeout?
• If we want retries (attempts=2), should callTimeout be at least X-Server-Timeout × attempts + backoff?
• Is there a way to set a per-attempt timeout instead of a per-call timeout?

Q4: What happens when timeout is not set?

• Client-level: What is the default callTimeout? From bytecode analysis, it appears to only be set when HttpOptions.timeout() is present.
• Per-request: If no timeout is set on per-request HttpOptions, no X-Server-Timeout header is sent. What is the server's default processing timeout for image generation?

Environment

• SDK version: google-genai:1.32.0
• API: Vertex AI (not Gemini Developer API)
• Model: gemini-2.5-flash-image (image generation)
• Use case: High-concurrency image generation (~120 concurrent requests)

Stack trace (when callTimeout is hit)

GenAiIOException: Failed to execute HTTP request.
    at c.g.g.HttpApiClient.executeRequest(HttpApiClient.java:83)
Caused by: java.io.InterruptedIOException: timeout
    at okhttp3.internal.connection.RealCall.timeoutExit(RealCall.kt:398)
    at okhttp3.internal.connection.RealCall.callDone(RealCall.kt:360)
Caused by: java.io.IOException: Canceled
    at okhttp3.internal.connection.RealCall.noMoreExchanges$okhttp(RealCall.kt:325)

What I've tried

Config	Result
callTimeout=60s, attempts=5 (default)	Fast overall, but some image gen requests fail (too short)
callTimeout=180s, attempts=2, per-request timeout=180s	Still timeout — attempt 2 never has enough budget
callTimeout=300s, attempts=2, per-request timeout=180s	Same issue — 180+2+180=362 > 300

Thank you for any guidance!

[hemasekhar-p]

Hi @LukeYu22, thank you for reaching out. You've hit on a very common point of friction when dealing with long-running API calls wrapped by OkHttp. Here's a breakdown of exactly what's happening and how to configure this for your high-concurrency image generation use case.

What does X-Server-Timeout do on the server side?

This header explicitly instructs the Google / Vertex AI backend on how long it should spend processing the request. If the server hits this limit, it gracefully aborts the generation process and returns a clean HTTP error. typically a 504 Gateway Timeout or 408 Request Timeout instead of keeping the TCP connection hanging. And this is a universal Google Cloud / Vertex AI routing header, so it applies across all model types.

How do these two timeouts interact with RetryInterceptor

This is an global design choice by the OkHttp. callTimeout acts as an absolute, overarching deadline for the entire logical call. It pools all retry attempts, DNS resolution, and connection times into a single budget.

Because the SDK maps the client-level HttpOptions.timeout() directly to OkHttp's callTimeout(), it does not act as a per-attempt limit. If your first attempt consumes 180s of a 300s total budget, OkHttp will terminate the second attempt in middle once callTimeout() reaches the 300s.

What is the recommended configuration?

For long-running operations like image generation that require retries, you must configure both.

X-Server-Timeout: the per-request timeout as your strict per-attempt limit to force the server to fail fast.
callTimeout: client-level timeout as the Global timeout allowing all retries and backoff delays to safely execute.
Following is the recommended configuration for callTimeout
Client timeout >= (Per-request timeout × attempts) + (total backoff time)

// Client-level:
HttpOptions clientHttpOptions = HttpOptions.builder()
        .timeout(600000)  // (600s >= (180*2)+backoff time)
        .retryOptions(HttpRetryOptions.builder().attempts(2).build())
        .build();
Client client = Client.builder()
        .project(GenaiUtil.PROJECT) // your projectId
        .location(GenaiUtil.LOCATION) //your location
        .httpOptions(clientHttpOptions)
        .build();

// Request-level
GenerateContentConfig config = GenerateContentConfig.builder()
        .responseModalities(Arrays.asList("IMAGE"))
        .httpOptions(HttpOptions.builder()
                .timeout(180000)  //180s
                .build())
        .build();

What happens when timeout is not set?

callTimeout: OkHttp defaults callTimeout to 0. that mean no-timeout.
X-Server-Timeout: If we didn't pass timeout from header, request falls back to Vertex AI's implicit backend infrastructure timeout limits.

[LukeYu22]

Thank you @hemasekhar-p for the detailed response! The formula callTimeout >= (per-request timeout × attempts) + total backoff makes everything clear now.

I have two follow-up questions specific to image generation workloads:

Q5: What is the recommended X-Server-Timeout for Nanobanana image generation?

In your example, you used timeout(180000) (180s) as the per-request timeout. Is 180s the officially recommended value for image generation (gemini-2.5-flash-image with responseModalities=IMAGE)?

Specifically:

• What is the p99 latency for a typical image generation request on Vertex AI? (e.g., single image, 1024x1024)
• Under high concurrency (~100 concurrent requests), does server-side queuing significantly increase latency beyond the normal processing time?

Q6: Does the server use X-Server-Timeout to manage queued requests?

Under high concurrency, I assume Vertex AI queues requests. When a queued request has waited longer than X-Server-Timeout:

• Does the server discard it from the queue without processing? (saving resources)
• Or does it only start the timeout countdown after processing begins?

This matters for choosing the right value — if the timeout includes queue wait time, we'd need a larger value under high concurrency.

My current thinking

Since we have business-layer retry via SQS (exponential backoff: 5s → 10s → 20s → ... → 120s), I'm considering:

// Option A: No SDK retry, rely on business-layer retry
// callTimeout = 130s, per-request = 120s, attempts = 1
HttpOptions clientOpts = HttpOptions.builder()
    .timeout(130000)
    .retryOptions(HttpRetryOptions.builder().attempts(1).build())
    .build();

// per-request
.httpOptions(HttpOptions.builder().timeout(120000).build())

// Option B: SDK retry with your recommended formula
// callTimeout = 260s >= 120×2 + backoff, per-request = 120s, attempts = 2
HttpOptions clientOpts = HttpOptions.builder()
    .timeout(260000)
    .retryOptions(HttpRetryOptions.builder().attempts(2).build())
    .build();

// per-request
.httpOptions(HttpOptions.builder().timeout(120000).build())

Is 120s a reasonable per-request timeout for image generation, or should we stick with 180s as in your example? Any guidance on the typical latency distribution would help us make the right choice.

Thanks again!

[LukeYu22]

With ~160 concurrent consumers (80 for imageGen + 80 for imageEdit, across 2 machines), I'm seeing ~560 timeout errors per 5 minutes. The stack trace is always the same callTimeout pattern:

com.google.genai.errors.GenAiIOException: Failed to execute HTTP request.
    at com.google.genai.HttpApiClient.executeRequest(HttpApiClient.java:83)
    at com.google.genai.HttpApiClient.request(HttpApiClient.java:65)
    at com.google.genai.HttpApiClient.request(HttpApiClient.java:33)
    at com.google.genai.Models.privateGenerateContent(Models.java:4954)
    at com.google.genai.Models.generateContent(Models.java:6284)
    at com.google.genai.Models.generateContent(Models.java:6343)
    at ...GoogleVertexAiClient.imageEditByNanoBanana(GoogleVertexAiClient.java:453)
    ...
Caused by: java.io.InterruptedIOException: timeout
    at okhttp3.internal.connection.RealCall.timeoutExit(RealCall.kt:398)
    at okhttp3.internal.connection.RealCall.callDone(RealCall.kt:360)
    ...
Caused by: java.io.IOException: Canceled
    at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:72)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at com.google.genai.RetryInterceptor.intercept(RetryInterceptor.java:82)
    ...

Context

• Previously with 120 consumers and callTimeout=60s, timeouts were rare
• After increasing to 160 consumers and callTimeout=80s, timeouts became very frequent
• The model has not changed — it's the same gemini-2.5-flash-image throughout
• I suspect the increased concurrency is causing server-side queuing, and requests are timing out while waiting in the queue

Questions

1. What is the recommended max concurrency for gemini-2.5-flash-image on Vertex AI? Is there a per-project concurrent request limit that could cause queuing at ~160 concurrent requests?

2. What is the typical p99 latency for gemini-2.5-flash-image image generation under moderate vs. high concurrency? This would help me set an appropriate X-Server-Timeout value.

3. Does X-Server-Timeout include queue wait time? If the server queues my request for 50s before starting to process it, does the 70s X-Server-Timeout countdown start from when the request arrives (including queue time) or when processing begins? This is critical for choosing the right value.

4. Should I reduce concurrency or increase timeout? Given the formula callTimeout >= X-Server-Timeout × attempts + backoff:

   • Reducing consumers to 120 + keeping callTimeout=80s → fewer requests in queue → less timeout
   • Keeping 160 consumers + increasing callTimeout=180s → threads blocked longer → risk of cascading failure

   What does the Vertex AI team recommend for high-throughput image generation workloads?

[hemasekhar-p]

@LukeYu22, Here is the breakdown of how Vertex AI handles these high-throughput image generation workloads.

The X-Server-Timeout countdown begins from the load balancer receives your request, meaning it strictly includes all server-side queue wait time. If your requests are taking longer than 60 seconds, they are likely stuck in a queue, eating into your timeout budget before the GPU even starts processing. Because of this queueing behavior, a per-request timeout of 120s is widely recommended to ensure the server has enough time to both hold and process the image before intentionally dropping the job.

reduce concurrency vs increase timeout:
we recomand reduce your concurrency to match your actual Google Cloud quota limits. Keeping 160 consumers active while increasing the timeout to 180s will just hold your local JVM threads hostage waiting in Google's queue. This will lead to thread-pool exhaustion and memory spikes.

recommended architecture for high-throughput:
As your have business-layer retry via SQS, let your SQS message broker handle all the retries and exponential backoff. Configure the SDK to fail fast without retrying internally means use single attempt, using a 130s as client callTimeout and a 120s as X-Server-Timeout. This cleanly releases your JVM resources when the server is busy.