Pub/Sub trigger: a redelivered message re-runs the agent in a new session, repeating tool side effects that already happened

[FurkhanShaikh]

🔴 Required Information

Describe the Bug:
Suppose a Pub/Sub trigger run fails after a tool has already caused an external
side effect, such as a payment, an email or a ticket.

• The endpoint returns HTTP 500 and Pub/Sub redelivers the message.
• ADK runs the agent again from the start, in a new session with no record of
  the first run, so the side effect happens again.

Nothing inside the agent can prevent this. Pub/Sub keeps the messageId the
same across redeliveries ("A redelivered message retains the same message ID
between redelivery attempts"). ADK logs it but never passes it to the agent or
its tools, so a tool has nothing to deduplicate on.

In src/google/adk/cli/trigger_routes.py
(v2.10.0; byte-identical in 2.9.2):

• L419:
  session_id = str(uuid.uuid4()), once per HTTP delivery.
• L514-516:
  the agent receives {"data": ..., "attributes": ...}. The messageId is
  not included.
• L518-522:
  the messageId is only logged.
• L530-541:
  HTTP 500 for a TransientError after retries, and for any other exception.
• L486,
  the endpoint's own description: "errors trigger Pub/Sub retry".

The ambient agents docs describe the
design: "Each redelivery creates a new session. Trigger workloads are stateless
by design." They don't say what this means for tools with side effects.

The in-process 429 retry is fine: it reuses the session, so the model can see
the tool call that already completed. The problem is the hand-off to Pub/Sub.

Steps to Reproduce:

1. pip install google-adk==2.10.0
2. Save the script under "Minimal Reproduction Code" below as repro.py.
3. Run python repro.py.

It needs no network access and no API key.

• The agent is a stock LlmAgent on ADK's Gemini class, with
  base_url pointed at a local fake Gemini endpoint.
• The fake endpoint answers the first request with a pay_invoice call.
  It answers the request after the tool with HTTP 503 UNAVAILABLE ("The model
  is overloaded") once, then behaves normally.
• The loop at the bottom plays Pub/Sub. After a non-2xx response it posts
  the same push envelope again, with the same messageId.

Expected Behavior:

One payment for one message. Failing that, a way for the agent or its tools to
recognise a redelivery, so they can skip a side effect that already happened.

Observed Behavior:

google-adk 2.10.0
delivery 1 of messageId 1234567890: HTTP 500
delivery 2 of messageId 1234567890: HTTP 200
payments made for one message: 2

For delivery 1, ADK logs
Error processing Pub/Sub message: 503 UNAVAILABLE. {'error': {'code': 503, 'status': 'UNAVAILABLE', 'message': 'The model is overloaded. Please try again later.'}}.

Environment Details:

• ADK Library Version (pip show google-adk): 2.10.0. Also 2.9.2, whose
  trigger_routes.py is byte-identical.
• Desktop OS: Windows 11
• Python Version (python -V): 3.13.9

Model Information:

• Are you using LiteLLM: No
• Which model is being used: gemini-2.5-flash through ADK's Gemini class,
  answered by a local fake endpoint. No real model was called.

---

🟡 Optional Information

Regression:

No. The same code is in the commit that added the trigger endpoints
(e2d970f).

Additional Context:

The same duplicate has other routes. They were measured with the same kind of
local harness: a fake Gemini endpoint, and Pub/Sub emulated from its documented
redelivery rule.

What happens after the tool's side effect	Deliveries	Side effects
the next model call returns 503	500, 200	2
the model returns 429 through all 3 in-process retries	500, 200	2
the tool's own HTTP call times out after the provider committed	500, 200	2
no error: the run outlasts the subscription's acknowledgement deadline	resent while the first run is still running	2 (both runs complete)
the failure repeats on every delivery	500 ×5	5

Two rows deserve a note:

• The acknowledgement-deadline row needs no failure at all. A push
  subscription's acknowledgement deadline defaults to 10 seconds, and "if you
  send a negative acknowledgment or the acknowledgment deadline expires,
  Pub/Sub resends the message". The docs give "10 minutes (ack deadline)" as
  the maximum processing time, but don't mention that a subscription created
  with defaults gets 10 seconds.
• The last row has no cap. Without a dead-letter policy, a failure that
  repeats after the side effect repeats the side effect on every redelivery,
  until the message expires. The docs recommend a dead-letter queue but don't
  say why it matters here.

Two related observations:

• Eventarc (Python): direct CloudEvents do put ce-id in the agent's
  input (L617-627); Pub/Sub-wrapped events don't.

• google/adk-go appears to share the design. I read its source but did
  not run it:

  • RetriableRunner.RunAgent creates one session per delivery (its comment:
    "One session per delivery").
  • messageContentFromPubSub leaves out the message ID.
  • A failed run returns 500.

  I'm happy to open a companion issue there.

Possible changes, roughly by cost:

1. Pass the delivery's identity into the run. For example, put messageId
   (and ce-id) in session state or in the attributes the agent receives, so
   a tool can derive an idempotency key for its provider. This is small, and
   lets applications fix the problem themselves.
2. Derive session_id from the delivery (e.g. uuid5 of subscription and
   messageId) instead of uuid4().
   • A redelivery then lands in the session that holds the completed tool
     calls, as the in-process retry already does.
   • ADK could also acknowledge a redelivery whose session already holds a
     final response, which covers the acknowledgement-deadline case.
   • A redelivery that arrives while the first run is still going would need
     a guard.
3. Document it. Say that a redelivery repeats side effects that already
   happened. Recommend an acknowledgement deadline above the expected run time
   (up to 600 s), a dead-letter policy, and idempotent tools. Show how a
   publisher can carry an idempotency key in the message attributes, which do
   reach the agent today.

I'm happy to be told this is intended. In that case, (3) alone would still help.

Found while auditing retry behaviour across agent frameworks.

Minimal Reproduction Code:

"""Pub/Sub trigger: one message, two payments.

Self-contained: `pip install google-adk` then `python issue_repro.py`. No network
access and no API key; nothing leaves 127.0.0.1.

- The agent is a stock LlmAgent on ADK's stock Gemini class. `base_url` points
  at a local fake Gemini endpoint.
- The fake endpoint answers the first request with a `pay_invoice` call. It
  answers the request after the payment with HTTP 503 UNAVAILABLE ("The model
  is overloaded"), once, and then behaves normally.
- The loop at the bottom plays Pub/Sub: after a non-2xx response it posts the
  same push envelope again, with the same messageId.
"""

import base64
import json
import os
import sys
import tempfile
import textwrap
import threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path

GEMINI = ("127.0.0.1", 8944)
PAYMENTS = []  # the payment provider's ledger: one entry per irreversible charge
_calls = {"n": 0}


class FakeGemini(BaseHTTPRequestHandler):
    def do_POST(self):  # noqa: N802
        body = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
        _calls["n"] += 1
        after_payment = "functionResponse" in body["contents"][-1]["parts"][-1]
        if after_payment and _calls["n"] == 2:
            return self._send(503, {"error": {"code": 503, "status": "UNAVAILABLE",
                                              "message": "The model is overloaded. Please try again later."}})
        part = ({"text": "Paid."} if after_payment else
                {"functionCall": {"name": "pay_invoice",
                                  "args": {"invoice": "INV-1", "amount": "100.00"}}})
        self._send(200, {"candidates": [{"content": {"role": "model", "parts": [part]},
                                         "finishReason": "STOP"}]})

    def _send(self, status, obj):
        data = json.dumps(obj).encode()
        self.send_response(status)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(data)))
        self.end_headers()
        self.wfile.write(data)

    def log_message(self, *args):
        pass


AGENT = f'''
import __main__
from google.adk.agents import LlmAgent
from google.adk.models import Gemini

def pay_invoice(invoice: str, amount: str) -> dict:
    """Pay an invoice. Irreversible."""
    __main__.PAYMENTS.append((invoice, amount))
    return {{"status": "PAID", "payment_number": len(__main__.PAYMENTS)}}

root_agent = LlmAgent(
    name="payer",
    model=Gemini(model="gemini-2.5-flash", base_url="http://{GEMINI[0]}:{GEMINI[1]}"),
    instruction="Pay the invoice in the message once.",
    tools=[pay_invoice],
)
'''


def main() -> int:
    os.environ["GOOGLE_API_KEY"] = "placeholder"  # never sent anywhere but the fake
    os.environ.pop("GOOGLE_GENAI_USE_VERTEXAI", None)
    server = ThreadingHTTPServer(GEMINI, FakeGemini)
    threading.Thread(target=server.serve_forever, daemon=True).start()

    agents = Path(tempfile.mkdtemp())
    (agents / "payer").mkdir()
    (agents / "payer" / "__init__.py").write_text("from . import agent\n")
    (agents / "payer" / "agent.py").write_text(textwrap.dedent(AGENT))

    import google.adk
    from fastapi.testclient import TestClient
    from google.adk.cli.fast_api import get_fast_api_app

    app = get_fast_api_app(agents_dir=str(agents), web=False, trigger_sources=["pubsub"])
    client = TestClient(app)
    envelope = {"message": {"data": base64.b64encode(b'{"invoice": "INV-1", "amount": "100.00"}').decode(),
                            "messageId": "1234567890"},
                "subscription": "projects/p/subscriptions/invoices"}

    print(f"google-adk {google.adk.__version__}")
    for delivery in range(1, 4):
        r = client.post("/apps/payer/trigger/pubsub", json=envelope)
        print(f"delivery {delivery} of messageId 1234567890: HTTP {r.status_code}")
        if r.status_code in (102, 200, 201, 202, 204):
            break  # acknowledged; otherwise Pub/Sub redelivers the same message
    print(f"payments made for one message: {len(PAYMENTS)}")
    server.shutdown()
    return 0 if len(PAYMENTS) > 1 else 1


if __name__ == "__main__":
    sys.exit(main())

How often has this issue occurred?:

Every time, under the conditions above. The reproduction is deterministic.

[surajksharma07]

@FurkhanShaikh. Reproduced on main with your script (500 then 200, two payments) and it's the same on every release since the trigger endpoints landed in v1.29.0.

Until #7325 is in a workaround that worked: a small ASGI middleware copies subscription:messageId into message.attributes, a before_agent_callback puts it in state and the tool passes it to the provider as an idempotency key. Two deliveries, one charge and a new messageId still gets charged. Could you try that in your harness especially the ack-deadline case?

@devansh173 #7325 looks good. One thing for the docstring: state is new on every delivery so the dedupe has to happen at the provider or in an external store not in state. Since messageId is only unique per topic maybe also suggest including the subscription in the key.

Please re-run the trigger tests and your E2E once more after that before asking for a review. The docs part (ack deadline, dead-letter policy) would be great as a follow-up in adk-docs.

[devansh173]

Thanks for reproducing it and for the review. I pushed the changes to #7325:

• The TRIGGER_DELIVERY_STATE_KEY docstring now says session state is fresh on every delivery, so deduplication has to happen at the provider or in an external store. It recommends keying on subscription + id for Pub/Sub and event_source + id for Eventarc.
• The redelivery unit test now keys on subscription + messageId in a ledger outside the session.

Re-ran everything: all 78 trigger tests pass. In the E2E, the idempotent tool pays twice for one messageId on main and once on this branch, and a new messageId is still charged. The logs are in the PR description.

I'll follow up with an adk-docs PR covering the ack deadline, dead-letter policy and idempotent tools.

[surajksharma07]

0db8ecb reads well @devansh173. The docstring now says clearly where the dedupe has to happen and how to build the key. Re-checked: the 78 trigger tests pass and an idempotent tool keyed on trigger_delivery (subscription + id, ledger outside the session) pays once across the 500/200 redelivery while a new messageId is still charged. The same tool on main pays twice.

From my side the PR is ready for maintainer review. Will ask for the workflow approvals. Please link the adk-docs PR here once it's up.

@FurkhanShaikh whenever you get a chance it would be good to hear whether this (or the middleware workaround on current releases) holds up in your harness especially the ack-deadline case where the first run is still in flight.

Option 2 (deterministic session id) can go to a separate follow-up issue if maintainers want it so it doesn't hold this one up.

[devansh173]

The adk-docs follow-up is up: google/adk-docs#2274. It covers the ack deadline, dead-letter topic and idempotent tools using trigger_delivery, and asks to hold merging until #7325 is released.

[surajksharma07]

adk-docs#2274 reads well @devansh173 and matches #7325 (fields, key format and the "dedupe outside session state" point). Would take up your offer to split out the ack-deadline + dead-letter part. It applies to every release since v1.29.0 today while trigger_delivery waits on #7325 (still at 0db8ecb, not merged, main unchanged).

Two small additions while you're there. The ack deadline only helps if the Cloud Run request timeout is at least as long: the default is 300s so a 10-minute run gets cut at 5 min and redelivered anyway. Worth adding gcloud run services update --timeout=600 next to the ack-deadline step. For the dead-letter step the Pub/Sub service agent needs publisher on the dead-letter topic and subscriber on the subscription or nothing gets forwarded.

Might also be worth a line that for Eventarc the same settings go on the trigger's underlying Pub/Sub subscription. Please re-run mkdocs build --strict after the changes.

@FurkhanShaikh still keen to hear how the workaround or #7325 does in your harness especially the ack-deadline case.

[devansh173]

Thanks! I split the ack-deadline and dead-letter part out as google/adk-docs#2275, which applies to current releases. While there I added the Cloud Run --timeout=600 step next to the ack deadline, the service agent's Publisher and Subscriber roles for the dead-letter topic, and a note for Eventarc Standard with a command to find the trigger's subscription. google/adk-docs#2274 now builds on it and only adds the trigger_delivery part, which waits on #7325. mkdocs build --strict passes on both with no warnings.

[surajksharma07]

The split is exactly what I had in mind @devansh173. adk-docs#2275 covers the Cloud Run timeout, the service agent roles and the Eventarc subscription lookup and #2274 now only adds the trigger_delivery part on top of it.

One thing I noticed in the raw file (ecf96ea, also in b517386): the multi-line gcloud commands lost their trailing \ continuations so the dead-letter update, the two add-iam-policy-binding commands and the eventarc describe each render as one long line with runs of spaces. They still work when pasted which is probably why mkdocs --strict stays quiet but restoring the \ line breaks would match the rest of the page.

Once that's fixed #2275 is ready for docs review from my side and can go out independently. #2274 waits for #7325 (still 0db8ecb, not merged; main unchanged so the redelivery issue still applies from v1.29.0 through v2.10.0).

@FurkhanShaikh the offer still stands to try the workaround or #7325 in your harness whenever you get a chance.

[devansh173]

Thanks for spotting this, @surajksharma07. The trailing \ continuations got lost when I split the section. Fixed in google/adk-docs#2275 (ba0c416): the dead-letter update, both add-iam-policy-binding commands and the Eventarc describe are split across lines again, matching the rest of the page. google/adk-docs#2274 is rebased on top of it. I re-ran mkdocs build --strict on both with no warnings, and checked the raw file: all 8 continuation lines end in \ again.

[surajksharma07]

Checked ba0c416 @devansh173. All 8 continuations are back and no gcloud command is left on one long line in either #2275 or #2274. #2274 is cleanly on top of it and its trigger_delivery table and example still match #7325.

With that adk-docs#2275 is good to go from my side. Will ask for a docs reviewer and the workflow approvals so it can ship independently for current releases.

On the code side #7325 (0db8ecb) still merges cleanly onto today's main (e4c0d94) and the 78 trigger tests pass on the merged tree. Just waiting on a maintainer review now, nothing more needed from you there.

#2274 stays on hold until #7325 is released. Until then the redelivery behaviour applies to v1.29.0 through v2.10.0.

[FurkhanShaikh]

Thanks both, and sorry for the delay.

I ran #7325 (0db8ecb) and the middleware workaround (on 2.10.0)
through the harness, including the ack-deadline case.

Short version: the key works on both paths, and it prevents the duplicate
when the dedupe is atomic at the effect.

A provider idempotency key held in every case, including a real overlap.
One common reading of "check it against an external store" still pays twice.

Setup.

• Stock LlmAgent on the stock Gemini class, pointed at a local fake Gemini
  endpoint.
• A local Pub/Sub push emulator. In the two ack-deadline rows the deadline was
  scaled to 4 s (60 s elsewhere), and there was a 2 s wait before resending
  after a nack.
• A local payment provider with Stripe-style idempotency: the first request
  with a key charges; a concurrent request with the same key gets 409; a later
  one gets the stored reply.
• Nothing in ADK was patched.

The tool builds f"{subscription}:{id}" from trigger_delivery (#7325), or
from the attribute the middleware adds (workaround). It then dedupes one of
three ways:

• provider_key: passes the key as the provider's idempotency key;
• check_then_act: checks an external store, pays, then records the key;
• reserve: atomically inserts the key into the store before paying, and
  keeps it on failure.

Charges, identical on #7325 and on the workaround:

Scenario	no dedupe	provider_key	check_then_act	reserve
model 503 after the payment (500 → 200)	2	1	1	1
provider commits, reply exceeds the tool timeout	2	1	2	1
ack deadline passes while payment 1 is in flight, redelivery on a 2nd instance	2	1	2	1
same, one instance with an async tool	2	1	2	1
ack deadline passes before the first run reaches the tool	2	1	1	1
provider fails before committing (503)	1	1	1	0
two different messages	2	2	2	2

In the overlap cases with provider_key, the provider saw charged, then
409 for each redelivery that arrived while payment 1 was in flight. The tool
raised on the 409, so the run failed and the message was nacked; a later
delivery got the original reply replayed, and was acked with one charge.

Things that might be worth a line in the #7325 docstring or adk-docs#2274:

1. Prefer the provider's idempotency key. It was the only strategy that
   held in every row, because the provider is the one place that knows
   whether the effect happened.
2. "Check an external store" needs to be atomic, and to happen before the
   effect. Check → act → record pays twice after an ambiguous failure (a
   timeout after the provider committed), and again when a redelivery
   overlaps a run still in flight.
3. A reservation taken before acting avoids the duplicate, but turns a
   failure before commit into a skipped effect. In the last-but-one row the
   message was acked and nothing was paid. So a store-based dedupe needs a way
   to reconcile an ambiguous attempt, not just a flag.
4. Let a 409 (key in use) fail the run rather than returning it to the
   model as a tool result. That is what our tool did; we didn't test the
   alternative. The reasoning: the nack keeps the message alive until a
   delivery sees the original result, whereas acking on a 409 would lose the
   payment if the in-flight attempt then failed before committing.

One side observation on the ack-deadline case. With a synchronous tool,
the trigger path (runner.run_async) calls the tool inline on the event loop
(FunctionTool._invoke_callable, 2.10.0 lines 457–460).
RunConfig.tool_thread_pool_config doesn't change this: it is only read in
the live path (_process_function_live_helper). While the first payment's
reply was held, that instance could not start the redelivered run at all:

• the redeliveries queued;
• further deadlines expired;
• they then ran back to back (3 charges with no dedupe);
• so on one instance with a sync tool, the runs never overlapped in our tests.

It does happen across two instances, or with an async tool, which is why
those two rows are there. Probably a separate topic, but it seemed relevant
to the ack-deadline advice in adk-docs#2275: one slow sync tool call can stall
every trigger request on that instance.

Limits:

• The provider is a local model of the documented idempotency behaviour, not
  a real payment API. A provider whose keys don't cover concurrent in-flight
  requests would behave like check_then_act in the overlap rows.
• Pub/Sub is emulated, with scaled timings.
• Each message asked for one payment, and our fake model sent the same
  arguments on every delivery, so the key was subscription:id alone.
  adk-docs#2274's example also appends invoice_id, which a message with
  several payments needs. We didn't test a model that changes that argument
  between deliveries; if it did, the key would change with it.