Fix nondeterminism caused by workflow error reporting

[Kamefrede]

Description

Workflow error and panic reporting ran a local activity, guarded by workflow.IsReplaying:

if workflow.IsReplaying(ctx) {
    return
}
...
_ = workflow.ExecuteLocalActivity(disconnectedCtx, s.sentryActivities.ReportError, input).Get(disconnectedCtx, nil)

IsReplaying may only guard side effects that produce no commands (logging, metrics). ExecuteLocalActivity produces one: it writes a MarkerRecorded event into workflow history. Guarding it means the command is emitted on the original execution and never again — from the first replay onward the replayed command stream no longer matches history and every workflow task fails with:

[TMPRL1100] nondeterministic workflow: history event is MarkerRecorded:
(MarkerName:LocalActivity, ...), replay command is ProtocolMessage: (...)

There is no recovery: the marker is in history for good, replay always starts from the first event, and the server retries forever. Any non-nil handler error is enough to poison a run — ExecuteWorkflow, HandleSignal, HandleQuery, ValidateUpdate and ExecuteUpdate all route through this path. This is the same defect just root-caused in a wedged production wallet-service workflow (uphold/wallet-service#465), whose interceptor is a sibling of this library; the failure was reproduced from the recorded production history with worker.WorkflowReplayer, confirmed against SDK v1.35.0 and server v1.29.6 source, and reproduced end-to-end with a minimal synthetic workflow.

Fix

Report inline from the workflow goroutine — CaptureException/RecoverWithContext only enqueue on Sentry's async transport, so no command is emitted and the workflow task never blocks.

The IsReplaying guard now does the one thing it is valid for — suppressing duplicate reports during replay — and only on the handlers that actually re-execute from history (ExecuteWorkflow, HandleSignal, ExecuteUpdate). Queries are never re-delivered by replay, yet a cold worker that just rebuilt state from history serves them with IsReplaying() == true, so a blanket guard silently drops those errors; update validators never run during replay at all. Both now capture unconditionally.

Two more defects fixed along the way:

• Filtered panics were swallowed: the deferred recover() returned without re-panicking when filterWorkflowPanic matched, letting the workflow continue as if the panic never happened. The recover now always re-raises.
• Queries could not use the old path at all: query handlers run outside the dispatcher, where ExecuteLocalActivity panics.

Breaking change

WithWorkflowErrorActivityOptions and WithWorkflowPanicActivityOptions are removed — they configured the local activities that no longer exist. The README section documenting them is gone, and the transport note now reflects the real constraint: Sentry must use its (default) async transport, since a synchronous transport would block the workflow goroutine and trip the SDK's 1s deadlock detector.

workflowcheck now needs workflowcheck.config.yaml: the capture helpers are non-deterministic code by design, but emit no commands, which is what determinism actually requires. The Makefile passes the config.

How has this been tested

• make format, make lint (0 issues), make lint-workflows, make test — all green.
• New regression tests in workflow_test.go assert that error and panic reporting fire without executing any local activity (via SetOnLocalActivityStartedListener), that panics — filtered or not — still propagate, and that query-handler errors are reported. All three reporting tests fail against the previous implementation.
• The nondeterminism itself was reproduced outside this repo: replaying a real poisoned production history fails with the exact TMPRL1100 above, and a minimal dev-server scenario (update handler error → worker restart → replay) wedges with the old pattern and completes cleanly with this one.

Deploy notes

Fixes new runs only. Any run that already recorded a reporting marker still has it in history and will keep failing replay; those runs need a terminate (a new run can start under ALLOW_DUPLICATE_FAILED_ONLY) or a reset to a workflow task before the marker.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Fixes a determinism bug in workflow error/panic reporting by removing local-activity based reporting (which records a MarkerRecorded event) and switching to inline Sentry capture that emits no workflow commands, preventing permanent TMPRL1100 nondeterminism on replay.

Changes:

• Remove workflow local activities for reporting; report inline from the workflow goroutine with replay-aware suppression.
• Fix workflow panic handling so filtered panics are still re-raised, and query/validator reporting paths don’t get incorrectly suppressed.
• Add workflowcheck configuration + regression tests ensuring reporting emits no local-activity commands.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
workflow.go	Replaces local-activity reporting with inline Sentry capture; adjusts replay suppression semantics; fixes panic propagation behavior.
workflow_test.go	Adds regression tests verifying no local activity is executed for reporting, panics always propagate, and query errors are reported.
workflowcheck.config.yaml	Configures workflowcheck to allow the capture helpers (non-deterministic code that emits no commands).
Makefile	Passes the workflowcheck config to lint-workflows.
options.go	Removes workflow local-activity option fields and option setters that no longer apply.
interceptor.go	Updates top-level documentation to reflect inline workflow reporting constraints.
interceptor_test.go	Removes tests and setup for the removed workflow activity-options API.
README.md	Removes documentation for removed local-activity options and updates Sentry transport guidance.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.