Skip to content

[All] Graceful close on arrow streams#258

Merged
elenagaljak-db merged 8 commits into
mainfrom
arrow-graceful-stream-close
May 12, 2026
Merged

[All] Graceful close on arrow streams#258
elenagaljak-db merged 8 commits into
mainfrom
arrow-graceful-stream-close

Conversation

@elenagaljak-db
Copy link
Copy Markdown
Contributor

What changes are proposed in this pull request?

  • Adds stream_paused_max_wait_time_ms config option to ArrowStreamConfigurationOptions across all SDKs (Rust, Go, Python, TypeScript, Java)
  • Implements client-side graceful close state machine: when the server signals it will close the stream, the SDK enters a paused state, stops sending new batches but drains in-flight ack responses before triggering recovery
  • Graceful close recoveries don't count against the recovery_retries budget, since they're intentional server-side restarts (e.g. rolling deploys)

How is this tested?

  • 5 new integration tests in rust/tests/src/arrow_tests.rs covering: full server duration wait, immediate recovery, client-capped wait, early exit when all batches acked, ingest accepted during grace period

@elenagaljak-db elenagaljak-db force-pushed the arrow-graceful-stream-close branch from e64589d to e5de6d8 Compare April 29, 2026 14:47
Comment thread rust/sdk/src/arrow_stream.rs
Comment thread java/src/main/java/com/databricks/zerobus/ArrowStreamConfigurationOptions.java Outdated
Comment thread typescript/src/lib.rs Outdated
Comment thread python/rust/src/arrow.rs
Comment thread rust/sdk/src/arrow_stream.rs Outdated
@elenagaljak-db elenagaljak-db force-pushed the arrow-graceful-stream-close branch 2 times, most recently from 6e47f0c to 0f50eb0 Compare May 8, 2026 14:20
Comment thread rust/sdk/src/arrow_stream.rs Outdated
Comment thread python/NEXT_CHANGELOG.md Outdated
Comment thread java/NEXT_CHANGELOG.md Outdated
Comment thread go/NEXT_CHANGELOG.md Outdated
Comment thread java/NEXT_CHANGELOG.md Outdated
Comment thread rust/sdk/src/arrow_stream.rs Outdated
Comment thread rust/sdk/src/arrow_stream.rs
Comment thread rust/sdk/src/arrow_stream.rs Outdated
Signed-off-by: elenagaljak-db <elena.galjak@databricks.com>
Signed-off-by: elenagaljak-db <elena.galjak@databricks.com>
Signed-off-by: elenagaljak-db <elena.galjak@databricks.com>
Signed-off-by: elenagaljak-db <elena.galjak@databricks.com>
Signed-off-by: elenagaljak-db <elena.galjak@databricks.com>
Signed-off-by: elenagaljak-db <elena.galjak@databricks.com>
Signed-off-by: elenagaljak-db <elena.galjak@databricks.com>
@elenagaljak-db elenagaljak-db force-pushed the arrow-graceful-stream-close branch from da19d06 to 7d4b619 Compare May 12, 2026 09:58
@elenagaljak-db elenagaljak-db added this pull request to the merge queue May 12, 2026
Merged via the queue into main with commit 279924e May 12, 2026
55 checks passed
@elenagaljak-db elenagaljak-db deleted the arrow-graceful-stream-close branch May 12, 2026 10:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants