SPARK-34519 ExecutorPodsAllocator applies exponential backoff delays …

[fe2s]

…for executor pod requests when pods repeatedly fail to start

What changes were proposed in this pull request?

Introduces exponential backoff delays for executor pod requests when pods repeatedly fail to start. It tracks the following startup failures:

• API request failures - Pod creation requests to the Kubernetes API server throw exceptions
• Pod startup failures - Pods transition to PodFailed status before the executor registers with the driver (indicating the executor never successfully started)
• Creation timeouts - Pods do not appear within the timeout period (existing mechanism)

Operates as a state machine with two states:

• Normal state - Executor pods are requested without extra delay. The startup failures are tracked within a sliding time window. When the failure count exceeds the configured threshold, it transitions to Backoff state.
• Backoff state - Executor pod requests are throttled with exponentially increasing delays between requests. It immediately transitions back to Normal state when an executor that was requested during the Backoff state successfully starts, indicating the infrastructure issue has resolved.

When backoff is enabled, two new metrics added. Will update monitoring.md doc with new source if patch looks good.

Why are the changes needed?

When executor pods repeatedly fail to start due to Kubernetes infrastructure issues (control plane overload, resource exhaustion, service mesh issues), the current implementation continues requesting pods at full speed, amplifying load on already stressed infrastructure.

Relationship to ExecutorFailureTracker:
This backoff mechanism is complementary to the existing ExecutorFailureTracker. While ExecutorFailureTracker counts all Pod failures (including those that started successfully but failed during task execution) to determine when to abort the application (spark.executor.maxNumFailures), the backoff controller specifically tracks startup failures only to throttle allocation requests and protect infrastructure.

This is especially useful in the following scenarios:
(a) Long-running applications that process heterogeneous workloads, where spark.executor.maxNumFailures is set to Int.MAX_VALUE (or another high value). In such cases Spark will make many executor startup attempts before the application is shut down.
(b) Situations involving transient Kubernetes infrastructure issues (e.g with Istio) when the number of executors needs to scale up (e.g., due to dynamic allocation). In these cases it is preferable to wait for the issue to be resolved while continuing to run with the existing executors, rather than shutting down the application.

Does this PR introduce any user-facing change?

When enabled: executor pod allocation is throttled with exponential delays when startup failures exceed the threshold.
Observability changes when enabled:

• New log messages indicating backoff state transitions and current delays
• New metrics.

How was this patch tested?

1. Unit tests
2. API request failures simulated with kubectl create quota test --hard=cpu=1,memory=1G
3. Pod startup failures simulated with an init container that fails to start.

Was this patch authored or co-authored using generative AI tooling?

Cursor 2.2.9

[fe2s]

@dongjoon-hyun
Could you take a look when you have a chance? Thanks!

[github-actions]

JIRA Issue Information

=== Improvement SPARK-34519 ===
Summary: ExecutorPodsAllocator use exponential backoff strategy when request executor pod failed
Assignee: None
Status: Open
Affected: ["3.0.1"]

---

This comment was automatically generated by GitHub Actions