Skip to content

[release-4.21] OCPBUGS-90543: fix etcd operator deadlock when etcd-endpoints configmap is stale#1637

Open
openshift-cherrypick-robot wants to merge 1 commit into
openshift:release-4.21from
openshift-cherrypick-robot:cherry-pick-1631-to-release-4.21
Open

[release-4.21] OCPBUGS-90543: fix etcd operator deadlock when etcd-endpoints configmap is stale#1637
openshift-cherrypick-robot wants to merge 1 commit into
openshift:release-4.21from
openshift-cherrypick-robot:cherry-pick-1631-to-release-4.21

Conversation

@openshift-cherrypick-robot

Copy link
Copy Markdown

This is an automated cherry-pick of #1631

/assign dpateriya

…rList fails

When the etcd-endpoints configmap contains stale IPs (e.g. after VM
migration), the etcd client pool cannot reach any member, causing
MemberList to fail. This creates a deadlock: the operator cannot update
the configmap because MemberList is needed to get member addresses, but
MemberList fails because the configmap has stale addresses.

Break the deadlock directly in EtcdEndpointsController.syncConfigMap():
when MemberList fails, fall back to control-plane node internal IPs
discovered via the node lister and network config. This populates the
configmap with reachable IPs, allowing the etcd client to reconnect.
On the next successful sync, MemberList overwrites with authoritative
member data.

Also adds WithMemberListError option to FakeEtcdClient for testing.

Co-authored-by: Cursor <cursoragent@cursor.com>
@coderabbitai

coderabbitai Bot commented Jun 21, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: d8336313-3961-44a7-b4b0-932107b9d8e3

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci openshift-ci Bot requested review from hasbro17 and tjungblu June 21, 2026 10:53
@openshift-ci

openshift-ci Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign dusk125 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot

Copy link
Copy Markdown

@openshift-cherrypick-robot: Jira Issue OCPBUGS-88490 has been cloned as Jira Issue OCPBUGS-90543. Will retitle bug to link to clone.
/retitle [release-4.21] OCPBUGS-90543: fix etcd operator deadlock when etcd-endpoints configmap is stale

Details

In response to this:

This is an automated cherry-pick of #1631

/assign dpateriya

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci Bot changed the title [release-4.21] OCPBUGS-88490: fix etcd operator deadlock when etcd-endpoints configmap is stale [release-4.21] OCPBUGS-90543: fix etcd operator deadlock when etcd-endpoints configmap is stale Jun 21, 2026
@openshift-ci-robot openshift-ci-robot added jira/severity-moderate Referenced Jira bug's severity is moderate for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jun 21, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@openshift-cherrypick-robot: This pull request references Jira Issue OCPBUGS-90543, which is invalid:

  • release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
  • expected dependent Jira Issue OCPBUGS-88490 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is MODIFIED instead
  • expected dependent Jira Issue OCPBUGS-88490 to target a version in 4.22.0, but it targets "5.0" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

This is an automated cherry-pick of #1631

/assign dpateriya

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci

openshift-ci Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

@openshift-cherrypick-robot: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. jira/severity-moderate Referenced Jira bug's severity is moderate for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants