Skip to content

Fix createOrUpdate conflict loop and no-op update storm - #76

Open
jpinsonneau wants to merge 1 commit into
openshift:mainfrom
jpinsonneau:fix/createorupdate-conflict-retry
Open

Fix createOrUpdate conflict loop and no-op update storm#76
jpinsonneau wants to merge 1 commit into
openshift:mainfrom
jpinsonneau:fix/createorupdate-conflict-retry

Conversation

@jpinsonneau

@jpinsonneau jpinsonneau commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Summary

While testing netobserv with bgp-cloud-connector, I noticed the operator was stuck in an infinite reconcile loop.

  • Retry on conflict: Wrap the Update call in retry.RetryOnConflict so transient optimistic concurrency conflicts (e.g. when OVN-K's CUDN controller modifies the same object concurrently) are retried automatically instead of causing the reconciler to mark itself Degraded and trigger a reconcile storm.
  • Skip no-op updates: Add specUnchanged() to compare spec and labels before issuing an Update. When nothing changed, the write is skipped entirely, preventing the infinite reconcile loop caused by unconditional Updates bumping resourceVersion and re-triggering watches.

Affects all resources managed through createOrUpdate: ClusterUserDefinedNetwork, FRRConfiguration, and RouteAdvertisements.

Test plan

  • go build ./... passes
  • go test ./internal/controller/... -count=1 -short passes
  • Deployed on live OCP cluster with CUDNBgpRouting/CUDNBgpConfig
  • Verified no more conflict errors in operator logs
  • Verified reconcile loop settles after initial pass (no more rapid-fire reconciliations)
  • Verified spec changes (e.g. localASN update) trigger a single clean reconcile

Made with Cursor

The createOrUpdate helper performed a naive Get→Update without
retry-on-conflict and without checking whether the spec actually
changed. This caused two problems on live clusters:

1. Optimistic concurrency conflicts: when another controller (e.g.
   OVN-K CUDN controller) modified the same object between Get and
   Update, the resourceVersion mismatch caused a conflict error. The
   reconciler marked itself Degraded, which triggered a new reconcile,
   creating a self-sustaining Degraded/Ready flap storm.

2. No-op update loop: even when spec and labels were identical, the
   unconditional Update bumped resourceVersion, which triggered watches
   on CUDN/RouteAdvertisements/FRRConfiguration objects, which triggered
   new reconciles in an infinite loop.

Fix by:
- Wrapping the Update in retry.RetryOnConflict so transient conflicts
  are retried automatically with exponential backoff.
- Adding specUnchanged() to compare spec and labels before updating,
  skipping the write entirely when nothing changed.

Validated on a live OCP cluster with CUDNBgpRouting and CUDNBgpConfig.

Co-authored-by: Cursor <cursoragent@cursor.com>
@coderabbitai

coderabbitai Bot commented Aug 24, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: dd1dd1b9-f966-4244-a533-6adb08b55a09

📥 Commits

Reviewing files that changed from the base of the PR and between 25a6f39 and 58b5e99.

⛔ Files ignored due to path filters (3)
  • vendor/k8s.io/client-go/util/retry/OWNERS is excluded by !**/vendor/**, !vendor/**
  • vendor/k8s.io/client-go/util/retry/util.go is excluded by !**/vendor/**, !vendor/**
  • vendor/modules.txt is excluded by !**/vendor/**, !vendor/**
📒 Files selected for processing (1)
  • internal/controller/frr.go

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.


📝 Walkthrough

Walkthrough

The controller now compares existing and desired FRR resource specs and labels using semantic equality. It skips updates when no relevant values changed. When an update is required, it refreshes the resource version and retries after conflicts by rereading the current object.

Suggested reviewers: alebedev87, frobware

🚥 Pre-merge checks | ✅ 15
✅ Passed checks (15 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed Docstring check was indeterminate for this PR — some files could not be analyzed in time. Not blocking.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The PR changes only controller and vendored retry files; it adds no test files or Ginkgo title expressions, so it introduces no unstable test names.
Test Structure And Quality ✅ Passed The PR changes only production code and vendored retry code; it adds or modifies no Ginkgo test, so the stated test-structure requirements are not applicable.
Microshift Test Compatibility ✅ Passed The pull request changes only controller and vendored retry files; it adds no Ginkgo e2e tests or new It, Describe, Context, or When declarations.
Single Node Openshift (Sno) Test Compatibility ✅ Passed The PR changes internal/controller/frr.go and vendored retry code only; the diff adds no Ginkgo e2e tests or multi-node assumptions.
Topology-Aware Scheduling Compatibility ✅ Passed The diff changes only FRR resource update/retry logic and vendored retry code; it adds no affinity, topology spread, replica, PDB, taint, or node-targeting constraint.
Ote Binary Stdout Contract ✅ Passed The PR changes only controller comparison/retry logic and adds vendored retry code; no process-level stdout writes, klog setup, or suite-output changes were introduced.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed The PR changes only controller and vendored retry code. It adds no Ginkgo e2e tests, IPv4 assumptions, or external connectivity requirements.
No-Weak-Crypto ✅ Passed The PR changes only FRR reconciliation and vendored retry code; the diff introduces no MD5, SHA-1, DES/3DES, RC4, Blowfish, ECB, custom crypto, or secret comparisons.
Container-Privileges ✅ Passed The PR changes only Go and vendor files. Its patch adds no privilege settings; existing workloads use runAsNonRoot and allowPrivilegeEscalation: false.
No-Sensitive-Data-In-Logs ✅ Passed The PR diff adds no logging calls or sensitive values; frr.go adds comparison/retry logic, and vendored retry code does not log.
Title check ✅ Passed The title clearly summarizes the fixes for conflict handling and no-op update loops.
Description check ✅ Passed The description directly explains the conflict retry and no-op update changes, affected resources, and validation steps.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@openshift-ci
openshift-ci Bot requested review from daxelrod-rh and frobware August 24, 2026 13:24
@openshift-ci

openshift-ci Bot commented Aug 24, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: jpinsonneau
Once this PR has been reviewed and has the lgtm label, please assign frobware for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 27, 2026
@openshift-ci

openshift-ci Bot commented Aug 27, 2026

Copy link
Copy Markdown

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci

openshift-ci Bot commented Sep 2, 2026

Copy link
Copy Markdown

@jpinsonneau: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/ci-bundle-bgp-cloud-connector-bundle 58b5e99 link true /test ci-bundle-bgp-cloud-connector-bundle
ci/prow/e2e-aws-operator 58b5e99 link true /test e2e-aws-operator

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant