CNTRLPLANE-2549:test/library/controlplane: add WaitForControlPlaneRolloutAll helper

[wangke19]

Summary

Add a reusable test helper package test/library/controlplane for waiting on control-plane static-pod operator revision rollouts after disruptive cluster changes (e.g. CA rotation, cert regeneration).

Motivation

After a disruptive CA rotation, the three control-plane static-pod operators (kube-apiserver, kube-controller-manager, kube-scheduler) each trigger a node-by-node revision rollout. Tests that need to wait for the cluster to stabilize before asserting monitor health had no reusable utility for this in library-go.

New API: test/library/controlplane

// One-call shortcut: wait for all three operators sequentially.
WaitForControlPlaneRolloutAll(ctx, t, cfgClient, opClient) error

// Single-operator variant.
WaitForControlPlaneRollout(ctx, t, cfgClient, opClient, op) error

// Wait for a single ClusterOperator to reach Available=True, Progressing=False, Degraded=False.
WaitForClusterOperatorStable(ctx, t, cfgClient, name) error

Design

• Reads LatestAvailableRevision and nodeStatuses from the operator/v1 resource (KubeAPIServer, KubeControllerManager, KubeScheduler) — the canonical source for static pod rollout progress on OCP 4.x (no node annotations required).
• Re-reads LatestAvailableRevision each poll interval so mid-rollout re-revisions are chased automatically.
• Logs only on state transitions (not every poll tick) to keep test output readable.
• Accepts context.Context for cancellation; callers control the deadline.
• Uses library.LoggingT interface (compatible with both *testing.T and Ginkgo's GinkgoTB()).

Usage

ctx, cancel := context.WithTimeout(context.Background(), 15*time.Minute)
defer cancel()
if err := controlplane.WaitForControlPlaneRolloutAll(ctx, t, cfgClient, opClient); err != nil {
    t.Logf("WARNING: control plane did not stabilize: %v", err)
}

Test plan

• Validated live against OCP 4.21 nightly cluster (GCP, 3-master) as part of service-ca-operator's refresh-CA e2e test
• go vet ./test/library/controlplane/... passes

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: wangke19
Once this PR has been reviewed and has the lgtm label, please assign bertinatto for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

@wangke19: This pull request references CNTRLPLANE-2549 which is a valid jira issue.

Details

In response to this:

Summary

Add a reusable test helper package test/library/controlplane for waiting on control-plane static-pod operator revision rollouts after disruptive cluster changes (e.g. CA rotation, cert regeneration).

Motivation

After a disruptive CA rotation, the three control-plane static-pod operators (kube-apiserver, kube-controller-manager, kube-scheduler) each trigger a node-by-node revision rollout. Tests that need to wait for the cluster to stabilize before asserting monitor health had no reusable utility for this in library-go.

New API: test/library/controlplane

// One-call shortcut: wait for all three operators sequentially.
WaitForControlPlaneRolloutAll(ctx, t, cfgClient, opClient) error

// Single-operator variant.
WaitForControlPlaneRollout(ctx, t, cfgClient, opClient, op) error

// Wait for a single ClusterOperator to reach Available=True, Progressing=False, Degraded=False.
WaitForClusterOperatorStable(ctx, t, cfgClient, name) error

Design

• Reads LatestAvailableRevision and nodeStatuses from the operator/v1 resource (KubeAPIServer, KubeControllerManager, KubeScheduler) — the canonical source for static pod rollout progress on OCP 4.x (no node annotations required).
• Re-reads LatestAvailableRevision each poll interval so mid-rollout re-revisions are chased automatically.
• Logs only on state transitions (not every poll tick) to keep test output readable.
• Accepts context.Context for cancellation; callers control the deadline.
• Uses library.LoggingT interface (compatible with both *testing.T and Ginkgo's GinkgoTB()).

Usage

ctx, cancel := context.WithTimeout(context.Background(), 15*time.Minute)
defer cancel()
if err := controlplane.WaitForControlPlaneRolloutAll(ctx, t, cfgClient, opClient); err != nil {
   t.Logf("WARNING: control plane did not stabilize: %v", err)
}

Test plan

• Validated live against OCP 4.21 nightly cluster (GCP, 3-master) as part of service-ca-operator's refresh-CA e2e test
• go vet ./test/library/controlplane/... passes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

@wangke19: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.