Skip to content

[Debug] Add wait step to baremetalds-two-node-arbiter-upgrade for OCP 4.20 - #84057

Closed
etmurasaki wants to merge 2 commits into
openshift:mainfrom
etmurasaki:debug-baremetalds-two-node-arbiter-upgrade-4.20-20260825
Closed

[Debug] Add wait step to baremetalds-two-node-arbiter-upgrade for OCP 4.20#84057
etmurasaki wants to merge 2 commits into
openshift:mainfrom
etmurasaki:debug-baremetalds-two-node-arbiter-upgrade-4.20-20260825

Conversation

@etmurasaki

@etmurasaki etmurasaki commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds a wait step to enable debugging of test failures in OCP 4.20
  • Workflow: baremetalds-two-node-arbiter-upgrade
  • Timeout: 24h

The wait step pauses the workflow before tests run, allowing QE to:

  • SSH into the test environment
  • Inspect system state and logs
  • Debug configuration issues
  • Investigate test failures

⚠️ DO NOT MERGE — close this PR after debugging is complete.

Summary by CodeRabbit

  • Adds a best-effort 24-hour wait before tests in the OCP 4.20 baremetalds-two-node-arbiter-upgrade workflow. This gives QE time to inspect the environment, logs, configuration, and test failures.
  • Changes monitoring plugin UI test defaults to disable automatic COO installation and use a pinned COO FBC image.
  • The wait supports temporary debugging and should be removed after the investigation.

… 4.20

This adds a wait step to enable debugging of test failures in OCP 4.20.

The wait step pauses the workflow before tests run, allowing QE to:
- SSH into the test environment
- Inspect system state and logs
- Debug configuration issues
- Investigate test failures

OCP Version: 4.20
Workflow: baremetalds-two-node-arbiter-upgrade
Timeout: 24h
@coderabbitai

coderabbitai Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 78da3219-ffe1-43dd-a74c-7c4c887d0d06

📥 Commits

Reviewing files that changed from the base of the PR and between 061a797 and c538da0.

📒 Files selected for processing (3)
  • ci-operator/step-registry/baremetalds/two-node/arbiter/upgrade/baremetalds-two-node-arbiter-upgrade-workflow.yaml
  • ci-operator/step-registry/monitoring-plugin/tests/dashboards-ui/monitoring-plugin-tests-dashboards-ui-ref.yaml
  • ci-operator/step-registry/monitoring-plugin/tests/incidents-ui/monitoring-plugin-tests-incidents-ui-ref.yaml

Included review availability: Your plan provides up to 12 included reviews per hour; 9 remain after this review.


Walkthrough

The changes add a best-effort 24-hour wait before bare-metal IPI tests and update Cluster Observability Operator defaults for dashboard and incident UI tests.

Changes

Bare-metal upgrade workflow

Layer / File(s) Summary
Add pre-test wait
ci-operator/step-registry/baremetalds/two-node/arbiter/upgrade/...
The test phase adds a best-effort wait with a 24-hour timeout before baremetalds-ipi-test.

Monitoring plugin test defaults

Layer / File(s) Summary
Update COO test configuration
ci-operator/step-registry/monitoring-plugin/tests/dashboards-ui/..., ci-operator/step-registry/monitoring-plugin/tests/incidents-ui/...
CYPRESS_COO_UI_INSTALL now defaults to empty. CYPRESS_FBC_STAGE_COO_IMAGE now defaults to a pinned Cluster Observability Operator FBC image.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Merge Risk: ⚪ Minimal · up to c538d

This PR adds a temporary 24-hour debugging pause to one workflow and is explicitly marked for closure after debugging; no actionable merge-blocking risk is identified in the supplied evidence.

🚥 Pre-merge checks | ✅ 15
✅ Passed checks (15 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the primary change: adding a wait step to the bare-metal two-node arbiter upgrade workflow for OCP 4.20. The debug context is also accurate.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed PASS: The pull request changes only workflow/configuration YAML. The added wait step uses the static ref wait with a fixed 24h0m0s timeout, and the monitoring files change only environment defau…
Test Structure And Quality ✅ Passed PASS: The pull request changes only three CI YAML files. The diff adds a 24-hour wait step and changes Cypress environment defaults. It does not add or modify Ginkgo test code, It blocks, setup or…
Microshift Test Compatibility ✅ Passed PASS: The pull request changes only three YAML files. The diff from origin/main contains no Go files and adds no Ginkgo tests (It, Describe, Context, or When). The MicroShift compatibility che…
Single Node Openshift (Sno) Test Compatibility ✅ Passed PASS: The PR changes only three YAML files. The diff adds a 24-hour wait step and changes Cypress environment defaults. It adds no Go files, Ginkgo declarations, or new tests. The existing `baremetald…
Topology-Aware Scheduling Compatibility ✅ Passed PASS: The pull request changes only CI workflow and test-step reference YAML. It adds a generic 24-hour wait step and changes Cypress environment defaults. The diff adds no deployment manifests, ope…
Ote Binary Stdout Contract ✅ Passed PASS — The pull request changes only three YAML files. It adds a shell-based wait step and changes Cypress environment defaults. It adds no OTE binary code and no process-level Go code such as `main…
Ipv6 And Disconnected Network Test Compatibility ✅ Passed PASS: The pull request adds no Ginkgo tests. The diff changes only three YAML files, adds a wait step, and updates Cypress step defaults. The existing IP_STACK=v4 setting is unchanged. Therefore, th…
No-Weak-Crypto ✅ Passed PASS. The PR adds a wait step and changes two environment defaults. The only digest is SHA-256, not SHA-1. The added lines contain no MD5, SHA-1, DES, RC4, 3DES, Blowfish, ECB, custom crypto, or secre…
Container-Privileges ✅ Passed PASS. The pull request adds the existing wait step with a 24-hour timeout and changes two environment defaults. The referenced ci-operator/step-registry/wait/wait-ref.yaml uses from: cli, comman…
No-Sensitive-Data-In-Logs ✅ Passed PASS. The PR adds only a 24-hour wait step and changes two non-secret image/install defaults. The wait implementation logs pull-request metadata, timestamps, and a temporary file path, not passwords…
Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (3 skipped: 3 unsupported.)

Full details: Stable And Deterministic Test Names

Explanation

PASS: The pull request changes only workflow/configuration YAML. The added wait step uses the static ref wait with a fixed 24h0m0s timeout, and the monitoring files change only environment defaults. No Ginkgo title declarations or dynamic test names are introduced.

Full details: Test Structure And Quality

Explanation

PASS: The pull request changes only three CI YAML files. The diff adds a 24-hour wait step and changes Cypress environment defaults. It does not add or modify Ginkgo test code, It blocks, setup or cleanup hooks, cluster waits, or assertions. Therefore, the stated Test Structure and Quality requirements are not applicable.

Full details: Microshift Test Compatibility

Explanation

PASS: The pull request changes only three YAML files. The diff from origin/main contains no Go files and adds no Ginkgo tests (It, Describe, Context, or When). The MicroShift compatibility check is therefore not applicable.

Full details: Single Node Openshift (Sno) Test Compatibility

Explanation

PASS: The PR changes only three YAML files. The diff adds a 24-hour wait step and changes Cypress environment defaults. It adds no Go files, Ginkgo declarations, or new tests. The existing baremetalds-ipi-test chain remains unchanged, so this check has no applicable SNO compatibility failure.

Full details: Topology-Aware Scheduling Compatibility

Explanation

PASS: The pull request changes only CI workflow and test-step reference YAML. It adds a generic 24-hour wait step and changes Cypress environment defaults. The diff adds no deployment manifests, operator/controller code, or scheduling constraints such as affinity, topology spread, replica, node selector, toleration, or PDB settings. The existing TNA resource configuration is unchanged, so no explicit topology-compatibility failure condition is introduced.

Full details: Ote Binary Stdout Contract

Explanation

PASS — The pull request changes only three YAML files. It adds a shell-based wait step and changes Cypress environment defaults. It adds no OTE binary code and no process-level Go code such as main(), TestMain(), BeforeSuite(), RunSpecs(), klog, or fmt.Print*. The wait implementation writes diagnostic text from a shell script, which is outside this check's OTE stdout contract.

Full details: Ipv6 And Disconnected Network Test Compatibility

Explanation

PASS: The pull request adds no Ginkgo tests. The diff changes only three YAML files, adds a wait step, and updates Cypress step defaults. The existing IP_STACK=v4 setting is unchanged. Therefore, the custom check is not applicable to this pull request.

Full details: No-Weak-Crypto

Explanation

PASS. The PR adds a wait step and changes two environment defaults. The only digest is SHA-256, not SHA-1. The added lines contain no MD5, SHA-1, DES, RC4, 3DES, Blowfish, ECB, custom crypto, or secret/token comparisons. The referenced wait implementation only sleeps and checks a file.

Full details: Container-Privileges

Explanation

PASS. The pull request adds the existing wait step with a 24-hour timeout and changes two environment defaults. The referenced ci-operator/step-registry/wait/wait-ref.yaml uses from: cli, commands, and resource requests, but declares no privileged, host namespace, SYS_ADMIN, allowPrivilegeEscalation, or security context settings. The three changed manifests also contain none of these fields and do not set a root user.

Full details: No-Sensitive-Data-In-Logs

Explanation

PASS. The PR adds only a 24-hour wait step and changes two non-secret image/install defaults. The wait implementation logs pull-request metadata, timestamps, and a temporary file path, not passwords, tokens, API keys, PII, session IDs, internal hostnames, or customer data. The monitoring command scripts log the changed FBC image value, which is a public quay.io image reference, and do not log the kubeadmin password. No logging code changed.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@openshift-ci
openshift-ci Bot requested review from jgbernalp and kyoto August 25, 2026 23:35
@openshift-ci

openshift-ci Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: etmurasaki
Once this PR has been reviewed and has the lgtm label, please assign ggiguash for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

[REHEARSALNOTIFIER]
@etmurasaki: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-openshift-monitoring-plugin-main-e2e-dashboards openshift/monitoring-plugin presubmit Registry content changed
pull-ci-openshift-monitoring-plugin-release-5.2-e2e-dashboards openshift/monitoring-plugin presubmit Registry content changed
pull-ci-openshift-monitoring-plugin-release-5.1-e2e-dashboards openshift/monitoring-plugin presubmit Registry content changed
pull-ci-openshift-monitoring-plugin-release-5.0-e2e-dashboards openshift/monitoring-plugin presubmit Registry content changed
pull-ci-openshift-monitoring-plugin-release-4.23-e2e-dashboards openshift/monitoring-plugin presubmit Registry content changed
pull-ci-openshift-monitoring-plugin-release-4.22-e2e-dashboards openshift/monitoring-plugin presubmit Registry content changed
pull-ci-openshift-monitoring-plugin-release-4.21-e2e-dashboards openshift/monitoring-plugin presubmit Registry content changed
pull-ci-openshift-monitoring-plugin-release-4.20-e2e-dashboards openshift/monitoring-plugin presubmit Registry content changed
pull-ci-openshift-monitoring-plugin-release-4.19-e2e-dashboards openshift/monitoring-plugin presubmit Registry content changed
pull-ci-openshift-monitoring-plugin-release-coo-ocp-4.19-e2e-dashboards openshift/monitoring-plugin presubmit Registry content changed
pull-ci-openshift-monitoring-plugin-release-coo-1.5-ocp-4.19-e2e-dashboards openshift/monitoring-plugin presubmit Registry content changed
pull-ci-openshift-monitoring-plugin-release-coo-1.5-ocp-4.22-e2e-dashboards openshift/monitoring-plugin presubmit Registry content changed
pull-ci-openshift-monitoring-plugin-release-coo-ocp-4.22-e2e-dashboards openshift/monitoring-plugin presubmit Registry content changed
pull-ci-openshift-monitoring-plugin-release-coo-0.5-e2e-dashboards openshift/monitoring-plugin presubmit Registry content changed
pull-ci-openshift-monitoring-plugin-main-e2e-incidents openshift/monitoring-plugin presubmit Registry content changed
pull-ci-openshift-monitoring-plugin-release-5.2-e2e-incidents openshift/monitoring-plugin presubmit Registry content changed
pull-ci-openshift-monitoring-plugin-release-5.1-e2e-incidents openshift/monitoring-plugin presubmit Registry content changed
pull-ci-openshift-monitoring-plugin-release-5.0-e2e-incidents openshift/monitoring-plugin presubmit Registry content changed
pull-ci-openshift-monitoring-plugin-release-4.23-e2e-incidents openshift/monitoring-plugin presubmit Registry content changed
pull-ci-openshift-monitoring-plugin-release-4.22-e2e-incidents openshift/monitoring-plugin presubmit Registry content changed
pull-ci-openshift-monitoring-plugin-release-4.21-e2e-incidents openshift/monitoring-plugin presubmit Registry content changed
pull-ci-openshift-monitoring-plugin-release-4.20-e2e-incidents openshift/monitoring-plugin presubmit Registry content changed
pull-ci-openshift-monitoring-plugin-release-4.19-e2e-incidents openshift/monitoring-plugin presubmit Registry content changed
pull-ci-openshift-monitoring-plugin-release-coo-ocp-4.19-e2e-incidents openshift/monitoring-plugin presubmit Registry content changed
pull-ci-openshift-monitoring-plugin-release-coo-1.5-ocp-4.19-e2e-incidents openshift/monitoring-plugin presubmit Registry content changed

A total of 42 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs.

A full list of affected jobs can be found here
Prior to this PR being merged, you will need to either run and acknowledge or opt to skip these rehearsals.

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@etmurasaki

Copy link
Copy Markdown
Contributor Author

/test periodic-ci-openshift-release-main-nightly-4.21-e2e-metal-ovn-two-node-arbiter-upgrade

@openshift-ci

openshift-ci Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

@etmurasaki: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test boskos-config
/test boskos-config-generation
/test check-gh-automation
/test check-gh-automation-tide
/test check-trigger-trusted-apps
/test ci-operator-config
/test ci-operator-config-metadata
/test ci-operator-registry
/test ci-secret-bootstrap-config-validation
/test ci-testgrid-allow-list
/test cluster-manifest-verifier
/test clusterimageset-validate
/test config
/test core-valid
/test generated-config
/test generated-dashboards
/test hyperfleet-risk-scorer-test
/test image-mirroring-config-validation
/test jira-lifecycle-config
/test jira-solver-eval-images
/test labels
/test openshift-image-mirror-mappings
/test ordered-prow-config
/test owners
/test pr-reminder-config
/test prow-config
/test prow-config-filenames
/test prow-config-semantics
/test pylint
/test release-config
/test release-controller-config
/test rover-groups-config-validation
/test secret-generator-config-valid
/test services-valid
/test stackrox-stackrox-stackrox-stackrox-check
/test step-registry-metadata
/test step-registry-shellcheck
/test sync-rover-groups
/test verified-config
/test yamllint

The following commands are available to trigger optional jobs:

/test check-cluster-profiles-config

Use /test all to run the following jobs that were automatically triggered:

pull-ci-openshift-release-main-ci-operator-config
pull-ci-openshift-release-main-ci-operator-registry
pull-ci-openshift-release-main-core-valid
pull-ci-openshift-release-main-owners
pull-ci-openshift-release-main-release-controller-config
pull-ci-openshift-release-main-step-registry-metadata
pull-ci-openshift-release-main-step-registry-shellcheck
pull-ci-openshift-release-openshift-image-mirror-mappings
pull-ci-openshift-release-yamllint
Details

In response to this:

/test periodic-ci-openshift-release-main-nightly-4.21-e2e-metal-ovn-two-node-arbiter-upgrade

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@etmurasaki

Copy link
Copy Markdown
Contributor Author

/test periodic-ci-openshift-release-main-nightly-4.20-e2e-metal-ovn-two-node-arbiter-upgrade

@openshift-ci

openshift-ci Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

@etmurasaki: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test boskos-config
/test boskos-config-generation
/test check-gh-automation
/test check-gh-automation-tide
/test check-trigger-trusted-apps
/test ci-operator-config
/test ci-operator-config-metadata
/test ci-operator-registry
/test ci-secret-bootstrap-config-validation
/test ci-testgrid-allow-list
/test cluster-manifest-verifier
/test clusterimageset-validate
/test config
/test core-valid
/test generated-config
/test generated-dashboards
/test hyperfleet-risk-scorer-test
/test image-mirroring-config-validation
/test jira-lifecycle-config
/test jira-solver-eval-images
/test labels
/test openshift-image-mirror-mappings
/test ordered-prow-config
/test owners
/test pr-reminder-config
/test prow-config
/test prow-config-filenames
/test prow-config-semantics
/test pylint
/test release-config
/test release-controller-config
/test rover-groups-config-validation
/test secret-generator-config-valid
/test services-valid
/test stackrox-stackrox-stackrox-stackrox-check
/test step-registry-metadata
/test step-registry-shellcheck
/test sync-rover-groups
/test verified-config
/test yamllint

The following commands are available to trigger optional jobs:

/test check-cluster-profiles-config

Use /test all to run the following jobs that were automatically triggered:

pull-ci-openshift-release-main-ci-operator-config
pull-ci-openshift-release-main-ci-operator-registry
pull-ci-openshift-release-main-core-valid
pull-ci-openshift-release-main-owners
pull-ci-openshift-release-main-release-controller-config
pull-ci-openshift-release-main-step-registry-metadata
pull-ci-openshift-release-main-step-registry-shellcheck
pull-ci-openshift-release-openshift-image-mirror-mappings
pull-ci-openshift-release-yamllint
Details

In response to this:

/test periodic-ci-openshift-release-main-nightly-4.20-e2e-metal-ovn-two-node-arbiter-upgrade

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@etmurasaki

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-release-main-nightly-4.20-e2e-metal-ovn-two-node-arbiter-upgrade

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@etmurasaki: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@etmurasaki

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-release-main-nightly-4.20-e2e-metal-ovn-two-node-arbiter-upgrade

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@etmurasaki: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@etmurasaki

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-arbiter-upgrade

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@etmurasaki: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@etmurasaki etmurasaki closed this Aug 26, 2026
@etmurasaki
etmurasaki deleted the debug-baremetalds-two-node-arbiter-upgrade-4.20-20260825 branch August 26, 2026 16:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant