gcp: migrate e2e-gcp-ovn-rt to N4 with hyperdisk-balanced (1/2)

[mkowalski]

What this PR changes

Moves the openshift-e2e-gcp-ovn-rt workflow from GCP N2 to N4 worker nodes. N4 only supports hyperdisk-balanced disks (not Persistent Disk), so the workflow also switches the disk type and makes hyperdisk-balanced the default StorageClass. All four consumers of this workflow run on OCP 4.22+, safely above the 4.17 floor for N4 + hyperdisk.

1) Workflow pre steps — two steps added before install

flowchart TB
    S1["ipi-conf-gcp"] --> S2["ovn-conf"] --> S3["ipi-conf-debug-kdump-configure-logs"] --> S4["rhcos-conf-osstream"] --> S5["ipi-conf-gcp-osdisk-disktype<br/>NEW - sets worker OS disk = hyperdisk-balanced"]:::new --> S6["storage-conf-storageclass-pre-set-default-gcp-hyperdisk<br/>NEW - makes hyperdisk-balanced the default StorageClass"]:::new --> S7["ipi-install-stableinitial"]
    classDef new fill:#d4f7d4,stroke:#2ea043,stroke-width:2px,color:#000;

Loading

2) Workflow env — machine type + disk type

flowchart LR
    O1["COMPUTE_NODE_TYPE = n2-standard-8"]:::old -->|changed| N1["COMPUTE_NODE_TYPE = n4-standard-8"]:::new
    O2["(no disk type set)"]:::old -->|added| N2["COMPUTE_DISK_TYPE = hyperdisk-balanced"]:::new
    classDef old fill:#ffecec,stroke:#d1242f,color:#000;
    classDef new fill:#d4f7d4,stroke:#2ea043,color:#000;

Loading

3) The four consumer configs — drop the redundant override

Each e2e-gcp-ovn-rt-rhcos10-techpreview test used to pin the old N2 type. Removing that override lets the test inherit the new N4 + hyperdisk settings straight from the workflow (single source of truth).

flowchart TB
    C["nightly-4.22 / 4.23 / 5.0 / 5.1<br/>test: e2e-gcp-ovn-rt-rhcos10-techpreview"]
    C -->|before| CB["env: COMPUTE_NODE_TYPE = n2-standard-8<br/>(overrode the workflow)"]:::old
    C -->|after| CA["override removed → inherits<br/>n4-standard-8 + hyperdisk-balanced"]:::new
    classDef old fill:#ffecec,stroke:#d1242f,color:#000;
    classDef new fill:#d4f7d4,stroke:#2ea043,color:#000;

Loading

Files changed (5)

File	Change
step-registry/openshift/e2e/gcp/ovn-rt/openshift-e2e-gcp-ovn-rt-workflow.yaml	N4 + hyperdisk-balanced; add osdisk-disktype + hyperdisk-StorageClass steps
config/.../openshift-release-main__nightly-4.22.yaml	drop COMPUTE_NODE_TYPE: n2-standard-8 override
config/.../openshift-release-main__nightly-4.23.yaml	same
config/.../openshift-release-main__nightly-5.0.yaml	same
config/.../openshift-release-main__nightly-5.1.yaml	same

Context: forward step of the N2→N4 migration on top of the (merged) revert #84065.

⚠️ Status: rehearsals surfaced two N4 problems for this conformance job — intermittent N4 worker provisioning in us-central1, and a deterministic failure of the GCE Persistent-Disk conformance test (regional PD … sync across zones: "Regional disks is not supported for n4-standard-8"). Full root-cause: this comment. Because openshift/conformance/parallel exercises GCE‑PD, this job likely needs to stay on N2 — treat this PR as under review pending that decision.

---

This description was generated using AI. Please verify before acting on it.

Summary by CodeRabbit

• Migrates the openshift-e2e-gcp-ovn-rt workflow from GCP N2 to N4.
• Uses n4-standard-8 workers with hyperdisk-balanced disks.
• Configures GCP OS disks and sets Hyperdisk as the default StorageClass before installation.
• Removes redundant N2 overrides from the 4.22, 4.23, 5.0, and 5.1 nightly consumers.
• Skips the regional Persistent Disk conformance test for N4 consumers because N4 does not support regional Persistent Disks.
• Keeps N2 jobs unchanged for Persistent Disk coverage.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: ea9879a2-e699-4068-bc42-fd00871de5e2

📥 Commits

Reviewing files that changed from the base of the PR and between 9581846 and cf490ac.

📒 Files selected for processing (3)

• ci-operator/config/openshift/release/openshift-release-main__nightly-4.23.yaml
• ci-operator/config/openshift/release/openshift-release-main__nightly-5.0.yaml
• ci-operator/config/openshift/release/openshift-release-main__nightly-5.1.yaml

Included review availability: Your plan provides up to 12 included reviews per hour; 10 remain after this review.

---

Walkthrough

The GCP OVN realtime workflow adds OS-disk and HyperDisk settings and uses n4-standard-8 compute nodes. Four nightly configurations remove the explicit n2-standard-8 override. Three configurations skip the regional PD cross-zone synchronization test.

Changes

GCP OVN realtime workflow

Layer / File(s)	Summary
Workflow storage and compute defaults ci-operator/step-registry/openshift/e2e/gcp/ovn-rt/openshift-e2e-gcp-ovn-rt-workflow.yaml	The workflow adds GCP OS-disk and HyperDisk configuration steps. It uses n4-standard-8 compute nodes and hyperdisk-balanced compute disks.

Nightly configuration alignment

Layer / File(s)	Summary
Nightly test settings ci-operator/config/openshift/release/openshift-release-main__nightly-4.22.yaml, ci-operator/config/openshift/release/openshift-release-main__nightly-4.23.yaml, ci-operator/config/openshift/release/openshift-release-main__nightly-5.0.yaml, ci-operator/config/openshift/release/openshift-release-main__nightly-5.1.yaml	The configurations remove the explicit n2-standard-8 compute-node setting. The 4.23, 5.0, and 5.1 configurations skip the regional PD cross-zone synchronization test.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to cf490

The workflow moves from N2 to N4 with hyperdisk-balanced storage and removes redundant consumer overrides; no actionable merge-blocking risk remains in the supplied current-head review evidence.

Suggested reviewers: dgoodwin, neisw, nocturnalastro

🚥 Pre-merge checks | ✅ 15

✅ Passed checks (15 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely summarizes the main change: migrating the GCP OVN realtime workflow from N2 to N4 with hyperdisk-balanced storage. The step indicator is also consistent with the stated…
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	PASS. The PR changes only CI YAML. The workflow changes add pre-step references and set COMPUTE_NODE_TYPE/COMPUTE_DISK_TYPE; the consumer changes remove the old override and add the static `TEST_S…
Test Structure And Quality	✅ Passed	PASS — The PR changes only five YAML configuration/workflow files. The exact two-commit diff contains no Ginkgo or other test-source files, so it introduces no It blocks, resource setup, waits, or ass…
Microshift Test Compatibility	✅ Passed	PASS — The pull request changes only CI YAML workflow and test configuration. The PR diff adds no Go files and no new Ginkgo constructs such as It(), Describe(), Context(), or When(). It there…
Single Node Openshift (Sno) Test Compatibility	✅ Passed	PASS: The pull request changes only five YAML configuration/workflow files. The diff adds no Ginkgo test declarations or test source; it only changes worker, disk, pre-step, and TEST_SKIPS settings. T…
Topology-Aware Scheduling Compatibility	✅ Passed	PASS — The PR changes only CI workflow/configuration YAML. The diff changes GCP machine and disk environment variables, adds existing pre-steps, removes consumer overrides, and adds a test skip. It do…
Ote Binary Stdout Contract	✅ Passed	PASS: The PR changes only five YAML workflow/configuration files. The diff contains no Go files, OTE binary code, suite setup, process entry points, or stdout/logging writes. The added GCP settings an…
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	PASS — The pull request changes only five YAML CI configuration files. The diff adds no Ginkgo test declarations, test source, IPv4 parsing, hardcoded test addresses, or external-network test logic. T…
No-Weak-Crypto	✅ Passed	The PR diff only changes YAML workflow and test configuration. It switches GCP node and disk settings, adds two pre-step references, removes old node overrides, and adds a Persistent Disk test skip. N…
Container-Privileges	✅ Passed	PASS: The pull-request diff changes only workflow steps, GCP disk settings, and test environment values. It adds no privileged, hostPID, hostNetwork, hostIPC, SYS_ADMIN, `allowPrivilegeEscal…
No-Sensitive-Data-In-Logs	✅ Passed	PASS: The PR changes only GCP machine, disk, workflow-step, and test-skip configuration. The changed lines contain no passwords, tokens, API keys, PII, session IDs, hostnames, or customer data. The tw…

Full details: Title check

Explanation

The title clearly and concisely summarizes the main change: migrating the GCP OVN realtime workflow from N2 to N4 with hyperdisk-balanced storage. The step indicator is also consistent with the stated two-step migration.

Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (3 skipped: 3 unsupported.)

Full details: Stable And Deterministic Test Names

Explanation

PASS. The PR changes only CI YAML. The workflow changes add pre-step references and set COMPUTE_NODE_TYPE/COMPUTE_DISK_TYPE; the consumer changes remove the old override and add the static TEST_SKIPS string regional PD should store data and sync across zones. The exact diffs contain no It(), Describe(), Context(), or When() declarations and no dynamic test-title construction. No stated failure condition is introduced.

Full details: Test Structure And Quality

Explanation

PASS — The PR changes only five YAML configuration/workflow files. The exact two-commit diff contains no Ginkgo or other test-source files, so it introduces no It blocks, resource setup, waits, or assertions for this check to review.

Full details: Microshift Test Compatibility

Explanation

PASS — The pull request changes only CI YAML workflow and test configuration. The PR diff adds no Go files and no new Ginkgo constructs such as It(), Describe(), Context(), or When(). It therefore introduces no new MicroShift-incompatible e2e test to assess.

Full details: Single Node Openshift (Sno) Test Compatibility

Explanation

PASS: The pull request changes only five YAML configuration/workflow files. The diff adds no Ginkgo test declarations or test source; it only changes worker, disk, pre-step, and TEST_SKIPS settings. Therefore, the SNO compatibility check is not applicable.

Full details: Topology-Aware Scheduling Compatibility

Explanation

PASS — The PR changes only CI workflow/configuration YAML. The diff changes GCP machine and disk environment variables, adds existing pre-steps, removes consumer overrides, and adds a test skip. It does not add or modify deployment manifests, operator code, or controllers, and it introduces none of the listed pod scheduling constraints such as affinity, topology spread, node selectors, tolerations, replica counts, or PDBs.

Full details: Ote Binary Stdout Contract

Explanation

PASS: The PR changes only five YAML workflow/configuration files. The diff contains no Go files, OTE binary code, suite setup, process entry points, or stdout/logging writes. The added GCP settings and TEST_SKIPS values do not introduce an OTE stdout contract violation.

Full details: Ipv6 And Disconnected Network Test Compatibility

Explanation

PASS — The pull request changes only five YAML CI configuration files. The diff adds no Ginkgo test declarations, test source, IPv4 parsing, hardcoded test addresses, or external-network test logic. The new workflow entries configure GCP disks and nodes, and the consumer changes remove an override or add TEST_SKIPS; they do not add tests. Existing IPv4 configuration elsewhere in the files is outside the changed lines and does not trigger this check.

Full details: No-Weak-Crypto

Explanation

The PR diff only changes YAML workflow and test configuration. It switches GCP node and disk settings, adds two pre-step references, removes old node overrides, and adds a Persistent Disk test skip. No MD5, SHA1, DES, RC4, 3DES, Blowfish, ECB, cryptographic implementation, or secret/token comparison appears in the changed lines.

Full details: Container-Privileges

Explanation

PASS: The pull-request diff changes only workflow steps, GCP disk settings, and test environment values. It adds no privileged, hostPID, hostNetwork, hostIPC, SYS_ADMIN, allowPrivilegeEscalation, or root security settings. The added step scripts create a StorageClass and ClusterCSIDriver, not privileged Pod or container manifests.

Full details: No-Sensitive-Data-In-Logs

Explanation

PASS: The PR changes only GCP machine, disk, workflow-step, and test-skip configuration. The changed lines contain no passwords, tokens, API keys, PII, session IDs, hostnames, or customer data. The two added pre-steps emit only configuration/status text; the os-disk step reads the non-sensitive compute section and does not print credentials.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[openshift-merge-bot]

@mkowalski: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[mkowalski]

/pj-rehearse

[mkowalski]

/verified by CI

[openshift-ci-robot]

@mkowalski: This PR has been marked as verified by CI.

Details

In response to this:

/verified by CI

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-merge-bot]

@mkowalski: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[mkowalski]

In 3 failed tests we created an N4 instance but it vanished a few seconds later. This may be an instance of our well-known GCP issue. Or not. We shall see.

[mkowalski]

/pj-rehearse periodic-ci-openshift-release-main-nightly-4.22-e2e-gcp-ovn-rt-rhcos10-techpreview periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-rt-rhcos10-techpreview periodic-ci-openshift-release-main-nightly-5.1-e2e-gcp-ovn-rt-rhcos10-techpreview

Re-running the three failed rehearsals to determine whether the N4 worker provisioning failure (can't find created instance in us-central1) is a transient GCP capacity issue or systematic.

---

This comment was generated using AI. Please verify before acting on it.

[openshift-merge-bot]

@mkowalski: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[mkowalski]

/pj-rehearse periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-rt-rhcos10-techpreview

The previous 5.0 rehearsal failed on a transient CI infra flake (gcloud auth token refresh could not resolve oauth2.googleapis.com in the ipi-conf-gcp-zones step), not on the change under test. Re-running to get a clean result.

---

This comment was generated using AI. Please verify before acting on it.

[openshift-merge-bot]

@mkowalski: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[mkowalski]

Rehearsal root-cause summary

All four e2e-gcp-ovn-rt-rhcos10-techpreview rehearsals (4.22 / 4.23 / 5.0 / 5.1) failed. Investigating the artifacts shows two distinct N4 problems, one transient and one deterministic.

A) Intermittent N4 worker provisioning (install fails) — transient

• 4.22, 5.1: failed at ipi-install-install-stableinitial (~1h10m). Worker Machines go Failed with errorMessage: "can't find created instance"; providerStatus shows MachineCreationSucceeded and then, ~30s later, InstanceExists=False / InstanceMissing: "Instance not found on provider". The N4 (n4-standard-8) instances briefly reach GCP PROVISIONING, then vanish (region us-central1). Masters (e2-custom-8-16384) come up fine, so the differentiator is the N4 workers. With 0/3 workers, ingress/image-registry/monitoring/authentication/console never become Available → installer exit 6.
• 4.23, 5.0 installed successfully, so this is intermittent — consistent with transient GCP N4 capacity/stockout in the leased zones, not a hard config error.
• (One earlier 5.0 retry failed on an unrelated transient gcloud auth/DNS flake in ipi-conf-gcp-zones — infra noise.)

B) Systematic N4 storage-conformance incompatibility (e2e fails) — deterministic

On the runs that did install (4.23, 5.0), openshift-e2e-test failed. The N4-caused failure:

[sig-storage][Driver: pd.csi.storage.gke.io] regional PD should store data and sync across zones
  → AttachVolume.Attach failed: googleapi: Error 400:
    Regional disks is not supported for n4-standard-8 machine type

N4 supports only hyperdisk-balanced, not Persistent Disk (regional or zonal). The openshift/conformance/parallel suite includes GCE‑PD storage tests, which therefore fail deterministically on N4 workers. (The other two failures — a [Feature:NetworkSegmentation] UDN test and a cluster-config-operator status timeout — look like ordinary flakes.)

Conclusion

This job runs the full parallel conformance suite, which exercises GCE Persistent Disk — something N4 cannot do. So moving openshift-e2e-gcp-ovn-rt (and any other job running the GCE‑PD storage conformance tests) to N4 is not viable: even when N4 capacity cooperates, the regional‑PD test will always fail. On top of that, N4 capacity in us-central1 for this project is currently flaky.

Recommendation: keep the GCP conformance jobs on N2 (they need PD‑capable workers for the storage conformance tests). N4 remains appropriate only where the suite does not exercise GCE‑PD (e.g. the CSI/hyperdisk‑specific jobs). I'll rescope the forward‑migration accordingly.

---

This comment was generated using AI. Please verify before acting on it.

[openshift-merge-bot]

[REHEARSALNOTIFIER]
@mkowalski: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name	Repo	Type	Reason
periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-rt-rhcos10-techpreview	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-release-main-nightly-4.22-e2e-gcp-ovn-rt-rhcos10-techpreview	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-release-main-nightly-4.23-e2e-gcp-ovn-rt-rhcos10-techpreview	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-release-main-nightly-5.1-e2e-gcp-ovn-rt-rhcos10-techpreview	N/A	periodic	Ci-operator config changed

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

[mkowalski]

/pj-rehearse

PD test should be disabled for N4 now

[openshift-merge-bot]

@mkowalski: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[mkowalski]

⚠️ The TEST_SKIPS added here is a temporary workaround — it must be removed later.

It skips regional PD should store data and sync across zones on the N4 e2e-gcp-ovn-rt-rhcos10-techpreview jobs (nightly 4.23/5.0/5.1). The durable fix makes the test skip itself on hyperdisk-only workers: openshift/origin#31572 (TRT-2930).

Once #31572 and its backports (release-5.1, release-5.0, release-4.23) reach the payloads, the TEST_SKIPS entries added by this PR must be removed in a follow-up — otherwise they would silently hide the test on these jobs forever.

---

This comment was generated using AI. Please verify before acting on it.

[mkowalski]

/retest-required
/pj-rehearse ack

[openshift-merge-bot]

@mkowalski: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[sdodson]

I was confused as to why the test job wasn't skipped in 4.22 but it's new in 4.23/5.0.
/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mkowalski, petr-muller, sdodson

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• ci-operator/config/openshift/release/OWNERS [petr-muller,sdodson]
• ci-operator/step-registry/openshift/e2e/gcp/ovn-rt/OWNERS [petr-muller,sdodson]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

@mkowalski: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/rehearse/periodic-ci-openshift-release-main-nightly-4.23-e2e-gcp-ovn-rt-rhcos10-techpreview	cf490ac	link	unknown	/pj-rehearse periodic-ci-openshift-release-main-nightly-4.23-e2e-gcp-ovn-rt-rhcos10-techpreview

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.