stackrox: automatically test candidate RCs

[davdhacs]

Test stackrox PRs and nightlies on the RCs for the next OCP version.

This changes to testing automatically at the first RC instead of waiting to test after GA of each next OCP version.

The prior approach was to wait and manually add testing after we see an RC is available, and then manually set it to run automatically after GA.

/uncc

Summary by CodeRabbit

• Updates StackRox CI to test the next OCP release from its first release candidate through the ocp/candidate tag.
• Adds nightly and postsubmit E2E coverage for QA, scanner, operator, UI, nongroovy, VM-scanning, FIPS, and compliance suites.
• Runs selected tests only when relevant files change while keeping nongroovy tests always enabled.
• Adds VM-scanning coverage with a seven-hour timeout and limits postsubmit jobs to six concurrent runs.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 3174219f-3688-4412-81e7-0ef595f2c6f1

📥 Commits

Reviewing files that changed from the base of the PR and between e45e449 and ce6162b.

⛔ Files ignored due to path filters (3)

• ci-operator/jobs/stackrox/stackrox/stackrox-stackrox-master-postsubmits.yaml is excluded by !ci-operator/jobs/**
• ci-operator/jobs/stackrox/stackrox/stackrox-stackrox-master-presubmits.yaml is excluded by !ci-operator/jobs/**
• ci-operator/jobs/stackrox/stackrox/stackrox-stackrox-nightlies-postsubmits.yaml is excluded by !ci-operator/jobs/**

📒 Files selected for processing (2)

• ci-operator/config/stackrox/stackrox/stackrox-stackrox-master__ocp-next-candidate.yaml
• ci-operator/config/stackrox/stackrox/stackrox-stackrox-nightlies__ocp-next-candidate.yaml

🚧 Files skipped from review as they are similar to previous changes (1)

• ci-operator/config/stackrox/stackrox/stackrox-stackrox-nightlies__ocp-next-candidate.yaml

Included review availability: Your plan provides up to 2 included reviews per hour; 0 remain after this review.

---

Walkthrough

The CI configuration adds path-filtered master E2E jobs, postsubmit merge jobs, and a nightlies configuration for OpenShift next-candidate. It also adds VM-scanning workflows with seven-hour timeouts.

Changes

Stackrox CI workflows

Layer / File(s)	Summary
Master E2E filtering and merge workflows ci-operator/config/stackrox/stackrox/stackrox-stackrox-master__ocp-next-candidate.yaml	QA, scanner, operator, UI, and compliance jobs use path-based run_if_changed filters. Nongroovy runs always. VM scanning runs with always_run: false and a seven-hour timeout. Postsubmit merge jobs cover six E2E suites and allow up to six concurrent runs.
Nightlies next-candidate workflows ci-operator/config/stackrox/stackrox/stackrox-stackrox-nightlies__ocp-next-candidate.yaml	The configuration defines images, resource requests, test-binary build commands, seven postsubmit E2E workflows, VM-scanning timeout settings, and generated repository metadata.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to ce616

This change automates Stackrox testing against release candidates for the next OpenShift version, and no actionable merge-blocking risk remains beyond normal checks and review.

🚥 Pre-merge checks | ✅ 15

✅ Passed checks (15 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the primary change: automatically testing StackRox against candidate OpenShift releases.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	The pull request changes only CI configuration and generated Prow job YAML. The diff adds static suite/job identifiers and TEST_SUITE values. It adds no It(), Describe(), Context(), or `When()…
Test Structure And Quality	✅ Passed	PASS: The pull request changes only CI YAML configuration and generated Prow job YAML. The diff contains no Ginkgo test source, test declarations, cluster setup or cleanup code, wait calls, or asserti…
Microshift Test Compatibility	✅ Passed	PASS: The pull-request diff from the merge base changes only five CI YAML files. It adds or updates job configuration and suite selectors, but it adds no Ginkgo test code (It, Describe, Context,…
Single Node Openshift (Sno) Test Compatibility	✅ Passed	PASS: The pull request adds or changes only CI YAML configuration and generated Prow job YAML. The diff against main contains five YAML paths and no Go or other Ginkgo test source. The added entries o…
Topology-Aware Scheduling Compatibility	✅ Passed	PASS — The check is not applicable. The pull request changes only five CI YAML files under ci-operator/config and ci-operator/jobs. The diff adds or updates E2E and Prow job definitions, with no d…
Ote Binary Stdout Contract	✅ Passed	PASS: The pull request changes only five YAML CI configuration files. The diff adds or updates StackRox test scheduling, suite names, workflows, and generated Prow jobs; it does not change an OTE bina…
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	PASS: The pull request changes only StackRox CI YAML and generated Prow jobs. It adds or schedules existing suites through TEST_SUITE values; it does not add Ginkgo source containing It(), `Descri…
No-Weak-Crypto	✅ Passed	PASS: The pull-request changes are CI YAML only. The relevant diffs add or modify E2E job filters, OCP_VERSION/TEST_SUITE environment values, timeouts, and postsubmit jobs. No MD5, SHA1, DES, RC4,…
Container-Privileges	✅ Passed	PASS: The changed StackRox candidate configurations add test selection, workflow references, and timeouts only. The added lines and generated StackRox job changes contain no privileged: true, `hostP…
No-Sensitive-Data-In-Logs	✅ Passed	The pull request changes only StackRox CI configuration and generated Prow job YAML. The added jobs set non-sensitive test variables such as OCP_VERSION, TEST_SUITE, and FIPS_ENABLED. No logging…

Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (2 skipped: 2 unsupported.)

Full details: Stable And Deterministic Test Names

Explanation

The pull request changes only CI configuration and generated Prow job YAML. The diff adds static suite/job identifiers and TEST_SUITE values. It adds no It(), Describe(), Context(), or When() declarations, and no dynamic test-title construction. Therefore, the check's failure conditions are not introduced.

Full details: Test Structure And Quality

Explanation

PASS: The pull request changes only CI YAML configuration and generated Prow job YAML. The diff contains no Ginkgo test source, test declarations, cluster setup or cleanup code, wait calls, or assertions. Therefore the five Ginkgo test quality requirements are not applicable to these changes.

Full details: Microshift Test Compatibility

Explanation

PASS: The pull-request diff from the merge base changes only five CI YAML files. It adds or updates job configuration and suite selectors, but it adds no Ginkgo test code (It, Describe, Context, or When) and no test body that references MicroShift-unavailable APIs or features. The custom check is therefore not applicable.

Full details: Single Node Openshift (Sno) Test Compatibility

Explanation

PASS: The pull request adds or changes only CI YAML configuration and generated Prow job YAML. The diff against main contains five YAML paths and no Go or other Ginkgo test source. The added entries only select existing test suites through TEST_SUITE; they do not add It(), Describe(), Context(), or When() tests. Therefore the SNO multi-node compatibility check is not applicable.

Full details: Topology-Aware Scheduling Compatibility

Explanation

PASS — The check is not applicable. The pull request changes only five CI YAML files under ci-operator/config and ci-operator/jobs. The diff adds or updates E2E and Prow job definitions, with no deployment manifests, operator code, controllers, or topology-dependent workload scheduling constraints.

Full details: Ote Binary Stdout Contract

Explanation

PASS: The pull request changes only five YAML CI configuration files. The diff adds or updates StackRox test scheduling, suite names, workflows, and generated Prow jobs; it does not change an OTE binary or process-level code. No changed line contains main, TestMain, suite setup, fmt.Print*, klog, or logging output configuration. The generated jobs invoke ci-operator, so the pull request introduces no stdout write covered by this check.

Full details: Ipv6 And Disconnected Network Test Compatibility

Explanation

PASS: The pull request changes only StackRox CI YAML and generated Prow jobs. It adds or schedules existing suites through TEST_SUITE values; it does not add Ginkgo source containing It(), Describe(), Context(), or When(). The added StackRox lines contain no hardcoded IPv4 addresses or public URLs. Job networking uses dynamic HTTP_SERVER_IP and registry credentials. The check is therefore not applicable.

Full details: No-Weak-Crypto

Explanation

PASS: The pull-request changes are CI YAML only. The relevant diffs add or modify E2E job filters, OCP_VERSION/TEST_SUITE environment values, timeouts, and postsubmit jobs. No MD5, SHA1, DES, RC4, 3DES, Blowfish, ECB, custom cryptography, or secret/token comparisons appear in the changed lines.

Full details: Container-Privileges

Explanation

PASS: The changed StackRox candidate configurations add test selection, workflow references, and timeouts only. The added lines and generated StackRox job changes contain no privileged: true, hostPID, hostNetwork, hostIPC, SYS_ADMIN, allowPrivilegeEscalation: true, or root-execution declarations. The referenced StackRox workflow and its step assets also contain none of these settings.

Full details: No-Sensitive-Data-In-Logs

Explanation

The pull request changes only StackRox CI configuration and generated Prow job YAML. The added jobs set non-sensitive test variables such as OCP_VERSION, TEST_SUITE, and FIPS_ENABLED. No logging, shell output, environment dumps, passwords, tokens, API keys, PII, hostnames, or customer data were added. The generated jobs reference standard CI credential files by path, but do not print or expose their contents.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: davdhacs

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• ci-operator/config/stackrox/OWNERS [davdhacs]
• ci-operator/jobs/stackrox/stackrox/OWNERS [davdhacs]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[davdhacs]

/pj-rehearse pull-ci-stackrox-stackrox-master-ocp-next-candidate-qa-e2e-tests
still on 4.22 earlier today

[openshift-merge-bot]

@davdhacs: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

[openshift-merge-bot]

[REHEARSALNOTIFIER]
@davdhacs: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name	Repo	Type	Reason
pull-ci-stackrox-stackrox-master-ocp-next-candidate-nongroovy-e2e-tests	stackrox/stackrox	presubmit	Presubmit changed
pull-ci-stackrox-stackrox-master-ocp-next-candidate-vm-scanning-e2e-tests	stackrox/stackrox	presubmit	Presubmit changed
pull-ci-stackrox-stackrox-master-ocp-next-candidate-compliance-e2e-tests	stackrox/stackrox	presubmit	Ci-operator config changed
pull-ci-stackrox-stackrox-master-ocp-next-candidate-operator-e2e-tests	stackrox/stackrox	presubmit	Ci-operator config changed
pull-ci-stackrox-stackrox-master-ocp-next-candidate-qa-e2e-tests	stackrox/stackrox	presubmit	Ci-operator config changed
pull-ci-stackrox-stackrox-master-ocp-next-candidate-scanner-v4-install-tests	stackrox/stackrox	presubmit	Ci-operator config changed
pull-ci-stackrox-stackrox-master-ocp-next-candidate-ui-e2e-tests	stackrox/stackrox	presubmit	Ci-operator config changed

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

[openshift-ci]

@davdhacs: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[davdhacs]

/cc @msugakov

what: With the change to testing on OCP RCs instead of waiting for post-GA stability, we can enable always/change-trigger for testing on the floating "candidate" label and it will always be testing the latest RC for the next OCP version. What do you think?

why: I'm re-considering this now because of this change to testing on RCs and because we cannot yet add those tests for OCP 5.0 RC.0 since it is not published. If we add testing of the label candidate then it will automatically switch from 4.22's latest patch RC to the 5.0 RC.0 when that RC is published.

Examples:

1. now, next is still in ECs and so 'candidate' shows patch updates to the last GA minor version, 4.22:
   26th August 2026 (5.0 is still in ECs, 4.22 is GA and has RCs, 4.23 ECs have not started):
   candidate: 4.22.11 https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/candidate/
   dev-preview: 5.0.0-ec.6 https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp-dev-preview/candidate/
   fast: 4.22.11
   latest: 4.22.11
   stable: 4.22.10

2. soon, once RC.0 is published for the next minor version
   candidate: 5.0.0 RC.0
   dev-preview: 5.0.0 EC.X
   stable: 4.22.11

3. later, when 4.23 EC.0 is cut
   candidate: 5.0.0 RC.X
   dev-preview: 4.23.0 EC.0
   stable: 4.22.11

4. after 5.0 GA and 4.23 feature-complete (*4.23 is potentially unique as the OCP sequencing might mean they will not publish it as the next 'candidate' and that may then point at 5.0.1 ECs).
   A.
   candidate-5.0: 5.0.0 RC.X
   candidate-4.23: 4.23.0 RC.0
   candidate: 4.23.0 RC.X
   dev-preview: 4.23.0 EC.X
   dev-preview 5.0: 5.0.1 EC.X
   dev-preview 4.23: 4.23.0 EC.X
   stable: 5.0.0
   B.
   candidate-5.0: 5.0.0 RC.X
   candidate-4.23: 4.23.0 RC.0
   candidate: 5.0.0 RC.X
   dev-preview: 4.23.0 EC.X
   dev-preview 5.0: 5.0.1 EC.X
   dev-preview 4.23: 4.23.0 EC.X
   stable: 5.0.0

The concerns I have are (some resurfaced from prior discussions):

1. Will the test name of "ocp-next-candidate" instead of "ocp-4-23" cause problems or confusion for ci triage or PR creators?
2. If we similarly use the ocp-dev-preview/candidate label for testing ECs, that label gets left at the last EC during the RC process. If our tests fail on the last EC because of an OCP issue, then they will continue to fail even if a fix is in OCP RCs.
3. We still need to pin to the actual supported OCP versions for the release branches. For instance, we could keep releasing patches for release-4.11 testing on candidate and can stop and pin the highest OCP version when we stop full support releases for 4.11.

[msugakov]

(Not about this line, but starting a thread here because replying in the thread is more convenient.)

I appreciate a lot your explanation and examples in #84050 (comment)!

I see that we already have

• stackrox-stackrox-master__ocp-dev-preview.yaml using ocp-dev-preview/candidate which resolves to 5.0.0-ec.6 today, and
• stackrox-stackrox-master__ocp-next-candidate.yaml using ocp/candidate which resolves today to 4.22.11.

This makes me really wonder if we should have stackrox-stackrox-master__ocp-5-0.yaml config at all at this point. It can appear later or we can even even rename+reuse the previous config for that, but this is a bit off-topic.

I see that in the current version of the Confluence page https://redhat.atlassian.net/wiki/spaces/StackRox/pages/309338571/Add+support+for+new+OpenShift+version you have a place for ocp-dev-preview and ocp-next-candidate. So it's a matter of making the actual configs correspond to

• this for ocp-dev-preview

• and this for ocp-next-candidate

I haven't checked everything but I see that in this PR you're enabling ocp-next-candidate for master postsubmits, presubmits and nightlies.

This makes sense but I worry about avoidable cluster costs a bit.
If I get where you're going, we could have ocp-dev-preview and ocp-next-candidate just permanently set up and enjoy new OCP versions getting picked up and tested when they appear.

Please check whether I understood most things correctly and please point out where you see things differently!
I have a couple suggestions after I sync my understanding with yours.
Maybe we should just have a meeting?

[davdhacs]

ocp-dev-preview and ocp-next-candidate just permanently set up and enjoy new OCP versions getting picked up and tested when they appear

yes, precisely. That is a better way to describe it. And so we'd always be testing on the latest EC and RC and we'll find out about failures during EC/RC like we do now, but without changing the config names and pinned versions (keeping pinned for releases).

What avoidable cluster costs might this introduce? (I'll list the few I can think of right now, and I'd like to know if you anticipate others):

1. duplicate runs (when EC is the same as RC? EC.[-1] == RC.0)
2. unnecessary testing (early ECs? Maybe we don't really care if there are failures during ECs, or the first ones?)
3. re-test because of failures: For example the ui-e2e fails still (1 month later) on 5.0. I saw an updated PR yesterday and it looks like the UI team is nearly finished fixing it, but this wouldn't be acceptable if the dev-preview[EC] test was running and failing in nightlies for the last month. If it continues failing after the first RCs, then it would show up on master and PRs and then bot/retest would re-run it despite it being a known failure.