Skip to content

fix(kserve): build localmodel operands for e2e-kserve-module - #84074

Open
bartoszmajsak wants to merge 1 commit into
openshift:mainfrom
bartoszmajsak:kserve-localmodel-e2e-images
Open

fix(kserve): build localmodel operands for e2e-kserve-module#84074
bartoszmajsak wants to merge 1 commit into
openshift:mainfrom
bartoszmajsak:kserve-localmodel-e2e-images

Conversation

@bartoszmajsak

@bartoszmajsak bartoszmajsak commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Builds the two localmodel operand images for opendatahub-io/kserve and passes them to the e2e-kserve-module test.

Why

The ModelCache e2e deploys the localmodel controller and node agent from the image tags in params.env, while the manifests that configure them come from the PR under test. Those two move independently, so a change that adds a flag to a localmodel binary together with the arg that passes it gets tested as new arg against previously published binary. The binary rejects the unknown flag, crash-loops, ModelCacheReady never goes true, and the ModelCache tests time out.

This is currently breaking e2e-kserve-module on release-v0.17 PRs. The five other operands the test deploys are already immune because the job builds and injects them; localmodel was the only one left reading a published tag.

Summary by CodeRabbit

The OpenShift CI configuration for opendatahub-io/kserve now builds the local model controller and node agent images from the PR source.

The e2e-kserve-module test now uses these images, along with the KServe agent and router images. This keeps binaries and manifests compatible during CI testing.

The kserve-module e2e deploys its operands from the tags in params.env,
while the manifests configuring them come from the PR under test. Those
move independently, so a change that adds a flag to a binary and the arg
that passes it lands as a pair but gets tested as new arg against the
previously published binary, which rejects it and crash-loops.

The localmodel controller and node agent were the operands still reading
published tags and actually running in this job; build them and pass them
through. The agent and router images were already built here but never
handed to the test, so wire those up too at no extra build cost.

Every operand the job deploys now comes from the PR. The consuming side
already exists in the repo.

Signed-off-by: Bartosz Majsak <bartosz.majsak@gmail.com>
@bartoszmajsak

Copy link
Copy Markdown
Contributor Author

/pj-rehearse-auto-ack

@openshift-ci
openshift-ci Bot requested review from maskarb and spolti August 26, 2026 09:46
@coderabbitai

coderabbitai Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 3db0eafe-e85c-40d7-8350-c090a020dd8c

📥 Commits

Reviewing files that changed from the base of the PR and between ace104d and 1ca46b8.

⛔ Files ignored due to path filters (4)
  • ci-operator/jobs/opendatahub-io/kserve/opendatahub-io-kserve-master-postsubmits.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/opendatahub-io/kserve/opendatahub-io-kserve-master-presubmits.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/opendatahub-io/kserve/opendatahub-io-kserve-release-v0.17-postsubmits.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/opendatahub-io/kserve/opendatahub-io-kserve-release-v0.17-presubmits.yaml is excluded by !ci-operator/jobs/**
📒 Files selected for processing (2)
  • ci-operator/config/opendatahub-io/kserve/opendatahub-io-kserve-master.yaml
  • ci-operator/config/opendatahub-io/kserve/opendatahub-io-kserve-release-v0.17.yaml

Included review availability: Your plan provides up to 2 included reviews per hour; 1 remains after this review.


Walkthrough

The KServe master and release v0.17 CI configurations now build local model images and pass those images, along with existing KServe dependencies, to the KServe module E2E test.

Changes

KServe local model CI

Layer / File(s) Summary
Local model image builds
ci-operator/config/opendatahub-io/kserve/opendatahub-io-kserve-master.yaml, ci-operator/config/opendatahub-io/kserve/opendatahub-io-kserve-release-v0.17.yaml
Both configurations build the local model controller and local model node agent images.
KServe E2E image dependencies
ci-operator/config/opendatahub-io/kserve/opendatahub-io-kserve-master.yaml, ci-operator/config/opendatahub-io/kserve/opendatahub-io-kserve-release-v0.17.yaml
The KServe module E2E test declares the KServe agent, router, local model controller, local model node agent, and storage initializer images.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to 1ca46

This change builds and supplies the localmodel images used by the end-to-end test, keeping the tested binaries aligned with the manifests under test. No actionable merge-blocking risk remains beyond normal checks and review.

Suggested reviewers: jooho, jlost, andresllh

🚥 Pre-merge checks | ✅ 15
✅ Passed checks (15 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: building the localmodel operand images for the KServe end-to-end module test.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The pull request does not add or change any Ginkgo test title. Its six changed files contain CI image and dependency configuration only. The exact diff adds Dockerfile image entries and `e2e-kserve-mo…
Test Structure And Quality ✅ Passed PASS: The pull request changes only CI YAML configuration and generated job YAML. The diff adds image build entries, test image dependencies, and Dockerfile path filters. It does not modify Ginkgo tes…
Microshift Test Compatibility ✅ Passed The check is not applicable. The pull request changes only CI YAML and generated Prow job YAML. The diff adds Docker image builds, image dependencies, and sparse-checkout entries. It adds no Ginkgo te…
Single Node Openshift (Sno) Test Compatibility ✅ Passed PASS: The pull request adds only CI YAML image/dependency configuration and generated job sparse-checkout entries. The commit changes no Go or e2e test files and adds no Ginkgo It(), Describe(), Conte…
Topology-Aware Scheduling Compatibility ✅ Passed PASS: The pull request changes only CI image and job configuration. It adds Dockerfile build entries, image dependencies, and sparse-checkout filenames. It does not modify deployment manifests, operat…
Ote Binary Stdout Contract ✅ Passed PASS — The pull request changes only CI YAML. The committed diff adds two image build entries, test image dependencies, and Dockerfile sparse-checkout paths. It does not modify Go or test-suite source…
Ipv6 And Disconnected Network Test Compatibility ✅ Passed PASS: The pull request adds only CI YAML image-build entries, test image dependencies, and generated job triggers. The exact diff changes no Ginkgo test code and adds no IPv4 address, IP parsing, URL …
No-Weak-Crypto ✅ Passed PASS. The pull request changes only CI YAML files. The added lines add Dockerfile entries, image targets, sparse-checkout paths, and test dependency mappings. The diff contains no MD5, SHA1, DES, RC4,…
Container-Privileges ✅ Passed The pull request adds image-build entries, test image dependencies, and Dockerfile sparse-checkout paths only. The parent-to-HEAD diff adds no privileged, hostPID, hostNetwork, hostIPC, `SYS_A…
No-Sensitive-Data-In-Logs ✅ Passed PASS. The commit changes only CI image build entries and E2E dependency mappings in the two YAML files. The added values are Dockerfile and image identifiers. No logging command, secret, token, passwo…
Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (2 skipped: 2 unsupported.)

Full details: Stable And Deterministic Test Names

Explanation

The pull request does not add or change any Ginkgo test title. Its six changed files contain CI image and dependency configuration only. The exact diff adds Dockerfile image entries and e2e-kserve-module dependencies; no It, Describe, Context, or When title construct appears in the added lines.

Full details: Test Structure And Quality

Explanation

PASS: The pull request changes only CI YAML configuration and generated job YAML. The diff adds image build entries, test image dependencies, and Dockerfile path filters. It does not modify Ginkgo test code or any It, BeforeEach, AfterEach, Eventually, or Consistently blocks, so the listed test-structure requirements are not applicable.

Full details: Microshift Test Compatibility

Explanation

The check is not applicable. The pull request changes only CI YAML and generated Prow job YAML. The diff adds Docker image builds, image dependencies, and sparse-checkout entries. It adds no Ginkgo test declarations or test source, and the existing e2e-kserve-module command remains unchanged. Therefore, the pull request introduces no MicroShift-incompatible test API or feature under this check.

Full details: Single Node Openshift (Sno) Test Compatibility

Explanation

PASS: The pull request adds only CI YAML image/dependency configuration and generated job sparse-checkout entries. The commit changes no Go or e2e test files and adds no Ginkgo It(), Describe(), Context(), or When() declarations. Therefore, the SNO check is not applicable.

Full details: Topology-Aware Scheduling Compatibility

Explanation

PASS: The pull request changes only CI image and job configuration. It adds Dockerfile build entries, image dependencies, and sparse-checkout filenames. It does not modify deployment manifests, operator code, controllers, replica settings, affinity, topology spread, node selectors, tolerations, or PDBs. Therefore, it introduces none of the stated topology-compatibility failures.

Full details: Ote Binary Stdout Contract

Explanation

PASS — The pull request changes only CI YAML. The committed diff adds two image build entries, test image dependencies, and Dockerfile sparse-checkout paths. It does not modify Go or test-suite source, main(), TestMain(), suite setup, logging configuration, or any stdout write. The added YAML contains no process-level stdout operation, so it introduces no OTE JSON stdout contract violation.

Full details: Ipv6 And Disconnected Network Test Compatibility

Explanation

PASS: The pull request adds only CI YAML image-build entries, test image dependencies, and generated job triggers. The exact diff changes no Ginkgo test code and adds no IPv4 address, IP parsing, URL construction, external host, download, or public-registry access. The existing e2e-kserve-module command remains unchanged.

Full details: No-Weak-Crypto

Explanation

PASS. The pull request changes only CI YAML files. The added lines add Dockerfile entries, image targets, sparse-checkout paths, and test dependency mappings. The diff contains no MD5, SHA1, DES, RC4, 3DES, Blowfish, ECB, custom crypto implementation, or non-constant-time secret/token comparison.

Full details: Container-Privileges

Explanation

The pull request adds image-build entries, test image dependencies, and Dockerfile sparse-checkout paths only. The parent-to-HEAD diff adds no privileged, hostPID, hostNetwork, hostIPC, SYS_ADMIN, allowPrivilegeEscalation, capability, or security-context fields. The existing USER 0:0 build-root instruction is unchanged and is followed by USER 1001:0; it is not introduced by this pull request.

Full details: No-Sensitive-Data-In-Logs

Explanation

PASS. The commit changes only CI image build entries and E2E dependency mappings in the two YAML files. The added values are Dockerfile and image identifiers. No logging command, secret, token, password, personal data, session identifier, or customer data was added. Existing log-gather commands and the existing BASE_DOMAIN value are unchanged.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 26, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@bartoszmajsak: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@pierDipi pierDipi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Aug 26, 2026
@openshift-ci

openshift-ci Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bartoszmajsak, pierDipi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

[REHEARSALNOTIFIER]
@bartoszmajsak: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-opendatahub-io-kserve-release-v0.17-e2e-graph opendatahub-io/kserve presubmit Ci-operator config changed
pull-ci-opendatahub-io-kserve-release-v0.17-e2e-kserve-module opendatahub-io/kserve presubmit Ci-operator config changed
pull-ci-opendatahub-io-kserve-release-v0.17-e2e-llm-inference-service opendatahub-io/kserve presubmit Ci-operator config changed
pull-ci-opendatahub-io-kserve-release-v0.17-e2e-predictor opendatahub-io/kserve presubmit Ci-operator config changed
pull-ci-opendatahub-io-kserve-release-v0.17-e2e-raw opendatahub-io/kserve presubmit Ci-operator config changed
pull-ci-opendatahub-io-kserve-release-v0.17-images opendatahub-io/kserve presubmit Ci-operator config changed
pull-ci-opendatahub-io-kserve-release-v0.17-pr-image-mirror-kserve-agent opendatahub-io/kserve presubmit Ci-operator config changed
pull-ci-opendatahub-io-kserve-release-v0.17-pr-image-mirror-kserve-controller opendatahub-io/kserve presubmit Ci-operator config changed
pull-ci-opendatahub-io-kserve-release-v0.17-pr-image-mirror-kserve-module-controller opendatahub-io/kserve presubmit Ci-operator config changed
pull-ci-opendatahub-io-kserve-release-v0.17-pr-image-mirror-kserve-router opendatahub-io/kserve presubmit Ci-operator config changed
pull-ci-opendatahub-io-kserve-release-v0.17-pr-image-mirror-kserve-storage-initializer opendatahub-io/kserve presubmit Ci-operator config changed
pull-ci-opendatahub-io-kserve-release-v0.17-pr-image-mirror-llmisvc-controller opendatahub-io/kserve presubmit Ci-operator config changed
pull-ci-opendatahub-io-kserve-master-e2e-graph opendatahub-io/kserve presubmit Ci-operator config changed
pull-ci-opendatahub-io-kserve-master-e2e-kserve-module opendatahub-io/kserve presubmit Ci-operator config changed
pull-ci-opendatahub-io-kserve-master-e2e-llm-inference-service opendatahub-io/kserve presubmit Ci-operator config changed
pull-ci-opendatahub-io-kserve-master-e2e-predictor opendatahub-io/kserve presubmit Ci-operator config changed
pull-ci-opendatahub-io-kserve-master-e2e-raw opendatahub-io/kserve presubmit Ci-operator config changed
pull-ci-opendatahub-io-kserve-master-images opendatahub-io/kserve presubmit Ci-operator config changed
pull-ci-opendatahub-io-kserve-master-pr-image-mirror-kserve-agent opendatahub-io/kserve presubmit Ci-operator config changed
pull-ci-opendatahub-io-kserve-master-pr-image-mirror-kserve-controller opendatahub-io/kserve presubmit Ci-operator config changed
pull-ci-opendatahub-io-kserve-master-pr-image-mirror-kserve-module-controller opendatahub-io/kserve presubmit Ci-operator config changed
pull-ci-opendatahub-io-kserve-master-pr-image-mirror-kserve-router opendatahub-io/kserve presubmit Ci-operator config changed
pull-ci-opendatahub-io-kserve-master-pr-image-mirror-kserve-storage-initializer opendatahub-io/kserve presubmit Ci-operator config changed

Prior to this PR being merged, you will need to either run and acknowledge or opt to skip these rehearsals.

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@bartoszmajsak: job(s): /pj-rehearse-auto-ack either don't exist or were not found to be affected, and cannot be rehearsed

@bartoszmajsak

Copy link
Copy Markdown
Contributor Author

/pj-rehearse auto-ack

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@bartoszmajsak: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-merge-bot openshift-merge-bot Bot added the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label Aug 26, 2026
@openshift-ci

openshift-ci Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

@bartoszmajsak: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. rehearsals-ack Signifies that rehearsal jobs have been acknowledged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants