Skip to content

Update quay ARO Azure e2e tests: quay 3.17/3.18 on OCP 4.21 with bug fixes #80771

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
openshift-merge-bot merged 4 commits into from
Jun 22, 2026
Merged

Update quay ARO Azure e2e tests: quay 3.17/3.18 on OCP 4.21 with bug fixes #80771

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
8f4dbc1
Update quay ARO Azure e2e tests to quay 3.17 on OCP 4.21
LiZhang19817 Jun 19, 2026
089db74
Fix expired SAS token and add missing QBO/CSO source env vars
LiZhang19817 Jun 19, 2026
b102771
Determinize quay ARO config env var ordering
LiZhang19817 Jun 19, 2026
91d7e8b
Fix QBO test SIGPIPE and make it non-blocking for CSO
LiZhang19817 Jun 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
89 changes: 26 additions & 63 deletions ci-operator/config/quay/quay-tests/quay-quay-tests-master__quay-aro-ocp.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
base_images:
cli:
name: "4.19"
name: "4.21"
namespace: ocp
tag: cli
upi-installer:
name: "4.19"
name: "4.21"
namespace: ocp
tag: upi-installer
build_root:
image_stream_tag:
name: release
namespace: openshift
tag: rhel-9-release-golang-1.21-openshift-4.18
tag: rhel-9-release-golang-1.25-openshift-4.21
prowgen:
expose: true
private: true
Expand All @@ -21,7 +21,7 @@ releases:
architecture: amd64
product: ocp
stream: nightly
version: "4.19"
version: "4.21"
resources:
'*':
limits:
Expand All @@ -30,82 +30,45 @@ resources:
cpu: 100m
memory: 200Mi
tests:
- as: quay-e2e-tests-quay312-aro-ocp419
cron: 0 12 * * 3
reporter_config:
channel: '#quay-qe'
job_states_to_report:
- success
- failure
- error
report_template: '{{if eq .Status.State "success"}} :slack-green: Job *{{.Spec.Job}}*
ended with *{{.Status.State}}*. <{{.Status.URL}}|View logs> {{else}} :failed:
Job *{{.Spec.Job}}* ended with *{{.Status.State}}*. <{{.Status.URL}}|View logs>
{{end}}'
- as: quay-e2e-tests-quay317-aro
cron: 0 12 1 1 *
steps:
cluster_profile: azure-qe
env:
ARO_WORKER_COUNT: "6"
ARO_WORKER_VM_SIZE: Standard_D8s_v3
CSO_CHANNEL: stable-3.12
QBO_CHANNEL: stable-3.12
QUAY_OPERATOR_CHANNEL: stable-3.12
QUAY_VERSION: "3.12"
CSO_CHANNEL: stable-3.17
CSO_SOURCE: fbc-operator-catalog
MULTISTAGE_PARAM_OVERRIDE_QUAY_INDEX_IMAGE: quay.io/redhat-user-workloads/quay-eng-tenant/stable-3-17-v4-21@sha256:2a048c3d57231d7cb7885439217af05cb36df14589c2b7dda2d9da657108319b
QBO_CHANNEL: stable-3.17
QBO_SOURCE: fbc-operator-catalog
QUAY_OPERATOR_CHANNEL: stable-3.17
QUAY_OPERATOR_SOURCE: fbc-operator-catalog
QUAY_VERSION: "3.17"
test:
- ref: quay-tests-enable-quay-catalogsource
- ref: quay-tests-deploy-quay-azure
- ref: quay-tests-test-quay-e2e
- ref: quay-tests-qbo-qe-test
- ref: quay-tests-cso-qe-test
workflow: cucushift-installer-rehearse-azure-aro
- as: quay-e2e-tests-quay314-aro-ocp419
cron: 0 12 * * 4
reporter_config:
channel: '#quay-qe'
job_states_to_report:
- success
- failure
- error
report_template: '{{if eq .Status.State "success"}} :slack-green: Job *{{.Spec.Job}}*
ended with *{{.Status.State}}*. <{{.Status.URL}}|View logs> {{else}} :failed:
Job *{{.Spec.Job}}* ended with *{{.Status.State}}*. <{{.Status.URL}}|View logs>
{{end}}'
- as: quay-e2e-tests-quay318-aro
cron: 0 12 1 1 *
steps:
cluster_profile: azure-qe
env:
ARO_WORKER_COUNT: "6"
ARO_WORKER_VM_SIZE: Standard_D8s_v3
CSO_CHANNEL: stable-3.14
QBO_CHANNEL: stable-3.14
QUAY_OPERATOR_CHANNEL: stable-3.14
QUAY_VERSION: "3.14"
test:
- ref: quay-tests-deploy-quay-azure
- ref: quay-tests-test-quay-e2e
- ref: quay-tests-qbo-qe-test
- ref: quay-tests-cso-qe-test
workflow: cucushift-installer-rehearse-azure-aro
- as: quay-e2e-tests-quay313-aro-ocp419
cron: 0 12 * * 5
reporter_config:
channel: '#quay-qe'
job_states_to_report:
- success
- failure
- error
report_template: '{{if eq .Status.State "success"}} :slack-green: Job *{{.Spec.Job}}*
ended with *{{.Status.State}}*. <{{.Status.URL}}|View logs> {{else}} :failed:
Job *{{.Spec.Job}}* ended with *{{.Status.State}}*. <{{.Status.URL}}|View logs>
{{end}}'
steps:
cluster_profile: azure-qe
env:
ARO_WORKER_COUNT: "6"
ARO_WORKER_VM_SIZE: Standard_D8s_v3
CSO_CHANNEL: stable-3.13
QBO_CHANNEL: stable-3.13
QUAY_OPERATOR_CHANNEL: stable-3.13
QUAY_VERSION: "3.13"
CSO_CHANNEL: stable-3.18
CSO_SOURCE: fbc-operator-catalog
MULTISTAGE_PARAM_OVERRIDE_QUAY_INDEX_IMAGE: quay.io/redhat-user-workloads/quay-eng-tenant/stable-3-18-v4-21@sha256:5684950962c8d86d0dc82ef502a19664c8ad6f274a4f391145711e8babdc241a
QBO_CHANNEL: stable-3.18
QBO_SOURCE: fbc-operator-catalog
QUAY_OPERATOR_CHANNEL: stable-3.18
QUAY_OPERATOR_SOURCE: fbc-operator-catalog
QUAY_VERSION: "3.18"
test:
- ref: quay-tests-enable-quay-catalogsource
- ref: quay-tests-deploy-quay-azure
- ref: quay-tests-test-quay-e2e
- ref: quay-tests-qbo-qe-test
Expand Down
139 changes: 8 additions & 131 deletions ci-operator/jobs/quay/quay-tests/quay-quay-tests-master-periodics.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7965,7 +7965,7 @@ periodics:
secretName: result-aggregator
- agent: kubernetes
cluster: build03
cron: 0 12 * * 3
cron: 0 12 1 1 *
decorate: true
decoration_config:
skip_cloning: true
Expand All @@ -7978,121 +7978,9 @@ periodics:
ci-operator.openshift.io/cloud-cluster-profile: azure-qe
ci-operator.openshift.io/variant: quay-aro-ocp
ci.openshift.io/generator: prowgen
job-release: "4.19"
pj-rehearse.openshift.io/can-be-rehearsed: "true"
name: periodic-ci-quay-quay-tests-master-quay-aro-ocp-quay-e2e-tests-quay312-aro-ocp419
reporter_config:
slack:
channel: '#quay-qe'
job_states_to_report:
- success
- failure
- error
report_template: '{{if eq .Status.State "success"}} :slack-green: Job *{{.Spec.Job}}*
ended with *{{.Status.State}}*. <{{.Status.URL}}|View logs> {{else}} :failed:
Job *{{.Spec.Job}}* ended with *{{.Status.State}}*. <{{.Status.URL}}|View
logs> {{end}}'
spec:
containers:
- args:
- --gcs-upload-secret=/secrets/gcs/service-account.json
- --image-import-pull-secret=/etc/pull-secret/.dockerconfigjson
- --lease-server-credentials-file=/etc/boskos/credentials
- --oauth-token-path=/usr/local/github-credentials/oauth
- --report-credentials-file=/etc/report/credentials
- --secret-dir=/secrets/ci-pull-credentials
- --target=quay-e2e-tests-quay312-aro-ocp419
- --variant=quay-aro-ocp
command:
- ci-operator
env:
- name: HTTP_SERVER_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
image: quay-proxy.ci.openshift.org/openshift/ci:ci_ci-operator_latest
imagePullPolicy: Always
name: ""
ports:
- containerPort: 8080
name: http
resources:
requests:
cpu: 10m
volumeMounts:
- mountPath: /etc/boskos
name: boskos
readOnly: true
- mountPath: /secrets/ci-pull-credentials
name: ci-pull-credentials
readOnly: true
- mountPath: /secrets/gcs
name: gcs-credentials
readOnly: true
- mountPath: /usr/local/github-credentials
name: github-credentials-openshift-ci-robot-private-git-cloner
readOnly: true
- mountPath: /secrets/manifest-tool
name: manifest-tool-local-pusher
readOnly: true
- mountPath: /etc/pull-secret
name: pull-secret
readOnly: true
- mountPath: /etc/report
name: result-aggregator
readOnly: true
serviceAccountName: ci-operator
volumes:
- name: boskos
secret:
items:
- key: credentials
path: credentials
secretName: boskos-credentials
- name: ci-pull-credentials
secret:
secretName: ci-pull-credentials
- name: github-credentials-openshift-ci-robot-private-git-cloner
secret:
secretName: github-credentials-openshift-ci-robot-private-git-cloner
- name: manifest-tool-local-pusher
secret:
secretName: manifest-tool-local-pusher
- name: pull-secret
secret:
secretName: registry-pull-credentials
- name: result-aggregator
secret:
secretName: result-aggregator
- agent: kubernetes
cluster: build03
cron: 0 12 * * 5
decorate: true
decoration_config:
skip_cloning: true
extra_refs:
- base_ref: master
org: quay
repo: quay-tests
labels:
ci-operator.openshift.io/cloud: azure4
ci-operator.openshift.io/cloud-cluster-profile: azure-qe
ci-operator.openshift.io/variant: quay-aro-ocp
ci.openshift.io/generator: prowgen
job-release: "4.19"
job-release: "4.21"
pj-rehearse.openshift.io/can-be-rehearsed: "true"
name: periodic-ci-quay-quay-tests-master-quay-aro-ocp-quay-e2e-tests-quay313-aro-ocp419
reporter_config:
slack:
channel: '#quay-qe'
job_states_to_report:
- success
- failure
- error
report_template: '{{if eq .Status.State "success"}} :slack-green: Job *{{.Spec.Job}}*
ended with *{{.Status.State}}*. <{{.Status.URL}}|View logs> {{else}} :failed:
Job *{{.Spec.Job}}* ended with *{{.Status.State}}*. <{{.Status.URL}}|View
logs> {{end}}'
name: periodic-ci-quay-quay-tests-master-quay-aro-ocp-quay-e2e-tests-quay317-aro
spec:
containers:
- args:
Expand All @@ -8102,7 +7990,7 @@ periodics:
- --oauth-token-path=/usr/local/github-credentials/oauth
- --report-credentials-file=/etc/report/credentials
- --secret-dir=/secrets/ci-pull-credentials
- --target=quay-e2e-tests-quay313-aro-ocp419
- --target=quay-e2e-tests-quay317-aro
- --variant=quay-aro-ocp
command:
- ci-operator
Expand Down Expand Up @@ -8167,7 +8055,7 @@ periodics:
secretName: result-aggregator
- agent: kubernetes
cluster: build03
cron: 0 12 * * 4
cron: 0 12 1 1 *
decorate: true
decoration_config:
skip_cloning: true
Expand All @@ -8180,20 +8068,9 @@ periodics:
ci-operator.openshift.io/cloud-cluster-profile: azure-qe
ci-operator.openshift.io/variant: quay-aro-ocp
ci.openshift.io/generator: prowgen
job-release: "4.19"
job-release: "4.21"
pj-rehearse.openshift.io/can-be-rehearsed: "true"
name: periodic-ci-quay-quay-tests-master-quay-aro-ocp-quay-e2e-tests-quay314-aro-ocp419
reporter_config:
slack:
channel: '#quay-qe'
job_states_to_report:
- success
- failure
- error
report_template: '{{if eq .Status.State "success"}} :slack-green: Job *{{.Spec.Job}}*
ended with *{{.Status.State}}*. <{{.Status.URL}}|View logs> {{else}} :failed:
Job *{{.Spec.Job}}* ended with *{{.Status.State}}*. <{{.Status.URL}}|View
logs> {{end}}'
name: periodic-ci-quay-quay-tests-master-quay-aro-ocp-quay-e2e-tests-quay318-aro
spec:
containers:
- args:
Expand All @@ -8203,7 +8080,7 @@ periodics:
- --oauth-token-path=/usr/local/github-credentials/oauth
- --report-credentials-file=/etc/report/credentials
- --secret-dir=/secrets/ci-pull-credentials
- --target=quay-e2e-tests-quay314-aro-ocp419
- --target=quay-e2e-tests-quay318-aro
- --variant=quay-aro-ocp
command:
- ci-operator
Expand Down
4 changes: 2 additions & 2 deletions ...rator/step-registry/quay-tests/deploy-quay-azure/quay-tests-deploy-quay-azure-commands.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,8 +78,8 @@ data "azurerm_storage_account_sas" "quayazure" {
file = true
}

start = "2022-07-24"
expiry = "2024-07-25"
start = timestamp()
expiry = timeadd(timestamp(), "8760h")
Comment thread
LiZhang19817 marked this conversation as resolved.

permissions {
read = true
Expand Down
19 changes: 15 additions & 4 deletions ci-operator/step-registry/quay-tests/qbo-qe-test/quay-tests-qbo-qe-test-commands.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,13 @@
#!/bin/bash

set -o nounset
set -o errexit
set -o pipefail

QBO_RC=0

(
set -o errexit

#Install QBO
QBO_CHANNEL="$QBO_CHANNEL"
QBO_SOURCE="$QBO_SOURCE"
Expand Down Expand Up @@ -39,9 +43,9 @@ echo "Quay Bridge Operator is deployed successfully"
#execute sanity test
##Creating OAuth application and token
token=$(set +o pipefail; LC_ALL=C tr -dc A-Za-z0-9 </dev/urandom | head -c 40)
quay_ns=$(oc get quayregistry --all-namespaces | tail -n1 | tr " " "\n" | head -n1)
quay_registry=$(oc get quayregistry -n "$quay_ns" | tail -n1 | tr " " "\n" | head -n1)
quay_app_pod=$(oc -n "$quay_ns" get pods -l quay-component=quay-app -o name | head -n1)
quay_ns=$(oc get quayregistry --all-namespaces -o jsonpath='{.items[0].metadata.namespace}')
quay_registry=$(oc get quayregistry -n "$quay_ns" -o jsonpath='{.items[0].metadata.name}')
quay_app_pod=$(oc -n "$quay_ns" get pods -l quay-component=quay-app -o jsonpath='{.items[0].metadata.name}')

oc -n "$quay_ns" rsh "$quay_app_pod" python <<EOF
from app import app
Expand Down Expand Up @@ -519,3 +523,10 @@ for _ in {1..30}; do
sleep 20
done
echo "QE Test for Quay Bridge Operator is passed"
) || QBO_RC=$?

if [ $QBO_RC -ne 0 ]; then
echo "!!! QBO test failed with exit code $QBO_RC, but continuing to allow subsequent steps to run"
fi

exit 0
Comment on lines +526 to +532

@coderabbitai coderabbitai Bot Jun 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Test failures are masked - consider impact on failure visibility.

The forced exit 0 allows the CSO test step to run, but means QBO test failures won't fail the overall job. The failure is logged, but:

  1. In lengthy CI logs, the message may be missed
  2. The ARO workflow (per context) lacks firewatch configuration that could parse logs for failures
  3. Job dashboards will show this step as "passed" even when tests failed

If this trade-off is acceptable for this use case, consider adding a more prominent failure indicator (e.g., writing failure status to ${SHARED_DIR} for downstream consumption or aggregation).

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/step-registry/quay-tests/qbo-qe-test/quay-tests-qbo-qe-test-commands.sh`
around lines 526 - 532, The script forces exit 0 after a QBO test failure, which
masks the failure from job dashboards and logs. To address this while still
allowing subsequent steps to run, modify the code block where QBO_RC is checked
to write a failure indicator to the shared directory (${SHARED_DIR}) when QBO_RC
is non-zero, so downstream steps or workflow aggregation can detect and properly
report the failure. This way the failure is prominently recorded for failure
visibility without blocking subsequent test steps from executing.