Skip to content

[CI] Give each build_wheels.yaml instance distinct artifact names - #767

Merged
bmhowe23 merged 6 commits into
NVIDIA:mainfrom
bmhowe23:bmh/fix-nightly-wheel-artifact-collision
Aug 6, 2026
Merged

[CI] Give each build_wheels.yaml instance distinct artifact names#767
bmhowe23 merged 6 commits into
NVIDIA:mainfrom
bmhowe23:bmh/fix-nightly-wheel-artifact-collision

Conversation

@bmhowe23

@bmhowe23 bmhowe23 commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Problem

nightly_tests.yaml invokes build_wheels.yaml twice within a single run:

  • wheels — builds against the pinned .cudaq_version
  • cudaq-main-wheels — builds against CUDA-Q main

Both instances uploaded artifacts under identical names. Every wheel artifact
appears exactly twice per nightly run — all four cudaq-wheels-{arch}-cu1{2,3},
all twelve wheels-py3.*, and metapackages. For example, in run
30883444365:

Artifact ID Name Uploaded Built against
8882160104 cudaq-wheels-amd64-cu12 06:17:34 pinned, CUDA-Q 90b0f4d8
8882752498 cudaq-wheels-amd64-cu12 06:45:11 main, CUDA-Q 7f0f524a

actions/download-artifact resolves by name, so the test jobs pick up whichever
instance finished last. The pinned CPU test job downloaded 8882752498
main's CUDA-Q wheels — while testing CUDA-QX wheels built against the pin.

The ABI mismatch then surfaces as an ImportError on whatever symbol happens to
differ between the two CUDA-Q revisions. In the run above, CUDA-Q
#5005 changed a
quantum_platform member that is called from inline header code, so it leaks
into every downstream .so:

libcudaq-qec.so: undefined symbol: _ZNK5cudaq16quantum_platform13validateQpuIdEm

Because this is a race between two upload jobs, it fails sporadically across
architectures and Python versions rather than consistently — and it is not new.
The Aug 1 nightly failed the same way on a different symbol
(_ZN14cudaq_internal8compiler13compileModuleE...), simply because that was the
symbol that differed at the time.

Fix

Add an optional artifact_name_suffix input to build_wheels.yaml and append it
to every artifact name (4 uploads and their 8 matching downloads across the CPU
and GPU test jobs). nightly_tests.yaml passes -cudaq-main from the CUDA-Q
main instance, so the two instances produce disjoint sets:

cudaq-wheels-amd64-cu12 <- pinned (.cudaq_version)
cudaq-wheels-amd64-cu12-cudaq-main <- CUDA-Q main

The pinned instance keeps the default empty suffix, so its artifact names are
unchanged — this matters because pr_workflow.yaml and the
artifacts_from_run path both reference these names.

Notes for reviewers

  • artifacts_from_run requires a matching suffix. It reuses a prior run's
    artifacts by name, so re-running against a nightly that used -cudaq-main
    means passing the same artifact_name_suffix. This is documented in the
    workflow_dispatch input description, and the input is exposed there so it is
    settable from the UI — but it is a manual step.
  • A related latent issue is not addressed here. In
    get-cudaq-wheels/action.yaml (and get-cudaq-build), restore-keys is the
    full cache key minus the -nightly suffix, so suffixed and unsuffixed callers
    share a cache pool. It is harmless today because the CUDA-Q ref is part of the
    key, and it is unrelated to these failures.

Testing

YAML validated for both files. Full verification requires a nightly run or a
manual workflow_dispatch; this has not yet been exercised end to end.

bmhowe23 and others added 2 commits August 5, 2026 19:27
The nightly workflow invokes build_wheels.yaml twice in a single run: once
against the pinned .cudaq_version and once against CUDA-Q main. Both
instances uploaded artifacts under identical names, so every wheel artifact
appeared twice per run:

  8882160104  cudaq-wheels-amd64-cu12  06:17:34  <- pinned (CUDA-Q 90b0f4d8)
  8882752498  cudaq-wheels-amd64-cu12  06:45:11  <- main   (CUDA-Q 7f0f524a)

download-artifact resolves by name, so the test jobs picked up whichever
instance finished last. That mixes CUDA-QX wheels built against one CUDA-Q
with the libcudaq of another, and the mismatch surfaces as an ImportError on
whatever symbol happens to differ between the two revisions, for example:

  libcudaq-qec.so: undefined symbol:
    _ZNK5cudaq16quantum_platform13validateQpuIdEm

Because it is a race between two upload jobs, it failed sporadically across
architectures and Python versions rather than consistently.

Add an optional artifact_name_suffix input to build_wheels.yaml, append it to
every artifact name, and pass -cudaq-main from the CUDA-Q main instance in
nightly_tests.yaml. The pinned instance keeps the default empty suffix so its
artifact names are unchanged for pr_workflow.yaml and the artifacts_from_run
path.

Note that artifacts_from_run reuses a prior run's artifacts by name, so it
must be given the same artifact_name_suffix as the run it references.

Signed-off-by: Ben Howe <bhowe@nvidia.com>

@anjbur anjbur left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me! For testing, I'd suggest a pared-down manual run of the nightly build. We can comment out unnecessary parts of the workflow (i.e. every non-wheel job), then confirm that the correct artifacts were downloaded by the correct jobs and that any failures are not coming from this race condition. We can also double check that the wheels uploaded by the PR CI don't contain a suffix--that should be sufficient to confirm that artifacts_from_run usage will be unaffected.

Comment thread .github/workflows/build_wheels.yaml Outdated
Comment thread .github/workflows/build_wheels.yaml Outdated
Signed-off-by: Ben Howe <bhowe@nvidia.com>
DO NOT MERGE. This commit is meant to be reverted once the manual
verification run is done.

Running the full nightly to verify the artifact-name collision fix costs
roughly two hours and a large runner fleet. This trims it to a
workflow_dispatch that recreates the original failure with ~9 jobs.

nightly_tests.yaml:
  - resolve-cudaq-main pins CUDA-Q to 7f0f524a09 rather than resolving
    main. That is the ref from nightly run 30883444365 (2026-08-04),
    which is post-#5005 and therefore ABI-incompatible with the pinned
    .cudaq_version (90b0f4d8, pre-#5005). This recreates the exact skew
    that produced "undefined symbol:
    _ZNK5cudaq16quantum_platform13validateQpuIdEm".
  - build-cudaq-main is disabled. It only populates the /cudaq-install
    cache for the lib jobs; the wheel path builds its own CUDA-Q wheels.
    This also transitively skips cudaq-main-{all-libs,qec,solvers,docs}.
  - cudaq-main-wheels no longer needs build-cudaq-main.
  - cleanup-nightly-cache is disabled so it does not delete the warm
    "-nightly" caches, keeping the negative-control re-run cheap.

build_wheels.yaml:
  - the three wheel matrices are trimmed to python 3.11 / amd64 / cu12.6.
  - the GPU wheel test job is disabled; the CPU jobs exercise the fix.

Expected with the fix: every artifact name is unique and both
"Test CUDA-QX wheels (CPU)" jobs pass. Negative control: remove the
"artifact_name_suffix: -cudaq-main" line and re-run; the pinned
instance's test job should then fail on the undefined symbol above.

Every hunk is tagged TEMP(PR767-test); revert this commit before merging.

Signed-off-by: Ben Howe <bhowe@nvidia.com>
… fix"

This reverts commit 3d9310d.

Signed-off-by: Ben Howe <bhowe@nvidia.com>
@bmhowe23

bmhowe23 commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator Author

This looks good to me! For testing, I'd suggest a pared-down manual run of the nightly build. We can comment out unnecessary parts of the workflow (i.e. every non-wheel job), then confirm that the correct artifacts were downloaded by the correct jobs and that any failures are not coming from this race condition. We can also double check that the wheels uploaded by the PR CI don't contain a suffix--that should be sufficient to confirm that artifacts_from_run usage will be unaffected.

Thanks - I took your suggestion and ran a pared-down nightly. Details below.

Setup (run 31066167590): trimmed the nightly to just the two build_wheels.yaml instances — dropped build-cudaq-main and the lib/docs jobs, cut the matrices to py3.11 / amd64 / cu12.6, and disabled the GPU tests. I also pinned resolve-cudaq-main to 7f0f524a09 rather than resolving main. That's the CUDA-Q ref from the 2026-08-04 nightly, which is post-#5005 and therefore ABI-incompatible with our pinned .cudaq_version (90b0f4d8) — so the run recreates the exact skew that produced the original undefined symbol failure. Whole thing took ~35 min. The temp commit is reverted.

Result: green. Six artifacts, six unique names — three matched pairs:

cudaq-wheels-amd64-cu12          cudaq-wheels-amd64-cu12-cudaq-main
metapackages                     metapackages-cudaq-main
wheels-py3.11-amd64-cu12         wheels-py3.11-amd64-cu12-cudaq-main

On 2026-08-04 every one of those names appeared twice with two competing IDs.

Download routing — each test job resolved all three of its downloads to its own instance's IDs. The pinned job (the one that failed on 08-04) took cudaq-wheels-amd64-cu12 = 8953910594, not the main-built 8954227802 that was sitting in the same run under the colliding name previously. No undefined symbol anywhere in the log; suites green at 309 / 64 / 3 passed.

On your second point — confirmed the PR CI wheels carry no suffix. From this PR's own run 31065493462: cudaq-wheels-{amd64,arm64}-cu1{2,3} and metapackages, all unsuffixed, each appearing once. pr_workflow.yaml doesn't pass artifact_name_suffix, so it takes the '' default and names are byte-identical to before — artifacts_from_run against a PR run is unaffected.

@bmhowe23
bmhowe23 requested a review from anjbur August 6, 2026 14:00
@bmhowe23
bmhowe23 enabled auto-merge (squash) August 6, 2026 14:01

@anjbur anjbur left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@bmhowe23
bmhowe23 merged commit a21909f into NVIDIA:main Aug 6, 2026
48 checks passed
@bmhowe23
bmhowe23 deleted the bmh/fix-nightly-wheel-artifact-collision branch August 6, 2026 20:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants