[CI] Give each build_wheels.yaml instance distinct artifact names - #767
Conversation
The nightly workflow invokes build_wheels.yaml twice in a single run: once
against the pinned .cudaq_version and once against CUDA-Q main. Both
instances uploaded artifacts under identical names, so every wheel artifact
appeared twice per run:
8882160104 cudaq-wheels-amd64-cu12 06:17:34 <- pinned (CUDA-Q 90b0f4d8)
8882752498 cudaq-wheels-amd64-cu12 06:45:11 <- main (CUDA-Q 7f0f524a)
download-artifact resolves by name, so the test jobs picked up whichever
instance finished last. That mixes CUDA-QX wheels built against one CUDA-Q
with the libcudaq of another, and the mismatch surfaces as an ImportError on
whatever symbol happens to differ between the two revisions, for example:
libcudaq-qec.so: undefined symbol:
_ZNK5cudaq16quantum_platform13validateQpuIdEm
Because it is a race between two upload jobs, it failed sporadically across
architectures and Python versions rather than consistently.
Add an optional artifact_name_suffix input to build_wheels.yaml, append it to
every artifact name, and pass -cudaq-main from the CUDA-Q main instance in
nightly_tests.yaml. The pinned instance keeps the default empty suffix so its
artifact names are unchanged for pr_workflow.yaml and the artifacts_from_run
path.
Note that artifacts_from_run reuses a prior run's artifacts by name, so it
must be given the same artifact_name_suffix as the run it references.
Signed-off-by: Ben Howe <bhowe@nvidia.com>
anjbur
left a comment
There was a problem hiding this comment.
This looks good to me! For testing, I'd suggest a pared-down manual run of the nightly build. We can comment out unnecessary parts of the workflow (i.e. every non-wheel job), then confirm that the correct artifacts were downloaded by the correct jobs and that any failures are not coming from this race condition. We can also double check that the wheels uploaded by the PR CI don't contain a suffix--that should be sufficient to confirm that artifacts_from_run usage will be unaffected.
Signed-off-by: Ben Howe <bhowe@nvidia.com>
DO NOT MERGE. This commit is meant to be reverted once the manual
verification run is done.
Running the full nightly to verify the artifact-name collision fix costs
roughly two hours and a large runner fleet. This trims it to a
workflow_dispatch that recreates the original failure with ~9 jobs.
nightly_tests.yaml:
- resolve-cudaq-main pins CUDA-Q to 7f0f524a09 rather than resolving
main. That is the ref from nightly run 30883444365 (2026-08-04),
which is post-#5005 and therefore ABI-incompatible with the pinned
.cudaq_version (90b0f4d8, pre-#5005). This recreates the exact skew
that produced "undefined symbol:
_ZNK5cudaq16quantum_platform13validateQpuIdEm".
- build-cudaq-main is disabled. It only populates the /cudaq-install
cache for the lib jobs; the wheel path builds its own CUDA-Q wheels.
This also transitively skips cudaq-main-{all-libs,qec,solvers,docs}.
- cudaq-main-wheels no longer needs build-cudaq-main.
- cleanup-nightly-cache is disabled so it does not delete the warm
"-nightly" caches, keeping the negative-control re-run cheap.
build_wheels.yaml:
- the three wheel matrices are trimmed to python 3.11 / amd64 / cu12.6.
- the GPU wheel test job is disabled; the CPU jobs exercise the fix.
Expected with the fix: every artifact name is unique and both
"Test CUDA-QX wheels (CPU)" jobs pass. Negative control: remove the
"artifact_name_suffix: -cudaq-main" line and re-run; the pinned
instance's test job should then fail on the undefined symbol above.
Every hunk is tagged TEMP(PR767-test); revert this commit before merging.
Signed-off-by: Ben Howe <bhowe@nvidia.com>
… fix" This reverts commit 3d9310d. Signed-off-by: Ben Howe <bhowe@nvidia.com>
Thanks - I took your suggestion and ran a pared-down nightly. Details below. Setup (run 31066167590): trimmed the nightly to just the two Result: green. Six artifacts, six unique names — three matched pairs: On 2026-08-04 every one of those names appeared twice with two competing IDs. Download routing — each test job resolved all three of its downloads to its own instance's IDs. The pinned job (the one that failed on 08-04) took On your second point — confirmed the PR CI wheels carry no suffix. From this PR's own run 31065493462: |
Problem
nightly_tests.yamlinvokesbuild_wheels.yamltwice within a single run:wheels— builds against the pinned.cudaq_versioncudaq-main-wheels— builds against CUDA-QmainBoth instances uploaded artifacts under identical names. Every wheel artifact
appears exactly twice per nightly run — all four
cudaq-wheels-{arch}-cu1{2,3},all twelve
wheels-py3.*, andmetapackages. For example, in run30883444365:
cudaq-wheels-amd64-cu1290b0f4d8cudaq-wheels-amd64-cu127f0f524aactions/download-artifactresolves by name, so the test jobs pick up whicheverinstance finished last. The pinned CPU test job downloaded
8882752498—main's CUDA-Q wheels — while testing CUDA-QX wheels built against the pin.
The ABI mismatch then surfaces as an
ImportErroron whatever symbol happens todiffer between the two CUDA-Q revisions. In the run above, CUDA-Q
#5005 changed a
quantum_platformmember that is called from inline header code, so it leaksinto every downstream
.so:libcudaq-qec.so: undefined symbol: _ZNK5cudaq16quantum_platform13validateQpuIdEm
Because this is a race between two upload jobs, it fails sporadically across
architectures and Python versions rather than consistently — and it is not new.
The Aug 1 nightly failed the same way on a different symbol
(
_ZN14cudaq_internal8compiler13compileModuleE...), simply because that was thesymbol that differed at the time.
Fix
Add an optional
artifact_name_suffixinput tobuild_wheels.yamland append itto every artifact name (4 uploads and their 8 matching downloads across the CPU
and GPU test jobs).
nightly_tests.yamlpasses-cudaq-mainfrom the CUDA-Qmain instance, so the two instances produce disjoint sets:
cudaq-wheels-amd64-cu12 <- pinned (.cudaq_version)
cudaq-wheels-amd64-cu12-cudaq-main <- CUDA-Q main
The pinned instance keeps the default empty suffix, so its artifact names are
unchanged — this matters because
pr_workflow.yamland theartifacts_from_runpath both reference these names.Notes for reviewers
artifacts_from_runrequires a matching suffix. It reuses a prior run'sartifacts by name, so re-running against a nightly that used
-cudaq-mainmeans passing the same
artifact_name_suffix. This is documented in theworkflow_dispatchinput description, and the input is exposed there so it issettable from the UI — but it is a manual step.
get-cudaq-wheels/action.yaml(andget-cudaq-build),restore-keysis thefull cache key minus the
-nightlysuffix, so suffixed and unsuffixed callersshare a cache pool. It is harmless today because the CUDA-Q ref is part of the
key, and it is unrelated to these failures.
Testing
YAML validated for both files. Full verification requires a nightly run or a
manual
workflow_dispatch; this has not yet been exercised end to end.