Skip to content

gpu: GPU passthrough test suite - #6963

Open
rh-jugraham wants to merge 12 commits into
autotest:masterfrom
rh-jugraham:gpu-providers
Open

rh-jugraham wants to merge 12 commits into
autotest:masterfrom
rh-jugraham:gpu-providers

Conversation

@rh-jugraham

@rh-jugraham rh-jugraham commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

A comprehensive test suite for gpu passthrough.

GPU passthrough test framework providers

  • provider/gpu/gpu_base.py: GPUTest class with driver setup, VM configuration, NUMA/PCI topology, and NVIDIA driver installation
  • provider/gpu/check_points.py: Verification functions for GPU devices, IOMMU groups, nvidia-smi, and qemu logs
  • provider/sriov/sriov_base.py: SR-IOV base class with NIC hostdev support
  • provider/sriov/sriov_vfio.py: SR-IOV VFIO functions

Based on Dan's PR #6818, updated by Julia Graham

Basic GPU passthrough tests

  • single_passthrough_numa_pci_topo: Verify GPU can be passthrough to the guest with various PCI topology and numa topology
  • vm_cuda_sanity: Test Cuda app with single passthrough GPU in the guest
  • lifecycle_vm_with_gpu: Verify guest lifecycle with one GPU and one NIC device passthrough

single_passthrough_numa_pci_topo based on Dan's PR #6819; vm_cuda_sanity, lifecycle_vm_with_gpu based on Dan's unpublished work; all updated by Julia Graham

Advanced multi-GPU and multi-device tests

  • llm_workload: Run workload (like LLM) with GPU in the guest
  • interconnect_stress: Run an interconnect and stress test for multiple GPUs
  • multiple_devices_passthrough: Verify multiple devices (single GPU + single NIC) can passthrough to the guest
  • multiple_gpu_passthrough: Verify multiple GPU could be passthrough to one guest
  • emulated_nic_with_gpu: Verify emulated device with smmuv3 accel off can work with GPU passthrough
  • two_vms_gpu_passthrough: Verify two guests with iommufd hostdev devices can work well

multiple_devices_passthrough based on Dan's unpublished work, updated by Julia Graham
llm_workload, interconnect_stress, multiple_gpu_passthrough, emulated_nic_with_gpu, two_vms_gpu_passthrough authored by Julia Graham

Special and edge case GPU tests

  • plug_unplug_vfio_dev_with_gpu_passthrough: Verify one NIC can be hotplug/unplug to the guest which has a passthrough GPU
  • vm_mig_sanity: Sanity MIG test with GPU passthrough
  • invalid_numa_config: Verify assigning GPU to special NUMA nodes should be rejected
  • invalid_smmuv3_config: Verify error message can be reported against invalid xml configuration
  • memory_hotplug_with_gpu_passthrough: Verify memory device can be hotplugged to a guest with single GPU and vCMDQ enabled
  • default_smmuv3_attrs: Verify GPU can be passthrough with default attribute values in case smmuv3 accel on

plug_unplug_vfio_dev_with_gpu_passthrough based on Dan's PR #6855; vm_mig_sanity based on Dan's unpublished work; all updated by Julia Graham
invalid_numa_config, invalid_smmuv3_config, memory_hotplug_with_gpu_passthrough, default_smmuv3_attrs authored by Julia Graham


Signed-off-by: Julia Graham jugraham@redhat.com
Co-Authored-By: Claude Sonnet 4.5 noreply@anthropic.com

Depends on avocado-framework/avocado-vt#4422, avocado-framework/avocado-vt#4423, and avocado-framework/avocado-vt#4424

rh-jugraham and others added 4 commits November 10, 2026 23:49
- provider/gpu/gpu_base.py: GPUTest class with driver setup, VM configuration, NUMA/PCI topology, and NVIDIA driver installation
- provider/gpu/check_points.py: Verification functions for GPU devices, IOMMU groups, nvidia-smi, and qemu logs
- provider/sriov/sriov_base.py: SR-IOV base class with NIC hostdev support
- provider/sriov/sriov_vfio.py: SR-IOV VFIO functions

Based on Dan's PR autotest#6818, updated by Julia Graham

Signed-off-by: Julia Graham <jugraham@redhat.com>
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- single_passthrough_numa_pci_topo: Verify GPU can be passthrough to the guest with various PCI topology and numa topology
- vm_cuda_sanity: Test Cuda app with single passthrough GPU in the guest
- lifecycle_vm_with_gpu: Verify guest lifecycle with one GPU and one NIC device passthrough

single_passthrough_numa_pci_topo based on Dan's PR autotest#6819; vm_cuda_sanity, lifecycle_vm_with_gpu based on Dan's unpublished work; all updated by Julia Graham

Signed-off-by: Julia Graham <jugraham@redhat.com>
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
llm_workload: Run workload (like LLM) with GPU in the guest
interconnect_stress: Run an interconnect and stress test for multiple GPUs
multiple_devices_passthrough: Verify multiple devices (single GPU + single NIC) can passthrough to the guest
multiple_gpu_passthrough: Verify multiple GPU could be passthrough to one guest
emulated_nic_with_gpu: Verify emulated device with smmuv3 accel off can work with GPU passthrough
two_vms_gpu_passthrough: Verify two guests with iommufd hostdev devices can work well

multiple_devices_passthrough based on Dan's unpublished work, updated by Julia Graham
llm_workload, interconnect_stress, multiple_gpu_passthrough, emulated_nic_with_gpu, two_vms_gpu_passthrough authored by Julia Graham

Signed-off-by: Julia Graham <jugraham@redhat.com>
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
plug_unplug_vfio_dev_with_gpu_passthrough: Verify one NIC can be hotplug/unplug to the guest which has a passthrough GPU
vm_mig_sanity: Sanity MIG test with GPU passthrough
invalid_numa_config: Verify assigning GPU to special NUMA nodes should be rejected
invalid_smmuv3_config: Verify error message can be reported against invalid xml configuration
memory_hotplug_with_gpu_passthrough: Verify memory device can be hotplugged to a guest with single GPU and vCMDQ enabled
default_smmuv3_attrs: Verify GPU can be passthrough with default attribute values in case smmuv3 accel on

plug_unplug_vfio_dev_with_gpu_passthrough based on Dan's PR autotest#6855; vm_mig_sanity based on Dan's unpublished work; all updated by Julia Graham
invalid_numa_config, invalid_smmuv3_config, memory_hotplug_with_gpu_passthrough, default_smmuv3_attrs authored by Julia Graham

Signed-off-by: Julia Graham <jugraham@redhat.com>
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Use raw string for regex pattern to avoid invalid escape sequence warning.

Signed-off-by: Julia Graham <jugraham@redhat.com>
- Remove trailing whitespace from blank lines
- Fix continuation line indentation

Signed-off-by: Julia Graham <jugraham@redhat.com>
- Remove trailing whitespace from blank lines and end of lines
- Fix continuation line indentation
- Remove unused import os

Signed-off-by: Julia Graham <jugraham@redhat.com>
- Remove unused imports (viommu_base, sriov_vfio, exceptions, utils_path)
- Fix blank line spacing (E302, E303)
- Fix operator spacing (E225)

Signed-off-by: Julia Graham <jugraham@redhat.com>
Signed-off-by: Julia Graham <jugraham@redhat.com>
@crobinso

Copy link
Copy Markdown
Collaborator

please combine the other 3 PRs with this one, since they are dependencies. keep them all as separate commits, just one multi-commit PR.

Please drop a comment when the avocado-vt bits are merged so that we know this side is unblocked. thanks!

Consolidating all GPU test PRs into single multi-commit PR per
reviewer request.

Contains basic GPU passthrough tests: TC1, TC2, TC7, TC17
Consolidating all GPU test PRs into single multi-commit PR per
reviewer request.

Contains advanced multi-GPU and multi-device tests: TC3, TC4, TC5,
TC8, TC9, TC10, TC15
Consolidating all GPU test PRs into single multi-commit PR per
reviewer request.

Contains special and edge case GPU tests: TC6, TC11, TC12, TC14, TC16
@rh-jugraham

Copy link
Copy Markdown
Contributor Author

please combine the other 3 PRs with this one, since they are dependencies. keep them all as separate commits, just one multi-commit PR.

Please drop a comment when the avocado-vt bits are merged so that we know this side is unblocked. thanks!

Thanks for taking a look at this PR! Had separated them as I didn't want to create one mega PR, but if that is not a problem, I just merged them all into this PR and updated the title + description.

Two of the avocado-vt PRs were merged, and only avocado-framework/avocado-vt#4424 is left as of writing this comment.

Also, although many of these tests were run during CTC testing in August and the rest were run earlier this month as I was writing them, I will be running this entire test suite next week (week of Nov 16) for the next round of CTC testing and will update this PR with the results. So will be able to show more current results of the tests passing. And, I would hope I won't need to make any changes but if I find anything, I will update this PR accordingly.

@rh-jugraham rh-jugraham changed the title provider/gpu: Add GPU passthrough test framework providers gpu: GPU passthrough test suite Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants