Multi-BMG Level Zero device enumeration broken by deferred internal-engine init (regression NEO 25.40→25.44, Arc Pro B50/B60/B70)

[F3zz1k]

Pre-submission Checklist

• I am using the latest GPU driver version (releases)
• I have searched for similar issues and found none

GPU Hardware

Intel Arc Pro B70

DRI Devices Information

0 crw-rw----+ 1 root video 226, 0 May 2 23:05 /dev/dri/card0
0 crw-rw----+ 1 root video 226, 1 May 2 22:31 /dev/dri/card1
0 crw-rw-rw- 1 root render 226, 128 May 2 22:31 /dev/dri/renderD128
0 crw-rw-rw- 1 root render 226, 129 May 2 22:31 /dev/dri/renderD129

/dev/dri/by-path:
total 0
0 lrwxrwxrwx 1 root root 8 May 2 22:31 pci-0000:43:00.0-card -> ../card1
0 lrwxrwxrwx 1 root root 13 May 2 22:31 pci-0000:43:00.0-render -> ../renderD128
0 lrwxrwxrwx 1 root root 8 May 2 22:31 pci-0000:47:00.0-card -> ../card0
0 lrwxrwxrwx 1 root root 13 May 2 22:31 pci-0000:47:00.0-render -> ../renderD129

GPU Detailed Information (lspci output)

47:00.0 VGA compatible controller: Intel Corporation Battlemage G31 [Intel Graphics] (prog-if 00 [VGA controller])
Subsystem: Intel Corporation Device 1701
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin ? routed to IRQ 60
NUMA node: 1
Region 0: Memory at 7e800000000 (64-bit, prefetchable) [size=16M]
Region 2: Memory at 7e000000000 (64-bit, prefetchable) [size=32G]
Expansion ROM at 82000000 [disabled] [size=2M]
Capabilities: [40] Vendor Specific Information: Intel Capabilities v1
CapA: Peg60Dis- Peg12Dis- Peg11Dis- Peg10Dis- PeLWUDis- DmiWidth=x4
EccDis- ForceEccEn- VTdDis- DmiG2Dis- PegG2Dis- DDRMaxSize=Unlimited
1NDis- CDDis- DDPCDis- X2APICEn- PDCDis- IGDis- CDID=0 CRID=0
DDROCCAP+ OCEn- DDRWrtVrefEn+ DDR3LEn+
CapB: ImguDis- OCbySSKUCap- OCbySSKUEn- SMTCap- CacheSzCap 0x0
SoftBinCap- DDR3MaxFreqWithRef100=Disabled PegG3Dis-
PkgTyp- AddGfxEn- AddGfxCap- PegX16Dis- DmiG3Dis- GmmDis-
DDR3MaxFreq=2932MHz LPDDR3En-
Capabilities: [70] Express (v2) Endpoint, IntMsgNum 0
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W TEE-IO-
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- FltModeDis-
LnkSta: Speed 2.5GT/s, Width x1
TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
AtomicOpsCtl: ReqEn-
IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported, FltMode-
Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
Address: 00000000fee1e000 Data: 0020
Masking: 00000000 Pending: 00000000
Capabilities: [d0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [110 v1] Null
Capabilities: [200 v1] Address Translation Service (ATS)
ATSCap: Invalidate Queue Depth: 00
ATSCtl: Enable-, Smallest Translation Unit: 00
Capabilities: [420 v1] Physical Resizable BAR
BAR 2: current size: 32GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
Capabilities: [220 v1] Virtual Resizable BAR
BAR 2: current size: 8GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
Capabilities: [320 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration- 10BitTagReq+ IntMsgNum 0
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+ 10BitTagReq-
IOVSta: Migration-
Initial VFs: 4, Total VFs: 4, Number of VFs: 0, Function Dependency Link: 00
VF offset: 1, stride: 1, Device ID: e223
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 0000000000000000 (64-bit, prefetchable)
Region 2: Memory at 0000000000000000 (64-bit, prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [400 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Kernel driver in use: xe
Kernel modules: xe

43:00.0 VGA compatible controller: Intel Corporation Battlemage G31 [Intel Graphics] (prog-if 00 [VGA controller])
Subsystem: Intel Corporation Device 1701
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin ? routed to IRQ 58
NUMA node: 1
Region 0: Memory at 7f800000000 (64-bit, prefetchable) [size=16M]
Region 2: Memory at 7f000000000 (64-bit, prefetchable) [size=32G]
Expansion ROM at 82400000 [disabled] [size=2M]
Capabilities: [40] Vendor Specific Information: Intel Capabilities v1
CapA: Peg60Dis- Peg12Dis- Peg11Dis- Peg10Dis- PeLWUDis- DmiWidth=x4
EccDis- ForceEccEn- VTdDis- DmiG2Dis- PegG2Dis- DDRMaxSize=Unlimited
1NDis- CDDis- DDPCDis- X2APICEn- PDCDis- IGDis- CDID=0 CRID=0
DDROCCAP+ OCEn- DDRWrtVrefEn+ DDR3LEn+
CapB: ImguDis- OCbySSKUCap- OCbySSKUEn- SMTCap- CacheSzCap 0x0
SoftBinCap- DDR3MaxFreqWithRef100=Disabled PegG3Dis-
PkgTyp- AddGfxEn- AddGfxCap- PegX16Dis- DmiG3Dis- GmmDis-
DDR3MaxFreq=2932MHz LPDDR3En-
Capabilities: [70] Express (v2) Endpoint, IntMsgNum 0
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W TEE-IO-
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- FltModeDis-
LnkSta: Speed 2.5GT/s, Width x1
TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
AtomicOpsCtl: ReqEn-
IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported, FltMode-
Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
Address: 00000000fee1c000 Data: 0020
Masking: 00000000 Pending: 00000000
Capabilities: [d0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [110 v1] Null
Capabilities: [200 v1] Address Translation Service (ATS)
ATSCap: Invalidate Queue Depth: 00
ATSCtl: Enable-, Smallest Translation Unit: 00
Capabilities: [420 v1] Physical Resizable BAR
BAR 2: current size: 32GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
Capabilities: [220 v1] Virtual Resizable BAR
BAR 2: current size: 8GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
Capabilities: [320 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration- 10BitTagReq+ IntMsgNum 0
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+ 10BitTagReq-
IOVSta: Migration-
Initial VFs: 4, Total VFs: 4, Number of VFs: 0, Function Dependency Link: 00
VF offset: 1, stride: 1, Device ID: e223
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 0000000000000000 (64-bit, prefetchable)
Region 2: Memory at 0000000000000000 (64-bit, prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [400 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Kernel driver in use: xe
Kernel modules: xe

Driver Version

26.05.37020.3-2

Installed GPU Driver Packages

No response

Driver Installation Details

Kernel driver (xe): ships in-kernel with Linux 6.19. Installed via Manjaro's kernel manager:

sudo mhwd-kernel -i linux619                                                                          
# Reboot, select 6.19 from GRUB                                                                       
 
Userspace runtime + Level Zero (Manjaro official extra repo, Arch packages):                          
sudo pacman -S intel-compute-runtime level-zero-loader vulkan-intel \                               
               intel-media-driver intel-gpu-tools linux-firmware                                      
                                                                                                    
Resulting versions:

│                         Package                         │     Version      │
│ intel-compute-runtime (NEO)                             │ 26.05.37020.3-2  │
│ level-zero-loader                                       │ 1.27.0-2         │
│ vulkan-intel (Mesa)                                     │ 26.0.2-1         │
│ linux-firmware-meta (incl. bmg_guc_70.bin, bmg_huc.bin) │ 20260309-1       │
│ linux619 kernel                                         │ 6.19.8-1-MANJARO │
         
                                                                                                    
oneAPI toolkit: installed via Intel's official offline installer (not the Arch package, which is stuck
 at 2025.0.4):
cd /tmp                                                                                               
wget https://registrationcenter-download.intel.com/akdlm/IRC_NAS/99f4837a-25b7-425d-a897-60af022676ea/
intel-oneapi-base-toolkit-2025.3.2.21_offline.sh                                                      
sudo sh intel-oneapi-base-toolkit-2025.3.2.21_offline.sh -a --silent --cli --eula accept              
- Installed to /opt/intel/oneapi/2025.3/ (alongside an earlier 2025.0/ from pacman -S   
intel-oneapi-basekit)                                                                                 
- Activated per-shell via source /opt/intel/oneapi/setvars.sh (setvars.sh auto-resolves latest →      
2025.3)                                                                                               
- which icpx → /opt/intel/oneapi/compiler/2025.3/bin/icpx (version 2025.3.3.20260319)                 
                                                                                                    
Verification after install:                                                                           
$ uname -r                                                                                          
6.19.8-1-MANJARO                                                                                      
                                                                                                      
$ lspci -k | grep -A1 -i battlemage
0000:43:00.0 VGA compatible controller: Intel Corporation Battlemage G31 [Intel Graphics]             
    Kernel driver in use: xe                                                                        
0000:47:00.0 VGA compatible controller: Intel Corporation Battlemage G31 [Intel Graphics]             
    Kernel driver in use: xe                                                                        
                                                                                                      
$ sycl-ls                                                                                             
[level_zero:gpu][level_zero:0] Intel(R) oneAPI Unified Runtime over Level-Zero V2, Intel(R) Graphics  
[0xe223] 20.2.0 [1.14.37020]                                                                          
[level_zero:gpu][level_zero:1] Intel(R) oneAPI Unified Runtime over Level-Zero V2, Intel(R) Graphics  
[0xe223] 20.2.0 [1.14.37020]                                                                        
                                                                                                      
No PPAs, custom builds, or out-of-tree modules. No DKMS. Stock Manjaro repos for the KMD/NEO/loader 
stack + Intel's official offline installer for oneAPI 2025.3.                                         

Linux Distribution

Arch Linux

Other Linux Distribution

No response

Kernel Version & Boot Parameters

6.19.8-1-MANJARO

BOOT_IMAGE=/@/boot/vmlinuz-6.19-x86_64 root=UUID=3c85b363-532c-4253-bf74-194699d38c4e rw rootflags=subvol=@ quiet splash apparmor=1 security=apparmor udev.log_priority=3

xe 4222976 59
intel_vsec 28672 2 pmt_telemetry,xe
drm_ttm_helper 20480 1 xe
ttm 126976 2 drm_ttm_helper,xe
i2c_algo_bit 24576 2 igb,xe
drm_suballoc_helper 16384 1 xe
drm_buddy 32768 1 xe
video 81920 1 xe
gpu_sched 69632 1 xe
drm_gpuvm 57344 1 xe
drm_exec 12288 2 drm_gpuvm,xe
drm_gpusvm_helper 40960 1 xe
drm_display_helper 286720 1 xe
cec 98304 2 drm_display_helper,xe

Actual Behavior

Constructing a SYCL Level Zero context that spans both discrete Arc Pro B70 GPUs throws
UR_RESULT_ERROR_UNKNOWN (2147483646) and terminates. Single-device context construction succeeds,
and sycl-ls enumerates both cards correctly. Symptom shape matches #916 (regression in NEO 26.x for
multi-device L0 contexts), but on Battlemage with a different error code, suggesting failure happens
earlier — during urContextCreate itself rather than first USM allocation. Cross-reference:
intel/llm-scaler#382 (same hardware, downstream symptom inside the Intel-published llm-scaler-vllm
container).

sycl::_V1::exception: level_zero backend failed with error: 2147483646 (UR_RESULT_ERROR_UNKNOWN)
sycl::_V1::context::context(std::vectorsycl::device, ...)
+0x32f783 (throw site, libsycl.so.8)
std::terminate

UR_LOADER_DEBUG=1 trace shows both adapters load and libze_intel_gpu.so.1 loads; no zeDevice* or
zeContext* calls appear in the trace before the abort — failure occurs inside the v2 adapter's
urContextCreate path before it issues a Level Zero call.

[INFO]: loaded adapter (libur_adapter_level_zero.so.0) from .../libur_adapter_level_zero.so.0
[INFO]: loaded adapter (libur_adapter_level_zero_v2.so.0) from .../libur_adapter_level_zero_v2.so.0
zeInit with flags value of 1
[ze_loader] Loader Version 1.27.0
[ze_loader] Loading Driver libze_intel_gpu.so.1 succeeded
[crash]

Expected Behavior

Multi-device context constructs without error; subsequent allocations / queues work, as sycl-ls
already shows both devices are valid.

Reproduction Rate

Always reproduces - 100%

Steps to Reproduce

1. Build llama.cpp with the SYCL backend (commit d05fe1d or newer).
2. With both B70s visible to the runtime, run ./llama-ls-sycl-device.
3. (Equivalent) inside intel/llm-scaler-vllm:latest, run python -c "import torch; torch.xpu.device_count()".
4. (Equivalent, minimal) construct sycl::context(std::vector<sycl::device>{dev0, dev1}) from any
   SYCL hello-world.

Is this a regression?

• Yes, this is a regression - functionality that previously worked is now broken

Last Known Working Driver Version

No response

First Known Failing Driver Version

No response

API Call Logs

No response

strace Logs

No response

System Logs / dmesg Output

No response

Backtrace (if crash or hang occurred)

No response

Source Code / Reproducer

No response

Command Line / Application Details

No response

oneAPI Version (if applicable)

No response

Screenshots / Video

No response

Additional Notes

Workarounds tried that do NOT help (all crash identically)

Variable / change	Effect
ZE_P2P_DISABLE=1	no change
VLLM_SKIP_P2P_CHECK=1	no change
NEOReadDebugKeys=1 CreateMultipleRootDevices=2	no change
ONEAPI_DEVICE_SELECTOR=*:*	no change
ZE_FLAT_DEVICE_HIERARCHY={FLAT,COMPOSITE,COMBINED}	no change
UR_LOADER_USE_LEVEL_ZERO_V2=0	not recognised by loader
UR_L0_ADAPTER_VERSION=1	not recognised
SYCL_UR_USE_LEVEL_ZERO_V2=0	no change (still crashes via v2 path)
IOMMU disabled in BIOS	no change
Move card to different PCIe slot (NUMA + IOMMU groups change)	no change
All of the above combined	no change

Single-device workaround that works

ONEAPI_DEVICE_SELECTOR=level_zero:0 — application sees one card, no multi-device context
constructed, runs cleanly. Identical workaround to #916.

Notes

• Same crash reproduces inside the Intel-published intel/llm-scaler-vllm:latest container, so it is
  not local environment contamination — see also Unable to run Qwen3.6-35B-A3B FP8 on 2x B70s using llm-scaler-vllm:0.14.0-b8.2 llm-scaler#382
  (UR_RESULT_ERROR_OUT_OF_RESOURCES on the same 2×B70 setup, plausibly the same root cause surfacing
  one call later).
• Pattern (single-device OK, ONEAPI_DEVICE_SELECTOR=level_zero:0 works, broke in 26.x) matches
  [GSD-12641] USM device allocation fails with UR_RESULT_ERROR_OUT_OF_DEVICE_MEMORY when Level Zero context spans 2 discrete Arc A770 GPUs (regression from 25.40 to 26.09) #916 (2× Arc A770), but the error code here is UR_RESULT_ERROR_UNKNOWN
  rather than OUT_OF_DEVICE_MEMORY, suggesting the failure is in urContextCreate itself on
  Battlemage, before the first USM allocation that [GSD-12641] USM device allocation fails with UR_RESULT_ERROR_OUT_OF_DEVICE_MEMORY when Level Zero context spans 2 discrete Arc A770 GPUs (regression from 25.40 to 26.09) #916 fails on.
• oneAPI 2025.0.4 (Arch repo intel-oneapi-basekit) does not crash, because that release ships
  only the v1 UR L0 adapter — strongly implicating the v2 adapter / NEO multi-device-context path. (A
  binary built against 2025.3 against the 2025.0.4 v1 adapter has its own ABI mismatch in the alloc
  path; not requesting that be fixed, just noting.)
• SYCL_UR_USE_LEVEL_ZERO_V2=0 does not fall back to the v1 adapter on Xe2 — the v2 adapter is
  selected unconditionally on BMG. If a real fallback knob exists, documenting it would itself be a
  workaround.

Ask

• Confirm whether this is the same root cause as [GSD-12641] USM device allocation fails with UR_RESULT_ERROR_OUT_OF_DEVICE_MEMORY when Level Zero context spans 2 discrete Arc A770 GPUs (regression from 25.40 to 26.09) #916 (NEO 26.x multi-device L0 context regression) or
  a distinct Battlemage path.
• If it is [GSD-12641] USM device allocation fails with UR_RESULT_ERROR_OUT_OF_DEVICE_MEMORY when Level Zero context spans 2 discrete Arc A770 GPUs (regression from 25.40 to 26.09) #916, please link / dup. If distinct, a NEO debug knob to bypass the v2-adapter
  multi-device codepath on BMG would unblock dual-B70 vLLM users today.

[ClancyBuf]

Summary

I am seeing a regression with two Intel Arc Pro B60 GPUs on Fedora 43.

With the Fedora 43 Intel GPU runtime stack at 26.09.37435.12-4.fc43, each GPU works individually when selected with ZE_AFFINITY_MASK=0 or ZE_AFFINITY_MASK=1, but exposing both GPUs together with ZE_AFFINITY_MASK=0,1 causes PyTorch XPU device enumeration to fail inside:

torch._C._xpu_getDeviceCount()

The failure is:

RuntimeError: level_zero backend failed with error: 2147483646 (UR_RESULT_ERROR_UNKNOWN)

Downgrading the Intel GPU runtime stack to 25.31.34666.3-1.fc43 fixes dual-device enumeration: torch._C._xpu_getDeviceCount() returns 2.

Hardware

Two Intel Arc Pro B60 GPUs.

Both appear in sysfs:

/sys/class/drm/renderD128/device
vendor: 0x8086
device: 0xe211
driver: /sys/bus/pci/drivers/xe

/sys/class/drm/renderD129/device
vendor: 0x8086
device: 0xe211
driver: /sys/bus/pci/drivers/xe

OS / environment

Fedora 43 container on Fedora host, using Intel Arc Pro B60 GPUs passed through.

Kernel driver:

xe

Python environment:

Python: 3.12
torch: 2.11.0+xpu

Also reproduced with:

torch: 2.13.0.dev20260503+xpu

Broken package versions

Dual-GPU enumeration fails with:

intel-compute-runtime-26.09.37435.12-4.fc43.x86_64
intel-level-zero-26.09.37435.12-4.fc43.x86_64
intel-level-zero-devel-26.09.37435.12-4.fc43.x86_64
intel-opencl-26.09.37435.12-4.fc43.x86_64
intel-ocloc-26.09.37435.12-4.fc43.x86_64
intel-gmmlib-22.9.0-1.fc43.x86_64
intel-igc-2.30.3-1.fc43.x86_64
intel-igc-libs-2.30.3-1.fc43.x86_64

Working package versions

Dual-GPU enumeration works after downgrading to:

intel-compute-runtime-25.31.34666.3-1.fc43.x86_64
intel-level-zero-25.31.34666.3-1.fc43.x86_64
intel-level-zero-devel-25.31.34666.3-1.fc43.x86_64
intel-opencl-25.31.34666.3-1.fc43.x86_64
intel-ocloc-25.31.34666.3-1.fc43.x86_64
intel-gmmlib-22.8.2-1.fc43.x86_64
intel-igc-2.16.0-1.fc43.x86_64
intel-igc-libs-2.16.0-1.fc43.x86_64

Reproducer

Single GPU 0 works:

source /opt/vllm-xpu/bin/activate

unset SYCL_DEVICE_FILTER
unset ZE_ENABLE_ALT_DRIVERS
unset ZE_FLAT_DEVICE_HIERARCHY

export ONEAPI_DEVICE_SELECTOR=level_zero:gpu
export ZE_AFFINITY_MASK=0

python - <<'PY'
import torch, os
print("ZE_AFFINITY_MASK:", os.environ.get("ZE_AFFINITY_MASK"))
print("torch:", torch.version)
print("xpu available:", torch.xpu.is_available())
print("xpu count:", torch.xpu.device_count())
for i in range(torch.xpu.device_count()):
torch.xpu.set_device(i)
print(i, torch.xpu.get_device_name(i))
PY

Output:

ZE_AFFINITY_MASK: 0
torch: 2.11.0+xpu
xpu available: True
xpu count: 1
0 Intel(R) Arc(TM) Pro B60 Graphics

Single GPU 1 works:

export ZE_AFFINITY_MASK=1

python - <<'PY'
import torch, os
print("ZE_AFFINITY_MASK:", os.environ.get("ZE_AFFINITY_MASK"))
print("torch:", torch.version)
print("xpu available:", torch.xpu.is_available())
print("xpu count:", torch.xpu.device_count())
for i in range(torch.xpu.device_count()):
torch.xpu.set_device(i)
print(i, torch.xpu.get_device_name(i))
PY

Output:

ZE_AFFINITY_MASK: 1
torch: 2.11.0+xpu
xpu available: True
xpu count: 1
0 Intel(R) Arc(TM) Pro B60 Graphics

Both GPUs fail on 26.09:

export ZE_AFFINITY_MASK=0,1

python - <<'PY'
import torch, os, traceback
print("ZE_AFFINITY_MASK:", os.environ.get("ZE_AFFINITY_MASK"))
print("torch:", torch.version)
print("calling torch._C._xpu_getDeviceCount()")
try:
print("count:", torch._C._xpu_getDeviceCount())
except Exception:
traceback.print_exc()
PY

Output on 26.09.37435.12:

ZE_AFFINITY_MASK: 0,1
torch: 2.11.0+xpu
calling torch._C._xpu_getDeviceCount()
RuntimeError: level_zero backend failed with error: 2147483646 (UR_RESULT_ERROR_UNKNOWN)

Both GPUs work on 25.31:

After downgrading to the 25.31.34666.3 package set:

export ZE_AFFINITY_MASK=0,1
export ONEAPI_DEVICE_SELECTOR=level_zero:gpu

python - <<'PY'
import torch
print("torch:", torch.version)
print("count:", torch._C._xpu_getDeviceCount())
for i in range(torch.xpu.device_count()):
print(i, torch.xpu.get_device_name(i))
PY

Output:

torch: 2.11.0+xpu
count: 2
0 Intel(R) Arc(TM) Pro B60 Graphics
1 Intel(R) Arc(TM) Pro B60 Graphics

Native Level Zero device enumeration

A small C Level Zero test using zeInit, zeDriverGet, zeDeviceGet, and zeDeviceGetProperties can see both devices successfully even when ZE_AFFINITY_MASK=0,1:

zeInit: ZE_RESULT_SUCCESS
zeDriverGet count: 1
Driver 0 zeDeviceGet count: 2

Device 0:
name: Intel(R) Arc(TM) Pro B60 Graphics
deviceId: 0xe211
uuid includes PCI/device marker ... 0c ...

Device 1:
name: Intel(R) Arc(TM) Pro B60 Graphics
deviceId: 0xe211
uuid includes PCI/device marker ... 45 ...

So basic Level Zero enumeration appears to work, but the PyTorch/SYCL/UR path fails when creating or initializing the two-device context on 26.09.

Notes

The failure is not that either card is bad. Each card works individually.

This appears to be a regression between:

25.31.34666.3 -> works with two B60s
26.09.37435.12 -> fails with two B60s

Please let me know if this should instead be filed against the Unified Runtime / Intel LLVM SYCL stack. I am filing here first because the behavior tracks the intel-compute-runtime / intel-level-zero package version.

[F3zz1k]

Can confirm, the regression is between NEO 25.40 and 25.48.

[ClancyBuf]

25.44 changed BMG internal engine/context initialization from eager to deferred.
This may have caused Level Zero device creation/enumeration to fail in multi-BMG systems because an internal BCS context was not initialized early enough.

[ClancyBuf]

I think the multi-GPU enumeration regression may be related to the BMG internal engine/context initialization changes between 25.40.35563.4 and 25.44.36015.5.

Working version:

• intel-compute-runtime / intel-level-zero 25.40.35563.4

Broken version:

• intel-compute-runtime / intel-level-zero 25.44.36015.5 / 25.44.36015.8

Observed behaviour:

• System has two Intel BMG discrete GPUs.
• Older runtime enumerates both GPUs correctly.
• Newer 25.44 runtime does not enumerate both GPUs correctly.
• This appears before vLLM/oneCCL involvement, so it looks like a Level Zero / compute-runtime device discovery or device initialization issue.

Suspicious change:
Commit cf3f5d7

Title:
performance: defer internal context init, ptl, bmg

This added/changed BMG/PTL product-helper behaviour so that internal engine initialization is deferred:

bool ProductHelperHw<gfxProduct>::initializeInternalEngineImmediately() const {
    return false;
}

For BMG this is in:

shared/source/xe2_hpg_core/bmg/os_agnostic_product_helper_bmg.inl

For PTL this is in:

shared/source/xe3_core/ptl/os_agnostic_product_helper_ptl.inl

The likely behavioural change is:

Before:

• Internal engines/contexts were initialized immediately during device setup unless otherwise overridden.

After:

• BMG/PTL explicitly defer internal engine/context initialization.

There is also a follow-up commit that looks directly related:

Commit a673c5d

Title:
fix: always initialize internal bcs engine

This changed OsContext::isImmediateContextInitializationEnabled() from effectively:

if (engineUsage == EngineUsage::internal) {
    return initializeInternalEngineImmediately;
}

to:

if (engineUsage == EngineUsage::internal) {
    return initializeInternalEngineImmediately || engineType == aub_stream::EngineType::ENGINE_BCS;
}

File:

shared/source/os_interface/os_context.cpp

That follow-up suggests the original deferred-initialization change may have deferred too much, and that internal BCS still needs to be initialized eagerly.

My theory:

• The 25.44 runtime changed BMG internal engine/context initialization from eager to deferred.
• In a multi-BMG system, the Level Zero device creation/enumeration path may still rely on an internal BCS/context being available during early device setup.
• As a result, the second GPU may fail initialization or be skipped during enumeration.
• The later “always initialize internal bcs engine” fix looks like it may be addressing exactly this class of fallout.

Suggested thing to check:

• Does 25.44.36015.5 / 25.44.36015.8 include the BMG deferred internal context init change without the correct internal BCS eager-init behaviour?
• If so, can the BCS eager-init fix be backported to the affected 25.44 package?
• Alternatively, can BMG temporarily revert initializeInternalEngineImmediately() to true to confirm whether enumeration recovers?

Minimal test patch idea:

if (engineUsage == EngineUsage::internal) {
    return initializeInternalEngineImmediately || engineType == aub_stream::EngineType::ENGINE_BCS;
}

in:

shared/source/os_interface/os_context.cpp

If that restores enumeration of both BMG GPUs, it would strongly indicate the regression is caused by the deferred internal context/engine initialization change rather than PCIe topology, vLLM, or oneCCL.

[ClancyBuf]

After forcing BMG initializeInternalEngineImmediately() back to true and rebuilding compute-runtime 26.09.37435.12, both B60 GPUs enumerate and zeDeviceCanAccessPeer() reports peer access in both directions:

Drivers: 1
Driver 0 devices: 2
Device 0: Intel(R) Arc(TM) Pro B60 Graphics device=0xe211
Device 1: Intel(R) Arc(TM) Pro B60 Graphics device=0xe211
P2P 0 -> 1: ze_result=0x0 canAccess=1
P2P 1 -> 0: ze_result=0x0 canAccess=1

The only source change was:

shared/source/xe2_hpg_core/bmg/os_agnostic_product_helper_bmg.inl

ProductHelperHw::initializeInternalEngineImmediately()

from:
return false;

to:
return true;

This confirms the regression is tied to the BMG deferred internal engine/context initialization path, and not to PCIe topology or Level Zero P2P capability on this system.

[F3zz1k]

I've got claude opus trying to build and test, so far we got it enumerating but not running:

Hardware: 4× Intel Arc Pro B70 (BMG, device id 0xe223), Threadripper 1950X, Arch Linux
Software stack tested with the one-line patch applied
(initializeInternalEngineImmediately() false→true):

compute-runtime    igc           result                                                             
---------------    ----------    -----------------------------------------                        
26.05.37020.3      2.32.7        Built OK (skip-unit-tests). All SYCL                               
                                 workloads crash with                      
                                 UR_RESULT_ERROR_UNKNOWN even on a single                           
                                 device. llama-server --list-devices throws                        
                                 the exception from sycl::context().                                
26.14.37833.4      2.32.7        Built OK. Same crash as 26.05 on single                            
                                 and multi-device workloads.                                        
26.14 + UR_LOADER_USE_LEVEL_ZERO_V2=0                                                               
                   2.32.7        Devices enumerate cleanly (sycl-ls and                             
                                 llama-server --list-devices both report   
                                 4×Arc, ~32 GB free each), BUT model load                           
                                 dies at buffer alloc:                                            
                                   llama_model_load: error loading model:                           
                                   unable to allocate SYCL3 buffer                                  
26.18.38308.1      2.32.7        Won't compile — references IGFX_NVL                                
                                 (Nova Lake) which 2.32.7 doesn't define.                           
                                 Need igc ≥2.33. Will try updating.                                                    

25.40.35563.4 (the previously-recommended pin) is still the only version that
works in production on this rig. The patch is necessary but not
sufficient on 26.05/26.14/26.18 — a separate GPU-memory regression appears
to be tripping buffer allocation even with the L0 V2 loader disabled.

[F3zz1k]

One-line patch unblocks device enumeration but every 26.x compute-runtime we can build against the available igc (2.32.7 or 2.34.4-bin) still hits a separate buffer-allocation regression (unable to allocate SYCL buffer / ZE_RESULT_ERROR_OUT_OF_DEVICE_MEMORY) that the patch doesn't fix and 26.18 (which has plausibly-relevant fixes for that allocation bug in its commit log) won't compile because it references IGFX_NVL, a symbol only present in igc ≥ 2.35 which isn't packaged for Arch yet.
I've never built IGC before but I'll attempt it and see if it changes anything.

[F3zz1k]

4× Intel Arc Pro B70 (BMG-G31, 0xe223), Arch Linux + oneAPI 2025.3. Tried ~14 build combinations of compute-runtime 26.05 / 26.09 / 26.14 / 26.18, with the BMG initializeInternalEngineImmediately() one-line patch, with and without intel-graphics-compiler 2.35.2 built from source (clang+libc++) and intel-gmmlib's igfxfmid.h patched to define IGFX_NVL. Every 26.x driver either fails to compile, crashes at sycl-ls, or aborts at runtime in shared/source/gmm_helper/client_context/gmm_client_context.cpp:50 (ABI mismatch between compute-runtime built against the patched IGFX enum and the installed libgmm.so built without it). The BMG one-liner is necessary but not sufficient — david-bue's SVMAllocsManager::createUnifiedMemoryAllocatiofinding on #916 matches our observations exactly. 25.40.35563.4-4 remains the only working pin on this hardware. Endorsing bisect of dc85f33 ("fix: Disallow USM compression on peer access", NEO-15427) as the prime suspect.

[F3zz1k]

● 4× Arc Pro B70 on 26.05.37020.3 (Manjaro, kernel 7.0.3, xe driver): multi-device sycl::context
construction still throws UR_RESULT_ERROR_UNKNOWN even after reverting dc85f33 ("Disallow USM
compression on peer access") — single-device L0 works fine, so the regression is upstream of
SVMAllocsManager.

I don't think there's anything else I can do. Sticking to intel-compute-runtime 25.40.35563.10-1 until B70 support is fixed.

[Stoney49th]

Same here as well. And thanks for all the info regarding the last working version before it broke. Really a good hint on how to make llama.cpp work again. Is this beeing looked at already? What is the planned timeline for a fix?

[F3zz1k]

@Stoney49th I don't think anyone is actively looking at fixing this yet. Seems Arc Pro is low priority (few people have them) and I'm not familiar enough with this platform to contribute a fix. We will probably have to wait for someone at intel to get involved.