From 3b48f1851613f78b793b72eab7c00438c9153fcb Mon Sep 17 00:00:00 2001 From: Jimmy Song Date: Sun, 20 Sep 2026 18:25:12 +0800 Subject: [PATCH 01/13] Add Lab 17: Ascend NPU Sharing with HAMi DRA - Introduced Lab 17 documentation detailing the installation of HAMi DRA 0.2.3 and the Ascend DRA driver on an Ascend 310P3 node. - Updated the overview and sidebar to include Lab 17, highlighting its focus on request-to-ResourceClaim conversion, two-Pod NPU sharing, memory quota enforcement, and scheduler capacity accounting. - Enhanced the tutorials overview to reflect the addition of Lab 17 and its objectives. Signed-off-by: Jimmy Song --- blog/hami-dra-npu-sharing/index.md | 145 ++++ .../hami-dra-npu-sharing/index.md | 145 ++++ .../docusaurus-plugin-content-blog/tags.yml | 3 + .../current/labs/ascend-hami-dra.md | 790 ++++++++++++++++++ .../current/overview.md | 1 + sidebars-tutorials.js | 5 + tutorials/labs/ascend-hami-dra.md | 790 ++++++++++++++++++ tutorials/overview.md | 1 + 8 files changed, 1880 insertions(+) create mode 100644 blog/hami-dra-npu-sharing/index.md create mode 100644 i18n/zh/docusaurus-plugin-content-blog/hami-dra-npu-sharing/index.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md create mode 100644 tutorials/labs/ascend-hami-dra.md diff --git a/blog/hami-dra-npu-sharing/index.md b/blog/hami-dra-npu-sharing/index.md new file mode 100644 index 000000000..6440b7189 --- /dev/null +++ b/blog/hami-dra-npu-sharing/index.md @@ -0,0 +1,145 @@ +--- +title: "From HAMi to HAMi-DRA: The Evolution of Heterogeneous Compute Management" +date: "2026-09-20" +description: "What HAMi DRA is, why HAMi needs it, and how it is designed: the four bottlenecks of the Device Plugin era (quota accounting, preemption compatibility, scheduling performance, deadlocks), the HAMi-DRA webhook conversion, the DRA driver ecosystem timeline, the role HAMi-core keeps, and adoption guidance with real-cluster examples." +authors: [rootsongjc] +tags: ["HAMi", "DRA", "Ascend", "NPU Sharing", "Kubernetes"] +--- + +Kubernetes 1.34 took the DRA (Dynamic Resource Allocation) core APIs to GA, and together with the Consumable Capacity model, accelerators became first-class objects in Kubernetes' native resource model: they have attributes, they have capacity, and multiple Pods may share them. For sharing stacks built on Device Plugin, like HAMi, this raises a practical question: requesting and scheduling devices can now be done natively by Kubernetes, but isolating resources inside a container is still something Kubernetes does not handle. + +[HAMi DRA](https://github.com/Project-HAMi/HAMi-DRA) is the HAMi community's answer: a migration layer that automatically converts the HAMi-style resource requests in existing workloads into native ResourceClaims, hands scheduling and accounting back to kube-scheduler, keeps runtime isolation with HAMi-core, and leaves the business side without a single line to change. The current 0.2.3 release covers NVIDIA GPUs; with the [Ascend DRA driver](https://github.com/4pdOss/hami-dra-driver) entering preview, the chain has been verified end to end on a real 310P3 cluster, and the community shipped the companion [Lab 17: Ascend NPU Sharing with HAMi DRA](/tutorials/labs/ascend-hami-dra) with every command and real output. This post is a systematic introduction to HAMi DRA: its motivation, design, usage, and current boundaries. + + + +## What HAMi solved, and where it hit the ceiling + +HAMi (Heterogeneous AI Computing Virtualization Middleware), formerly k8s-vGPU-scheduler, is now a CNCF incubating project. The problem it solves is concrete: the Device Plugin's extended resources can only express whole cards (`nvidia.com/gpu: 1`, `huawei.com/Ascend310P: 1`), while sharing an accelerator means "part of one card's memory plus part of its compute". On top of the old API, HAMi built a complete stack: a mutating webhook rewrites Pods, a scheduler extender does shared scheduling, device plugins report and mount, and HAMi-core (`libvgpu` / `libvnpu`) enforces quotas inside containers. The stack has held up in production (case figures from a [community talk](https://aicon.infoq.cn/2026/shenzhen/presentation/7168)): SF Express achieved up to 57% GPU savings through GPU sharing, SNOW cut costs by 55%, and China Merchants Bank reached 100% hardware pool utilization with topology-aware scheduling. + +But precisely because the whole stack sits on Device Plugin and scheduler extension, the ceiling is structural. The real-world bottlenecks of the existing HAMi approach group into four (see the [community talk](https://aicon.infoq.cn/2026/shenzhen/presentation/7168)): + +| Bottleneck | Root cause | +| :-- | :-- | +| ResourceQuota accounting | extended resources stay outside the native quota system and need a separate quota implementation | +| Preemption compatibility | Extender API limitations make Kubernetes scheduling features such as preemption awkward | +| Scheduling performance | node-lock exclusivity: scheduling several Pods onto one node degrades concurrency to serial | +| Deadlocks | the default lock window is 5 minutes; a failed request during creation can leave a node locked | + +None of these are bugs in the code; they are architectural limits: doing fine-grained allocation on a node-centric allocation model comes with exactly these costs. + +## What DRA changed + +Kubernetes 1.34 took the DRA (Dynamic Resource Allocation) core APIs to GA, and cluster device management moved from a node-centric allocation model to a declarative model centered on resource objects. Four new objects: + +- **ResourceSlice**: the device inventory. Each accelerator is one device with attributes (uuid, model, PCIe, and so on) and capacity (memory, compute). +- **ResourceClaim**: the Pod's device demand, expressed and constrained before the Pod is created. +- **DeviceClass**: filtering and configuration templates for devices. +- **ResourceClaimTemplate**: lets controllers such as StatefulSet generate claims in bulk. + +The key enabler for shared scheduling is **Consumable Capacity** (KEP-5075): devices publish capacity along two independent dimensions (memory and a compute percentage), `allowMultipleAllocations` lets several claims consume one device, and kube-scheduler reconciles consumed capacity dimension by dimension, with each allocation carrying an independent shareID. "Multiple workloads fairly consuming parts of one accelerator" becomes a native scheduler capability for the first time, with no out-of-tree component simulating it. Note the feature gate needs manual enabling on 1.34/1.35 and is on by default from 1.36. + +The traditional Device Plugin interface can only expose simple capacity, and the scheduler discovers "not enough resource" only after landing on a node; with DRA, device attributes are matched precisely at scheduling time. The migration logic here is straightforward: the bottleneck was never HAMi's scheduling algorithm, but the API layer beneath it. + +## The design of HAMi DRA + +HAMi DRA does not start over; it replaces only the upper part: scheduling and accounting move to the Kubernetes-native implementation, and runtime isolation stays with HAMi-core. + +```mermaid +%% title: Component split of the HAMi DRA path +flowchart TD + POD["User Pod
HAMi-style resource request"] --> HOOK["HAMi-DRA Webhook
creates and reclaims ResourceClaims"] + HOOK --> DRA["Kubernetes DRA
ResourceClaim / DeviceClass
ResourceSlice / kube-scheduler"] + DRA --> DRV["DRA driver
kubelet plugin + CDI
discovery and injection"] + DRV --> CORE["HAMi-core
memory quota / compute time slice
libvgpu / libvnpu interception"] + CORE --> ACC["physical accelerator
shared by multiple Pods"] +``` + +Three components, three responsibilities: the **HAMi-DRA Webhook** decides "how the request is declared", the **DRA driver** decides "how a device is allocated", and **HAMi-core** decides "how a device is shared". + +The **webhook** does two things. First, automatic multi-dimensional resource conversion: it turns HAMi's multi-dimensional resource names (for example `huawei.com/Ascend310P` plus `-memory` plus `-core`, or on the NVIDIA side `nvidia.com/gpu` plus `gpumem` plus `gpucores`) into the ResourceClaim's `count` and `capacity.requests`, and HAMi's card-selection annotations (`use-*-uuid` and friends) into CEL selectors. Second, ResourceClaim lifecycle management: claims are created bound to the Pod and cleaned up when the Pod is deleted. Notably, HAMi-DRA itself no longer contains a scheduler component and works with the native kube-scheduler as well as Volcano and others. + +The **DRA driver** is the node-side implementation: it discovers devices and publishes the ResourceSlice (attributes plus capacity), handles kubelet's `NodePrepareResources` to inject environment variables and create shared directories, and uses `UnprepareResourceClaims` to clean up HAMi-core's temporary files at the right moment, which is part of why device lifecycle management is more complete than in the Device Plugin era. Device injection reaches containerd through the standard CDI interface. + +**HAMi-core** stays as it is: DRA only addresses the request-and-scheduling half; the enforce-inside-the-container half (memory quotas, compute time slices) remains with `libvgpu` / `libvnpu`. That is exactly where the conclusion of [Does Kubernetes DRA Replace HAMi?](/blog/does-kubernetes-dra-replace-hami) lands architecturally. + +The scheduling-performance gain comes from the same replacement (the [AICon](https://aicon.infoq.cn/2026/shenzhen/presentation/7168) and [KCD Beijing](https://dynamia.ai/zh/blog/kcd-beijing-2026-dra-gpu-scheduling) talks give mutually confirming comparisons): by riding DRA's own scheduling, HAMi-DRA avoids the node-lock performance decay; when several Pods land on the same node concurrently, the original approach's Pod creation time degrades nearly linearly (peaking in the 42-second range), while HAMi-DRA stays clearly lower (by more than 30%). This owes mainly to DRA's resource pre-binding: the allocation is settled at scheduling time, which removes scheduling conflicts and retries. The overall change summarizes as four mappings: NodeLock to No Locks, ResourceName to ResourceClaim, Device Plugin to DRA Driver, and Annotation to Attributes. + +Another easily underrated change is observability. In the traditional model, resource information comes from Nodes and usage from Pods, and a complete resource view has to be assembled by inference; in the DRA model, ResourceSlice describes the device inventory and ResourceClaim the allocations, so the resource view is itself a first-class citizen. Observability moves from "inferring" to "directly modeling" ([community talk](https://dynamia.ai/zh/blog/kcd-beijing-2026-dra-gpu-scheduling)): operators can read from a ResourceClaim who occupies which accelerator, how much memory was allocated, and how much is left, instead of reverse-engineering it from node status and Pod specs. + +## Two usage modes + +One cost of DRA's capability upgrade is often overlooked: the writing complexity, the often-mentioned "UX regression" ([community talk](https://dynamia.ai/zh/blog/kcd-beijing-2026-dra-gpu-scheduling)). The Device Plugin style is one line: + +```yaml +resources: + limits: + nvidia.com/gpu: 1 +``` + +The native DRA style is a separate ResourceClaim object with nested `count` and `capacity.requests`, plus a CEL selector filtering by device attributes: + +```yaml +spec: + devices: + requests: + - exactly: + allocationMode: ExactCount + count: 1 + capacity: + requests: + memory: 4194304k +``` + +For an enterprise already on Device Plugin, the migration cost is not just rewriting YAML; the whole team has to learn a new resource-declaration paradigm. HAMi-DRA's answer is two submission styles with the same scheduling and slicing underneath: + +- **DRA native mode**: create the ResourceClaim by hand declaring memory and compute, and reference it from the Pod via `resourceClaims`. Suited to new workloads; CEL selectors express device-attribute constraints directly (topology awareness, NUMA affinity; the same mechanism extends to NICs and other resources, and `firstAvailable` expresses multiple candidate allocation plans, see the [community talk](https://aicon.infoq.cn/2026/shenzhen/presentation/7168)). +- **DevicePlugin-compatible mode**: keep the traditional `nvidia.com/gpu` / `huawei.com/Ascend310P` resource syntax; the webhook intercepts and converts it into a ResourceClaim. Existing workloads migrate with zero changes, which is where "seamless migration" comes from: the design turns DRA from an "expert interface" into an "ordinary user interface" ([community talk](https://dynamia.ai/zh/blog/kcd-beijing-2026-dra-gpu-scheduling)). + +In compatible mode the conversion happens at admission. For a Pod submitted to a real Ascend 310P cluster, the three `huawei.com/*` entries in `resources.limits` are replaced by `resources.claims` and `spec.resourceClaims`, and in the generated ResourceClaim HAMi semantics enter DRA semantics field by field: + +| ResourceClaim field | Comes from | Conversion | +| :-- | :-- | :-- | +| `count: 1` | `Ascend310P: 1` | integer passed through | +| `capacity.requests.memory: 8589934592` | `-memory: 8192` (MiB) | MiB to bytes (8192 × 1024 × 1024) | +| `capacity.requests.cores: "50"` | `-core: 50` | percent passed through | +| CEL selectors | `use-Ascend310P-uuid` annotation | `uuid in ["..."]`, the value taken from the ResourceSlice | + +When two Pods each request 8192 MiB plus 50 cores against the same card, both claims land on the same device with independent shareIDs, and the scheduler debits the first one's consumption before allocating the second. Once capacity is fully debited, the next request is refused by the scheduler itself (the claim stays pending); when a Pod is deleted, its claim is reclaimed automatically and the accounting recovers. In-container enforcement is still HAMi-core: `libvnpu.so` intercepts over-quota memory requests and turns them into an in-container OOM, and the device as seen from the container is virtualized to the requested 8 GiB rather than the card's 21 GiB. The full walkthrough is [Lab 17](/tutorials/labs/ascend-hami-dra). + +## The device support timeline, and "why now" + +What HAMi DRA can do is bounded by each vendor's DRA driver progress. The timeline shared by the community ([AICon talk](https://aicon.infoq.cn/2026/shenzhen/presentation/7168)): + +- 2025.09: NVIDIA DRA driver, Consumable Capacity ready +- 2026.03: Hygon DCU DRA driver integrated with HAMi DRA +- 2026.04: Enflame DRA driver, integration under way +- 2026.08: the **Ascend DRA driver entered development and preview** ([4pdOss/hami-dra-driver](https://github.com/4pdOss/hami-dra-driver), Paradigm's open-source repository) + +The Ascend path entering preview means HAMi DRA's reach has just extended to NPUs: [Lab 17](/tutorials/labs/ascend-hami-dra) has already run the complete chain from HAMi request to NPU sharing on a 310P3 server, and the Ascend DRA driver is currently chart 0.1.1 (the lab used the development build fixing uuid generation; behavior may change before an official release). The HAMi 2.9 release webinar mentioned that NPU DRA support is planned to ship with 2.10, with test builds released to the community beforehand; the window for trying it and giving feedback is now. + +## The practical constraints before adoption + +The adoption friction groups into four: + +- **High version dependency**: Consumable Capacity requires Kubernetes 1.34+, other DRA features possibly higher, and driving a production upgrade is not easy. +- **Limited vendor support**: few ready-to-use DRA drivers exist, and the set of supported heterogeneous devices is still small. +- **Missing scoring**: DRA lacks scoring at the device-scheduling layer, limiting support for specialized scheduling requirements. +- **Learning cost**: DRA's new concepts carry a threshold for the people deploying and maintaining clusters; the HAMi community's countermeasure is precisely the webhook-compatible mode keeping that cost out of the user's sight. + +On choosing between them (guidance from the [HAMi 2.9 release webinar](https://dynamia.ai/zh/blog/hami-2.9-webinar-recap)): for standardized clusters chasing the best scheduling outcome (topology-aware scheduling, for example), plain HAMi is recommended; for highly customized clusters (already running their own scheduler) or plug-and-play device reuse, HAMi DRA is recommended. Two operational facts are also worth remembering: DRA mode and the traditional mode are mutually exclusive, do not enable both; and DRA mode still goes through HAMi-core. + +## Outlook + +On the HAMi-DRA roadmap: more heterogeneous devices (MetaX, Iluvatar CoreX), more DRA feature adoption (List Attributes, Partitionable Devices), compatibility with the Kubernetes-standard device attribute `resource.kubernetes.io/pcieRoot`, and scheduler integrations with Volcano and Kueue. + +Zooming out ([community talk](https://dynamia.ai/zh/blog/kcd-beijing-2026-dra-gpu-scheduling)): Kubernetes is evolving into the control plane of AI infrastructure, and HAMi's position is the accelerator resource layer for Kubernetes, adapting heterogeneous devices downward, supporting training, inference, and agentic workloads upward, and providing scheduling, virtualization, and resource abstraction in between. HAMi-DRA is the step that aligns this resource layer with the Kubernetes-native model: HAMi's value converges onto runtime enforcement and heterogeneous ecosystem adaptation, while resource expression and scheduling move to the native model. If you are evaluating a migration from the Device Plugin model to DRA, or looking for a unified resource layer over a heterogeneous cluster, try [HAMi DRA](https://github.com/Project-HAMi/HAMi-DRA) and tell the community what you find. + +## Further reading + +- Yang Shouren at AICon Shenzhen 2026: [From HAMi to HAMi-DRA (talk, in Chinese)](https://aicon.infoq.cn/2026/shenzhen/presentation/7168) +- Wang Jifei and James Deng at KCD Beijing 2026: [From Device Plugin to DRA: GPU Scheduling Paradigm Shift and HAMi-DRA in Practice (recap, in Chinese)](https://dynamia.ai/zh/blog/kcd-beijing-2026-dra-gpu-scheduling) +- HAMi 2.9 release webinar recap by Li Mengxuan (in Chinese): [HAMi 2.9 Ascend soft slicing and DRA in depth](https://dynamia.ai/zh/blog/hami-2.9-webinar-recap) +- Community getting-started guide (in Chinese): [HAMi meets Kubernetes DRA: a practical guide](https://dynamia.ai/zh/blog/hami-dra-quickstart) +- Mesut Oezdil: [Does Kubernetes DRA Replace HAMi?](/blog/does-kubernetes-dra-replace-hami) ([CNCF blog original](https://www.cncf.io/blog/2026/08/07/does-kubernetes-dra-replace-hami/)) +- User guide: [How to use HAMi DRA](/docs/installation/how-to-use-hami-dra); hands-on: [Lab 17: Ascend NPU Sharing with HAMi DRA](/tutorials/labs/ascend-hami-dra) and [Lab 4: GPU Slicing with Dynamic Resource Allocation](/tutorials/labs/hami-dra) +- Components: [Project-HAMi/HAMi-DRA](https://github.com/Project-HAMi/HAMi-DRA) · [4pdOss/hami-dra-driver](https://github.com/4pdOss/hami-dra-driver) · [Project-HAMi/hami-vnpu-core](https://github.com/Project-HAMi/hami-vnpu-core) diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-dra-npu-sharing/index.md b/i18n/zh/docusaurus-plugin-content-blog/hami-dra-npu-sharing/index.md new file mode 100644 index 000000000..4f493a421 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-blog/hami-dra-npu-sharing/index.md @@ -0,0 +1,145 @@ +--- +title: "从 HAMi 到 HAMi-DRA：异构算力资源管理的演进" +date: "2026-09-20" +description: "HAMi DRA 是什么、为什么需要它、如何设计：回顾 Device Plugin 时代 HAMi 的四大瓶颈（配额计量、抢占兼容、调度性能、死锁），解析 HAMi-DRA 的 Webhook 转换、DRA Driver 生态时间线与 HAMi-core 的保留角色，并给出选型建议与真实集群中的运行示例。" +authors: [rootsongjc] +tags: ["HAMi", "DRA", "Ascend", "NPU 共享", "Kubernetes"] +--- + +Kubernetes 1.34 将 DRA（Dynamic Resource Allocation）核心 API 推进至 GA，配合 Consumable Capacity（可消费容量）模型，加速器第一次成为 Kubernetes 原生资源模型里的“一等公民”：有属性、有容量、允许多个 Pod 共享。这对 HAMi 这类基于 Device Plugin 的共享方案提出了一个现实问题：设备的申请和调度可以交给 Kubernetes 原生完成，但容器内的资源隔离，Kubernetes 并不负责。 + +[HAMi DRA](https://github.com/Project-HAMi/HAMi-DRA) 是 HAMi 社区交出的答案：一个迁移层，把存量 workload 里的 HAMi 风格资源请求自动转换成原生 ResourceClaim，调度与记账交还 kube-scheduler，运行时隔离继续由 HAMi-core 完成，业务侧一行代码不用改。当前 0.2.3 版本覆盖 NVIDIA GPU；随着[昇腾 DRA Driver](https://github.com/4pdOss/hami-dra-driver) 进入预览，这条链路已在真实的 310P3 集群上完成端到端验证，社区同步发布了配套的 [实验 17：用 HAMi DRA 共享昇腾 NPU](/zh/tutorials/labs/ascend-hami-dra)，提供每一步的命令与真实输出。本文系统介绍 HAMi DRA 的动机、设计、使用方式与当前边界。 + + + +## HAMi 解决了什么，又卡在哪里 + +HAMi（异构 AI 计算虚拟化中间件）的前身是 k8s-vGPU-scheduler，现在是 CNCF 孵化项目。它解决的问题很具体：Device Plugin 的 Extended Resource 只能表达整数张卡（`nvidia.com/gpu: 1`、`huawei.com/Ascend310P: 1`），而加速器共享需要的是“一块卡的一部分显存加一部分算力”。HAMi 在旧 API 之上搭起了完整体系：mutating webhook 改写 Pod、调度器扩展（extender）做共享调度、device plugin 上报与挂载、HAMi-core（`libvgpu` / `libvnpu`）在容器内强制执行配额。这套体系在生产中经受了检验（案例数据来自[社区分享](https://aicon.infoq.cn/2026/shenzhen/presentation/7168)）：顺丰通过 GPU 共享实现最高 57% 的 GPU 节省，SNOW 降低 55% 成本，招商银行借助拓扑感知调度实现硬件资源池 100% 利用。 + +但正因为整个体系建在 Device Plugin 和调度扩展之上，天花板也是结构性的。HAMi 现有方案的现实瓶颈可以归纳为四点（详见[社区分享](https://aicon.infoq.cn/2026/shenzhen/presentation/7168)）： + +| 瓶颈 | 根源 | +| :--------------------- | :-------------------------------------------------------------- | +| ResourceQuota 计量问题 | Extended Resource 不进原生配额体系，需要额外的 Quota 实现 | +| Preemption 兼容问题 | Extender API 的限制，在支持抢占等调度功能时捉襟见肘 | +| 调度性能劣化 | 节点锁的排它性：单节点同时调度多个 Pod 时，并发退化为串行 | +| 异常死锁 | 系统默认锁定时长 5 分钟，创建过程中的请求失败可能让节点无法解锁 | + +这四点都不是代码层面的 bug，而是架构限制：要在以节点为中心的分配模型上做精细化分配，就得付出这些代价。 + +## DRA 改变了什么 + +Kubernetes 1.34 将 DRA（Dynamic Resource Allocation）核心 API 推进至 GA，集群设备资源管理从以节点为中心的分配模型，演进为以资源对象为核心的声明式模型。新 API 一共四个对象： + +- **ResourceSlice**：设备清单。每块加速器是一个 device，携带属性（uuid、型号、PCIe 等）与容量（显存、算力）。 +- **ResourceClaim**：Pod 的设备需求声明，在 Pod 创建之前完成表达与约束。 +- **DeviceClass**：设备过滤与配置模板。 +- **ResourceClaimTemplate**：供 StatefulSet 等控制器批量生成 claim。 + +对共享调度最关键的是 **Consumable Capacity**（KEP-5075）：设备以两个独立维度（显存、算力百分比）发布容量，`allowMultipleAllocations` 允许同一设备被多个 claim 消费，kube-scheduler 逐维度对账已消费容量，每次分配带独立 shareID。这意味着“多个 workload 合理消费同一加速器的部分容量”第一次成为调度器的原生能力，不再需要树外组件模拟。注意该 feature gate 在 1.34/1.35 需手动开启，1.36 起默认开启。 + +传统 Device Plugin 接口只能暴露简单容量信息，调度器“调度到节点后才发现资源不够”；DRA 让设备属性在调度阶段就可精确匹配。这里的迁移逻辑很清楚：瓶颈不在 HAMi 的调度算法，而在它脚下那层 API。 + +## HAMi DRA 的设计 + +HAMi DRA 没有推倒重来，而是只替换了上半部分：调度和记账改用 Kubernetes 原生实现，运行时隔离继续用 HAMi-core。 + +```mermaid +%% title: HAMi DRA 的组件分工 +flowchart TD + POD["用户 Pod
HAMi 风格资源请求"] --> HOOK["HAMi-DRA Webhook
创建与回收 ResourceClaim"] + HOOK --> DRA["Kubernetes DRA
ResourceClaim / DeviceClass
ResourceSlice / kube-scheduler"] + DRA --> DRV["DRA driver
kubelet plugin + CDI
设备发现与注入"] + DRV --> CORE["HAMi-core
显存配额 / 算力时间片
libvgpu / libvnpu 拦截"] + CORE --> ACC["物理加速器
多个 Pod 共享"] +``` + +三个组件，各管一段：**HAMi-DRA Webhook** 负责“怎么声明”，**DRA driver** 负责“怎么分配设备”，**HAMi-core** 负责“怎么共享设备”。 + +**Webhook** 做两件事。一是多维度资源自动转换：把 HAMi 定义的多维度 ResourceName（如 `huawei.com/Ascend310P` 加 `-memory` 加 `-core`，NVIDIA 侧 `nvidia.com/gpu` 加 `gpumem` 加 `gpucores`）自动转换成 ResourceClaim 的 `count` 与 `capacity.requests`；HAMi 的选卡注解（`use-*-uuid` 等）转换成 CEL 选择器。二是 ResourceClaim 生命周期管理：claim 的创建与 Pod 绑定，Pod 删除时自动清理。注意 HAMi-DRA 自身不再包含调度器组件，原生 kube-scheduler、Volcano 等都能配合工作。 + +**DRA driver** 是节点侧实现：发现设备并发布 ResourceSlice（属性加容量），处理 kubelet 的 `NodePrepareResources` 注入环境变量、创建共享目录，通过 `UnprepareResourceClaims` 在合适时机清理 HAMi-core 引入的临时文件，这也是设备生命周期管理比 Device Plugin 时代更完善的地方。设备注入走 CDI 标准接口落到 containerd。 + +**HAMi-core** 原样保留：DRA 只解决“请求与调度”那一半，“容器内强制执行”那一半（显存配额、算力时间片）仍由 `libvgpu` / `libvnpu` 完成。这正是 [《Kubernetes DRA 会取代 HAMi 吗？》](/zh/blog/does-kubernetes-dra-replace-hami)的结论在架构上的落点。 + +调度性能的提升同样来自这次替换（[AICon](https://aicon.infoq.cn/2026/shenzhen/presentation/7168) 与 [KCD Beijing](https://dynamia.ai/zh/blog/kcd-beijing-2026-dra-gpu-scheduling) 两场分享给出的对比相互印证）：借助 DRA 自身的调度系统，HAMi-DRA 避免了节点锁带来的性能衰退，同节点并发调度多个 Pod 时，原方案的 Pod 创建时间近似线性劣化（峰值在 42 秒量级），HAMi-DRA 则明显更低（幅度在 30% 以上）。这主要得益于 DRA 的资源预绑定机制：资源分配在调度阶段就已确定，减少了调度冲突与重试。整体可以概括为四组对照：NodeLock 到 No Locks，ResourceName 到 ResourceClaim，Device Plugin 到 DRA Driver，Annotation 到 Attributes。 + +还有一个容易被低估的变化是可观测性。传统模型里，资源信息来自 Node、使用情况来自 Pod，完整的资源视图需要聚合推断才能得到；DRA 模型里 ResourceSlice 描述设备清单、ResourceClaim 描述分配，资源视角本身就是一等公民。可观测性从“推断”变成了“直接建模”（[社区分享](https://dynamia.ai/zh/blog/kcd-beijing-2026-dra-gpu-scheduling)）：运维可以直接从 ResourceClaim 看到每块加速器被谁占用、分配了多少显存、还剩多少余量，而不必从节点状态和 Pod 配置里反推。 + +## 两种使用模式 + +DRA 的能力升级有一面经常被忽视的代价：写法复杂度，也就是常被提到的“UX 退化”（[社区分享](https://dynamia.ai/zh/blog/kcd-beijing-2026-dra-gpu-scheduling)）。Device Plugin 的写法是一行： + +```yaml +resources: + limits: + nvidia.com/gpu: 1 +``` + +DRA 原生写法则是独立的 ResourceClaim 对象，`count`、`capacity.requests` 一层层嵌套，还要配上按设备属性过滤的 CEL 选择器： + +```yaml +spec: + devices: + requests: + - exactly: + allocationMode: ExactCount + count: 1 + capacity: + requests: + memory: 4194304k +``` + +对已在使用 Device Plugin 的企业，迁移成本不只是改 YAML，而是整个团队要学一套新的资源声明范式。HAMi-DRA 对此的回答是两种提交方式，底层调度与切分逻辑一致： + +- **DRA 原生模式**：手动创建 ResourceClaim 声明显存与算力，Pod 通过 `resourceClaims` 引用。适合新业务，能直接使用 CEL 选择器表达设备属性约束（例如拓扑感知、NUMA 亲和；相同机制已扩展到网卡等资源，配合 `firstAvailable` 还能表达多种候选分配方案，见[社区分享](https://aicon.infoq.cn/2026/shenzhen/presentation/7168)）。 +- **DevicePlugin 兼容模式**：沿用传统 `nvidia.com/gpu` / `huawei.com/Ascend310P` 资源申请语法，Webhook 自动拦截并转换为 ResourceClaim。存量业务零改造迁移，这就是“无感迁移”的由来：这个设计把 DRA 从“专家接口”变成了“普通用户接口”（[社区分享](https://dynamia.ai/zh/blog/kcd-beijing-2026-dra-gpu-scheduling)）。 + +兼容模式的转换发生在准入阶段。以一台昇腾 310P 真实集群里提交的 Pod 为例，`resources.limits` 里的三个 `huawei.com/*` 条目被替换为 `resources.claims` 与 `spec.resourceClaims`，生成的 ResourceClaim 中 HAMi 语义逐项进入 DRA 语义： + +| ResourceClaim 字段 | 来源 | 转换规则 | +| :-- | :-- | :-- | +| `count: 1` | `Ascend310P: 1` | 整数直传 | +| `capacity.requests.memory: 8589934592` | `-memory: 8192`（MiB） | MiB 转字节（8192 × 1024 × 1024） | +| `capacity.requests.cores: "50"` | `-core: 50` | 百分比直传 | +| CEL 选择器 | `use-Ascend310P-uuid` 注解 | `uuid in ["..."]`，值来自 ResourceSlice | + +两个 Pod 各请求 8192 MiB 显存加 50 算力、指向同一块卡时，两个 claim 落到同一个 device 上，各自带独立 shareID，调度器在分配第二个之前已把第一个的消费记账。容量记满后新请求被调度器直接拒绝（claim 停在 pending），Pod 删除后 claim 自动回收、记账恢复。容器内的强制执行依旧是 HAMi-core：`libvnpu.so` 拦截超限的显存申请并转为容器内 OOM，容器视角的设备被虚拟化为申请的 8 GiB 而非整卡 21 GiB。完整过程见[实验 17](/zh/tutorials/labs/ascend-hami-dra)。 + +## 设备支持的时间线，与“为什么是现在” + +HAMi DRA 的能力边界很大程度上取决于各厂商 DRA driver 的适配进度。社区分享中给出的时间线（[AICon](https://aicon.infoq.cn/2026/shenzhen/presentation/7168)）： + +- 2025.09：NVIDIA DRA Driver，Consumable Capacity 就绪 +- 2026.03：海光 DCU DRA Driver 完成与 HAMi DRA 的集成 +- 2026.04：Enflame（燧原）DRA Driver 集成推进中 +- 2026.08：**Ascend DRA Driver 进入开发与预览**（[4pdOss/hami-dra-driver](https://github.com/4pdOss/hami-dra-driver)，即范式的开源仓库） + +昇腾链路进入预览，意味着 HAMi DRA 的能力边界刚刚扩展到 NPU：[实验 17](/zh/tutorials/labs/ascend-hami-dra) 已在一台 310P3 服务器上把 HAMi 请求到 NPU 共享的完整链路跑通，昇腾 DRA driver 当前为 chart 0.1.1（实验所用镜像为修复 uuid 生成问题的开发构建，正式发布前行为可能变化）。HAMi 2.9 版本直播中提到 NPU 的 DRA 支持计划随 2.10 发布，在那之前会先放出测试版本供社区试用，现在正是尝鲜与反馈的窗口。 + +## 落地前的现实约束 + +落地阻力可以归纳为四条： + +- **高版本依赖**：Consumable Capacity 依赖 Kubernetes 1.34+，其他 DRA 特性可能要求更高版本，生产环境推动升级不轻松。 +- **有限的厂商支持**：开箱即用的 DRA Driver 数量还少，支持的异构设备有限。 +- **Scoring 缺失**：设备调度层面 DRA 的打分能力尚缺，对特性化调度要求的支持有限。 +- **学习成本**：DRA 引入的新概念对集群部署与维护同学有门槛，HAMi 社区的对策正是用 Webhook 兼容模式把这份成本挡在用户视线之外。 + +选型上（[HAMi 2.9 版本直播](https://dynamia.ai/zh/blog/hami-2.9-webinar-recap)给出的建议）：标准化集群、追求最优调度效果（如拓扑感知），推荐原生 HAMi；高度定制化（已有自己的调度器）或需要即插即用设备复用的集群，推荐 HAMi DRA。此外两条运维事实值得记住：DRA 模式与传统模式不兼容，不要同时启用；使用 DRA 模式仍然会经过 HAMi-core。 + +## 展望 + +HAMi-DRA 的 roadmap 上排着：更多异构设备（MetaX、天数智芯 Iluvatar CoreX）、更多 DRA 特性适配（List Attributes、Partitionable Devices）、兼容 Kubernetes 标准设备属性（`resource.kubernetes.io/pcieRoot`），以及与 Volcano、Kueue 的调度器集成。 + +再往大处看（[社区分享](https://dynamia.ai/zh/blog/kcd-beijing-2026-dra-gpu-scheduling)）：Kubernetes 正在演进为 AI 基础设施的控制平面，而 HAMi 的定位是面向 Kubernetes 的加速器资源层，向下适配异构设备，向上支撑训练、推理与 Agent 等 workload，中间提供调度、虚拟化与资源抽象。HAMi-DRA 正是把这层资源能力与 Kubernetes 原生模型对齐的关键一步：HAMi 的价值收敛到运行时强制执行与异构生态适配，资源表达与调度则交给原生模型。如果你正在评估从 Device Plugin 模式迁移到 DRA，或在为异构集群寻找统一的资源管理层，欢迎试用 [HAMi DRA](https://github.com/Project-HAMi/HAMi-DRA) 并向社区反馈。 + +## 延伸阅读 + +- 杨守仁 AICon 深圳 2026 分享：[从 HAMi 到 HAMi-DRA：异构环境的算力资源管理实践](https://aicon.infoq.cn/2026/shenzhen/presentation/7168) +- 王纪飞、James Deng KCD Beijing 2026 分享回顾：[从 Device Plugin 到 DRA：GPU 调度范式升级与 HAMi-DRA 实践](https://dynamia.ai/zh/blog/kcd-beijing-2026-dra-gpu-scheduling) +- HAMi 2.9 版本直播回顾（李孟轩）：[HAMi 2.9 昇腾软切分与 DRA 实战详解](https://dynamia.ai/zh/blog/hami-2.9-webinar-recap) +- 社区上手指南：[HAMi 正式接入 Kubernetes DRA：下一代 GPU 资源模型实践指南](https://dynamia.ai/zh/blog/hami-dra-quickstart) +- Mesut Oezdil：[Kubernetes DRA 会取代 HAMi 吗？](/zh/blog/does-kubernetes-dra-replace-hami)（[英文原文](https://www.cncf.io/blog/2026/08/07/does-kubernetes-dra-replace-hami/)，CNCF 博客） +- 用户指南：[如何使用 HAMi DRA](/zh/docs/installation/how-to-use-hami-dra)；动手实验：[实验 17：用 HAMi DRA 共享昇腾 NPU](/zh/tutorials/labs/ascend-hami-dra)、[实验 4：用 DRA 切分 GPU](/zh/tutorials/labs/hami-dra) +- 相关组件：[Project-HAMi/HAMi-DRA](https://github.com/Project-HAMi/HAMi-DRA) · [4pdOss/hami-dra-driver](https://github.com/4pdOss/hami-dra-driver) · [Project-HAMi/hami-vnpu-core](https://github.com/Project-HAMi/hami-vnpu-core) diff --git a/i18n/zh/docusaurus-plugin-content-blog/tags.yml b/i18n/zh/docusaurus-plugin-content-blog/tags.yml index f987ffd81..a8f5b3e8d 100644 --- a/i18n/zh/docusaurus-plugin-content-blog/tags.yml +++ b/i18n/zh/docusaurus-plugin-content-blog/tags.yml @@ -97,6 +97,9 @@ "软切分": label: "软切分" permalink: "/soft-slicing" +"NPU 共享": + label: "NPU 共享" + permalink: "/npu-sharing" "Release": label: "Release" permalink: "/release" diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md new file mode 100644 index 000000000..5af52ea2b --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md @@ -0,0 +1,790 @@ +--- +title: "实验 17：用 HAMi DRA 共享昇腾 NPU" +description: "在昇腾 310P3 节点上安装 HAMi DRA 0.2.3 与 Ascend DRA Driver，观察 webhook 如何把 huawei.com/Ascend310P 请求转换成 ResourceClaim，并验证双 Pod NPU 共享、显存配额隔离与调度器容量记账。" +sidebar_label: "实验 17：昇腾 + HAMi DRA" +lab: + level: Advanced + duration: 约 120 分钟 + environment: 昇腾 310P3 aarch64 服务器上的单节点 Kubernetes 1.35.7 集群（麒麟 V10、昇腾驱动 25.5.1、containerd 1.7 并启用 CDI） + cost: 需要专属的昇腾 310P3 aarch64 硬件；HAMivNPUCore 软切分仅支持 ARM + authors: + - rootsongjc + verified: "2026-09-17" +tags: + - DRA + - hami + - ascend + - 资源共享 +toc_max_heading_level: 2 +--- + +本实验在昇腾硬件上走完 HAMi DRA 的完整链路：安装 HAMi DRA 0.2.3 与 Ascend DRA Driver，提交一个使用普通 HAMi 语法（`huawei.com/Ascend310P` 加 `-memory` 与 `-core`）的 Pod，观察 admission webhook 把这个请求转换成 Kubernetes 原生的 ResourceClaim。随后 kube-scheduler 在一块物理 310P3 上分配切片，两个 Pod 以独立配额共享同一块卡，超额请求保持 Pending 直到容量被释放。本实验中所有输出块均来自验证服务器的真实采集。 + +## 实验目标 + +- HAMi DRA 如何把 HAMi 风格的 Extended Resource 转换成带 CEL 选择器与容量请求的 ResourceClaim +- 如何读懂 ResourceSlice：每块 NPU 一个 device，携带 `uuid`、`productName`、显存与算力容量、`allowMultipleAllocations` +- kube-scheduler 如何按 shareID 记账已消费容量，并把多个 Pod 放到同一块 NPU 上 +- HAMivNPUCore 如何在容器内强制显存配额，把超限分配转成容器内 OOM +- Pod 删除后容量如何释放与再分配，以及与存量 HAMi core 部署共存时什么会出问题 + +## 实验概览 + +```mermaid +%% title: HAMi DRA 在昇腾 310P 上的链路 +flowchart TD + POD["HAMi 语法的 Pod
Ascend310P + memory + core"] --> HOOK["HAMi-DRA webhook
创建 ResourceClaim"] + HOOK --> CLAIM["ResourceClaim
count + capacity + CEL 选择器"] + CLAIM --> SCHED["kube-scheduler
DRA + 可消费容量记账"] + SLICE["ResourceSlice
每块 310P3 一个 device"] --> SCHED + SCHED --> DRV["Ascend DRA driver
NodePrepareResources + CDI"] + DRV --> CORE["HAMivNPUCore
NPU_MEM_QUOTA / NPU_PRIORITY"] + CORE --> NPU["Ascend 310P
多个 Pod 共享"] +``` + +三个组件，三段职责，整个实验期间要分清： + +- **HAMi-DRA** 负责“请求怎么声明”。它是一组 admission webhook，不做任何 NPU 虚拟化，也不部署 kubelet plugin。 +- **Ascend DRA driver**（driver 名 `ascend.project-hami.io`，来自 [4pdOss/hami-dra-driver](https://github.com/4pdOss/hami-dra-driver)）负责“设备怎么分配”：设备发现、ResourceSlice 发布、kubelet Prepare/Unprepare、CDI 注入。 +- **HAMivNPUCore** 负责“设备怎么共享”：`libvnpu.so` 拦截加容器内 limiter，强制显存配额（`NPU_MEM_QUOTA`）与算力时间片（`NPU_PRIORITY`）。 + +## 前置条件 + +- Kubernetes 1.34 或更高版本（DRA 核心 API 自 1.34 起 GA）。本实验在 1.35.7 上验证，单节点，control-plane 与 worker 同机。 +- kube-apiserver、kube-scheduler、kubelet 三处均开启 `DRAConsumableCapacity` feature gate。容量请求与 `allowMultipleAllocations` 记账都依赖它。 +- containerd 开启 CDI：`enable_cdi = true` 且 `cdi_spec_dirs = ["/etc/cdi", "/var/run/cdi"]`。 +- ARM（aarch64）宿主机，昇腾驱动 25.5 或更高，用于软切分的每块 NPU 都开启 device-share 模式。 +- `ascend` RuntimeClass 存在（Pod 需要 `runtimeClassName: ascend`）。 +- cert-manager（HAMi-DRA 用它签发 webhook 证书）。 + +### 验证环境 + +| 组件 | 版本 / 取值 | +| :-- | :-- | +| 硬件 | 2 × Ascend 310P3（NPU ID 4 / 5，PCIe 0000:81:00.0 / 0000:85:00.0，每卡显存 21525 MB） | +| 操作系统 | 麒麟 Kylin Linux Advanced Server V10（Lance），aarch64，内核 4.19.90-52.48.v2207.ky10 | +| Kubernetes | v1.35.7，单节点 | +| 容器运行时 | containerd 1.7.29，已开启 CDI | +| Feature gates | kube-apiserver / kube-scheduler / kubelet 均开启 `DRAConsumableCapacity=true` | +| 昇腾驱动 / 固件 | 25.5.1 / 7.8.0.6.201 | +| HAMi-DRA | 0.2.3（chart 与镜像 tag 一致） | +| Ascend DRA driver | chart `ascend-dra-driver-0.1.1`（app 0.1.0），镜像 `projecthami/ascend-dra-driver:uuid-fix-20260909`（开发构建，见步骤 2） | +| 共存组件 | HAMi 企业版 2.10.0-r1（hami-scheduler、hami-ascend-device-plugin、hami-resource-pool-manager）、cert-manager v1.21.1 | + +### 检查 feature gate + +```bash +ps -ef | grep -E "kube-apiserver|kube-scheduler|kubelet" | grep -o "feature-gates=.*" +``` + +```text +feature-gates=DRAConsumableCapacity=true +feature-gates=DRAConsumableCapacity=true +``` + +列出的每个控制面组件都应带此 gate。任何一处缺失，容量请求会被拒绝，或者记账静默失效。 + +### 检查 containerd CDI + +```bash +grep -E "enable_cdi|cdi_spec_dirs" /etc/containerd/config.toml +``` + +```text + cdi_spec_dirs = ["/etc/cdi", "/var/run/cdi"] + enable_cdi = true +``` + +### 为 NPU 开启 device-share + +`-i` 为 `npu-smi info -l` 中的 NPU id（验证机上为 4 和 5）。该命令对指定 NPU 上所有 chip 生效： + +```bash +npu-smi set -t device-share -i 4 -d 1 +npu-smi set -t device-share -i 5 -d 1 +npu-smi info -t device-share +``` + +```text + NPU ID : 4 + Chip Count : 1 + + Device-share Status : True + Chip ID : 0 + + NPU ID : 5 + Chip Count : 1 + + Device-share Status : True + Chip ID : 0 +``` + +计划共享的每块卡都应是 `Device-share Status: True`。再确认 RuntimeClass 存在： + +```bash +kubectl get runtimeclass ascend +``` + +```text +NAME HANDLER AGE +ascend ascend ... +``` + +### 与存量 HAMi core 部署共存 + +如果集群已在运行 HAMi core 或 HAMi 企业版（hami-scheduler、hami-webhook、hami-ascend-device-plugin），它们的 mutating webhook 同样会拦截 `huawei.com/Ascend310P*` 资源请求，对同一个 Pod 二次改写。DRA 的 workload 必须在两层上同时豁免： + +```bash +kubectl create ns dra-ascend-e2e +kubectl label ns dra-ascend-e2e hami.io/webhook=ignore +``` + +本实验的 Pod 另外带有标签 `hami.io/webhook: ignore`。在没有 HAMi core 的干净集群上，这一步无害但非必需。 + +## 步骤 1：安装 HAMi DRA 0.2.3 + +cert-manager 是前置依赖；验证集群已有 v1.21.1。干净集群先安装 cert-manager。 + +官方 Helm 仓库方式： + +```bash +helm repo add hami-dra https://project-hami.github.io/HAMi-DRA +helm repo update +helm search repo hami-dra # 确认 0.2.3 +helm install hami-dra hami-dra/hami-dra -n hami-system \ + --version 0.2.3 -f ascend-values.yaml +``` + +`ascend-values.yaml` 只开启昇腾转换： + +```yaml +deviceVendors: + - ascend # 只开 Ascend 转换（910A/B2/B3/B4/B4-1/310P/910C） + +drivers: + nvidia: + enabled: false # 不部署 NVIDIA kubelet driver + fake: + enabled: false + +monitor: + enabled: false # 可选的 Prometheus 指标组件，本实验未开启 + +certs: + certManager: + enabled: true +``` + +确认 webhook 正常： + +```bash +kubectl get pods -n hami-system +kubectl get mutatingwebhookconfigurations,validatingwebhookconfigurations | grep hami-dra +``` + +```text +NAME READY STATUS RESTARTS AGE +hami-ascend-device-plugin-vp6tm 1/1 Running 0 5h22m +hami-dra-webhook-5b85c54c78-hbtjc 1/1 Running 0 13s +hami-resource-pool-manager-7b6bd856fb-phd7r 1/1 Running 0 5h23m +hami-scheduler-7dccfd9b96-7spsm 2/2 Running 0 5h28m +``` + +```text +mutatingwebhookconfiguration.admissionregistration.k8s.io/hami-dra-mutatingwebhookconfiguration +validatingwebhookconfiguration.admissionregistration.k8s.io/hami-dra-validatingwebhookconfiguration +``` + +这份列表里的 hami-scheduler 与 hami-ascend-device-plugin 属于先前已存在的 HAMi 企业版安装，不属于 HAMi-DRA。HAMi-DRA 自身只运行一个 deployment，即 webhook。它不部署任何 kubelet plugin：昇腾的节点侧 driver 来自下一步。 + +## 步骤 2：安装 Ascend DRA Driver + +```bash +git clone --recurse-submodules https://github.com/4pdOss/hami-dra-driver.git +cd hami-dra-driver +helm upgrade --install ascend-dra-driver \ + deployments/helm/ascend-dra-driver \ + -n ascend-dra-driver --create-namespace +``` + +chart 默认开启 HAMivNPUCore 模式。不要设置 `kubeletPlugin.fullCardAndTraditionalVNPU.enabled=true`：那会切换到整卡 / 模板 vNPU 路径，该路径仍在开发中，且会关闭 HAMivNPUCore gate。 + +:::note 关于本实验使用的镜像版本 + +验证运行使用 chart `ascend-dra-driver-0.1.1`，镜像覆盖为开发构建 `projecthami/ascend-dra-driver:uuid-fix-20260909`，该构建修复了早期版本 ResourceSlice uuid 生成的 bug。正式发布前行为可能变化；请固定你实际测试过的版本。 + +::: + +检查 driver DaemonSet 与它发布的 DRA 对象： + +```bash +kubectl get pods -n ascend-dra-driver +kubectl get deviceclass +kubectl get resourceslice +``` + +预期：driver Pod 全部 Running，一个名为 `hami-vnpu-core.project-hami.io` 的 DeviceClass，每个节点一条 driver 为 `ascend.project-hami.io` 的 ResourceSlice。 + +## 步骤 3：查看 DeviceClass 与 ResourceSlice + +DeviceClass 用 CEL 表达式按 driver 与 type 过滤设备： + +```bash +kubectl get deviceclass hami-vnpu-core.project-hami.io -o yaml +``` + +```yaml +apiVersion: resource.k8s.io/v1 +kind: DeviceClass +metadata: + name: hami-vnpu-core.project-hami.io +spec: + selectors: + - cel: + expression: |- + device.driver == "ascend.project-hami.io" && + device.attributes["ascend.project-hami.io"].type == "HAMivNPUCore" +``` + +ResourceSlice 是 Kubernetes 第一次“看见”NPU 的地方： + +```bash +kubectl get resourceslice +``` + +```text +NAME DRIVER NODE AGE +aio-node74-arm-ascend.project-hami.io-gktzs ascend.project-hami.io aio-node74-arm 3m +``` + +展开查看（节选；完整对象用 `kubectl get resourceslice -o yaml`）： + +```yaml +apiVersion: resource.k8s.io/v1 +kind: ResourceSlice +metadata: + name: aio-node74-arm-ascend.project-hami.io-gktzs + ownerReferences: + - kind: Node + name: aio-node74-arm +spec: + driver: ascend.project-hami.io + nodeName: aio-node74-arm + pool: + name: aio-node74-arm + resourceSliceCount: 1 + devices: + - allowMultipleAllocations: true # 允许多个 claim 消费同一设备 + name: npu-0-0 + attributes: + brand: { string: Huawei } + index: { int: 0 } + model: { string: 310P3 } + physicalID: { int: 0 } + productName: { string: 310P3 } + type: { string: HAMivNPUCore } + uuid: { string: 68496E64-20E05477-92C31323-6E78030A-BD003019 } + capacity: + cores: + value: "100" # 整卡算力记为 100 + requestPolicy: { default: "100", validRange: { min: "0", max: "100", step: "1" } } + memory: + value: 21525Mi # 与 npu-smi 一致 + requestPolicy: { default: 21525Mi, validRange: { min: 1Mi, max: 21525Mi, step: 1Mi } } + - allowMultipleAllocations: true + name: npu-1-0 + attributes: + index: { int: 1 } + physicalID: { int: 2 } + productName: { string: 310P3 } + type: { string: HAMivNPUCore } + uuid: { string: D8496E64-20C101B1-C0D42F23-AED8030A-40003039 } + capacity: + cores: { value: "100", ... } + memory: { value: 21525Mi, ... } +``` + +四处细节对后续步骤至关重要： + +- 每块 310P3 是一个 device（`npu-0-0`、`npu-1-0`），不是节点上的卡数计数。 +- `uuid` attribute 是芯片 UUID，例如 `68496E64-20E05477-92C31323-6E78030A-BD003019`，**不是**“节点名-序号”格式的字符串。选卡注解的值必须从这里抄，不能凭记忆手写。 +- 容量沿两个维度发布：`memory`（MiB）与 `cores`（整卡百分比），各自的 `requestPolicy` 声明合法请求范围与步长。 +- `allowMultipleAllocations: true` 允许同一设备被多个 ResourceClaim 分配。没有它就没有 NPU 共享。 + +ResourceSlice device 与物理卡的对应关系（共享实验期间用 `npu-smi info` 进程列表交叉验证）： + +| ResourceSlice device | uuid（前 8 位） | npu-smi NPU ID | PCIe 总线 | +| :------------------- | :-------------- | :------------- | :----------- | +| npu-0-0 | 68496E64 | 4 | 0000:81:00.0 | +| npu-1-0 | D8496E64 | 5 | 0000:85:00.0 | + +## 步骤 4：运行第一个昇腾 Workload + +资源请求语法与 HAMi 完全一致： + +- 计数：`huawei.com/Ascend310P`（整数，张数） +- 显存：`huawei.com/Ascend310P-memory`（MiB） +- 算力：`huawei.com/Ascend310P-core`（百分比） +- 选卡：`hami.io/use-Ascend310P-uuid`，值从 ResourceSlice 的 `uuid` 抄写 +- 选型：`hami.io/use-nputype`，与 `productName` 匹配（本实验未使用） + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: ascend-share-a + namespace: dra-ascend-e2e + labels: + hami.io/webhook: ignore + annotations: + hami.io/use-Ascend310P-uuid: "68496E64-20E05477-92C31323-6E78030A-BD003019" # 固定到 npu-0-0 +spec: + runtimeClassName: ascend + containers: + - name: app + image: quay.io/ascend/vllm-ascend:v0.23.0-310p + command: ["sh", "-c", "sleep 3600"] + resources: + limits: + huawei.com/Ascend310P: 1 + huawei.com/Ascend310P-memory: "8192" + huawei.com/Ascend310P-core: "50" +``` + +manifest 里的三个选择：`hami.io/webhook: ignore` 标签让共存的 HAMi core webhook 不碰这个 Pod；`runtimeClassName: ascend` 让 containerd 走 Ascend Docker Runtime；`vllm-ascend` 镜像内含 CANN、torch_npu 与 npu-smi，方便稍后从容器内部检查设备。 + +```bash +kubectl apply -f pod.yaml +kubectl get pod ascend-share-a -n dra-ascend-e2e +``` + +```text +NAME READY STATUS RESTARTS AGE IP NODE +ascend-share-a 1/1 Running 0 3m45s 10.244.0.28 aio-node74-arm +``` + +Running 只说明 Kubernetes 层面满足了。在信任它之前，先看 webhook 到底做了什么。 + +## 步骤 5：观察请求到 ResourceClaim 的转换 + +保存改写后的 Pod，与提交的内容对比： + +```bash +kubectl get pod ascend-share-a -n dra-ascend-e2e -o yaml > mutated-pod.yaml +``` + +`resources.limits` 里的三个 `huawei.com/*` 条目消失了。取而代之： + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: ascend-share-a + namespace: dra-ascend-e2e + labels: + hami.io/dra: "true" # webhook 新增 + hami.io/webhook: ignore + annotations: + hami.io/use-Ascend310P-uuid: 68496E64-20E05477-92C31323-6E78030A-BD003019 +spec: + runtimeClassName: ascend + containers: + - name: app + image: quay.io/ascend/vllm-ascend:v0.23.0-310p + resources: + claims: # 容器级引用 + - name: dra-ascend-e2e-ascend-share-a-app-ascend310p + resourceClaims: # Pod 级引用 + - name: dra-ascend-e2e-ascend-share-a-app-ascend310p + resourceClaimName: dra-ascend-e2e-ascend-share-a-app-ascend310p +``` + +生成的 ResourceClaim（已分配后的状态）： + +```bash +kubectl get resourceclaim -n dra-ascend-e2e +kubectl get resourceclaim dra-ascend-e2e-ascend-share-a-app-ascend310p -n dra-ascend-e2e -o yaml +``` + +```text +NAME STATE AGE +dra-ascend-e2e-ascend-share-a-app-ascend310p allocated,reserved 3m +``` + +```yaml +apiVersion: resource.k8s.io/v1 +kind: ResourceClaim +metadata: + name: dra-ascend-e2e-ascend-share-a-app-ascend310p + namespace: dra-ascend-e2e + finalizers: + - resource.kubernetes.io/delete-protection +spec: + devices: + requests: + - name: npu + exactly: + allocationMode: ExactCount + count: 1 + deviceClassName: hami-vnpu-core.project-hami.io + capacity: + requests: + cores: "50" # 百分比直传 + memory: "8589934592" # 8192 MiB × 1024 × 1024 = 8 GiB（字节） + selectors: + - cel: + expression: device.driver == "ascend.project-hami.io" && device.attributes["ascend.project-hami.io"].type == "HAMivNPUCore" + - cel: + expression: device.attributes["ascend.project-hami.io"].uuid in ["68496E64-20E05477-92C31323-6E78030A-BD003019"] +status: + allocation: + devices: + results: + - consumedCapacity: + cores: "50" + memory: 8Gi + device: npu-0-0 # 调度器选中的设备 + driver: ascend.project-hami.io + pool: aio-node74-arm + request: npu + shareID: be06132e-0aff-4efc-bd8e-42a4e06ed385 + nodeSelector: + nodeSelectorTerms: + - matchFields: + - key: metadata.name + operator: In + values: [aio-node74-arm] + reservedFor: + - name: ascend-share-a + resource: pods +``` + +逐字段对照，HAMi 语义就是这样进入 DRA 语义的： + +| ResourceClaim 字段 | 来源 | 转换规则 | +| :-- | :-- | :-- | +| `count: 1` | `huawei.com/Ascend310P: 1` | 整数直传 | +| `capacity.requests.memory: 8589934592` | `-memory: 8192`（MiB） | MiB 转字节，`value × 1024 × 1024` | +| `capacity.requests.cores: "50"` | `-core: 50` | 百分比直传 | +| selectors 第 1 条 | 固定 | 匹配 driver 加 `type == HAMivNPUCore`，与 DeviceClass 同源 | +| selectors 第 2 条 | 注解 `hami.io/use-Ascend310P-uuid` | `uuid in ["..."]`；`no-use-*` 反选注解生成取反表达式 | +| （如使用）第 3 条 | 注解 `hami.io/use-nputype` | 与 `productName` 匹配 | +| claim 名称 | namespace + pod + container | 超过 DNS-1123 的 63 字符时截断并附加 sha256 后缀 | + +两个细节值得停留。单位转换：HAMi 的 `-memory` 是 MiB 语义，DRA 的 capacity 是 Quantity（字节），所以 8192 MiB 变成 `8589934592`；用户写的是 HAMi 语法，记账落在 DRA 上。`shareID`：每个已分配的 claim 都有一个，它是 `DRAConsumableCapacity`（KEP-5075）记账模型的凭证，同一设备上的多份消费各自独立记账，共享步骤会看到两个不同的 shareID。 + +## 步骤 6：验证 NPU 分配 + +从集群到节点，分三层。 + +**Kubernetes 层。** claim 为 `allocated,reserved`，`status.allocation.devices.results[0].device` 为 `npu-0-0`，见上一节采集。 + +**容器层。** 配额与设备环境变量： + +```bash +kubectl exec -it ascend-share-a -n dra-ascend-e2e -- env | grep -E "NPU_|ASCEND_" +kubectl exec -it ascend-share-a -n dra-ascend-e2e -- ls /dev/davinci* /dev/davinci_manager /dev/devmm_svm +``` + +```text +NPU_PRIORITY=50 +ASCEND_VISIBLE_DEVICES=0 +NPU_GLOBAL_SHM_PATH=/hami-shared-region/0_global_registry +NPU_LOCAL_SHM_PATH=/hami-vnpu-shmem/vnpu_local_shmem +NPU_MEM_QUOTA=8192 +``` + +```text +/dev/davinci0 +/dev/davinci_manager +/dev/devmm_svm +``` + +`NPU_MEM_QUOTA=8192` 与 `NPU_PRIORITY=50` 是 HAMivNPUCore 的显存配额（MiB）与算力时间片，与请求的 `-memory 8192`、`-core 50` 一一对应。`ASCEND_VISIBLE_DEVICES=0` 与 `/dev/davinci0` 来自 CDI 注入；容器内由 `ld.so.preload` 指向的 `libvnpu.so` 同样由 CDI 挂载，这是拦截的入口。 + +**节点层。** driver 在 Prepare 阶段为每个 claim 生成 CDI spec（`CDI_ROOT=/var/run/cdi`）： + +```bash +ls /var/run/cdi/ +``` + +```text +k8s.ascend.project-hami.io-npu_8ec6bbf9-8ce0-4aa1-bc57-7078653d090c.yaml # 本 claim，uid 命名 +k8s.ascend.project-hami.io-npu_common.yaml +``` + +开启 `enable_cdi` 的 containerd 在创建容器时按 spec 注入设备与环境变量。DRA 的分配结果通过 CDI 这个标准接口落到容器运行时。设备层的最终证明在步骤 8：真实 workload 运行后，容器的 torch_npu 报告 8 GiB 设备而不是 21 GiB 整卡。 + +## 步骤 7：两个 Pod 共享一块 NPU + +核心实验：两个 Pod 请求同一块物理 310P3 的部分容量。关键是两个 Pod 使用同一个 uuid 注解。本机有两块卡，不固定的话调度器可能把 Pod 分散到两块卡上，什么也证明不了： + +```text +物理 Ascend 310P（uuid 68496E64-...，device npu-0-0，容量 21525 MiB / 100 cores） + ++-- Pod A（ascend-share-a）：memory 8192 MiB + core 50 ++-- Pod B（ascend-share-b）：memory 8192 MiB + core 50 +``` + +复制第一个 Pod 创建第二个： + +```bash +sed 's/ascend-share-a/ascend-share-b/' pod.yaml | kubectl apply -f - +kubectl get resourceclaim -n dra-ascend-e2e +``` + +```text +NAME STATE AGE +dra-ascend-e2e-ascend-share-a-app-ascend310p allocated,reserved 6m +dra-ascend-e2e-ascend-share-b-app-ascend310p allocated,reserved 2m +``` + +对比两个 claim 的分配结果： + +```yaml +# claim-a +results: + - device: npu-0-0 # 同一个 device + consumedCapacity: { cores: "50", memory: 8Gi } + shareID: be06132e-0aff-4efc-bd8e-42a4e06ed385 + +# claim-b +results: + - device: npu-0-0 # 同一个 device + consumedCapacity: { cores: "50", memory: 8Gi } + shareID: 5554db84-1574-46bd-a3ea-0b7197fbbbda # 各自独立的记账凭证 +``` + +两个 claim 都落在 `npu-0-0` 上，各有独立 shareID。分配 B 之前，调度器已把 A 消费的 8 GiB、50 cores 记在账上；这正是 `DRAConsumableCapacity` 加 `allowMultipleAllocations: true` 带来的能力。Pod 与设备层面： + +```bash +kubectl get pods -n dra-ascend-e2e -o wide +``` + +```text +NAME READY STATUS IP NODE +ascend-share-a 1/1 Running 10.244.0.28 aio-node74-arm +ascend-share-b 1/1 Running 10.244.0.29 aio-node74-arm +``` + +两个容器看到同一个设备节点与各自的配额： + +```text +pod a: /dev/davinci0 NPU_MEM_QUOTA=8192 NPU_PRIORITY=50 ASCEND_VISIBLE_DEVICES=0 +pod b: /dev/davinci0 NPU_MEM_QUOTA=8192 NPU_PRIORITY=50 ASCEND_VISIBLE_DEVICES=0 +``` + +软切分模式下两个容器看到的是同一个 `/dev/davinci0`，不是两个 vNPU 设备节点；隔离由 libvnpu 在用户态完成。步骤 8 中真实 workload 运行后，节点 `npu-smi` 的进程列表会在 NPU 4 上同时出现两个容器的进程。 + +## 步骤 8：验证隔离与配额强制 + +在 Pod A 的容器内运行 PyTorch/torch_npu 张量分配测试（镜像内含完整 CANN 运行时）。先分配到接近配额，再尝试超限： + +```text +# limiter（HAMivNPUCore 容器内管理器）注册进程 +[2026-09-17T10:18:46Z INFO limiter::worker] [Worker PID:381] Initialize SchedulerClient... +[2026-09-17T10:18:46Z INFO limiter::worker] [Worker PID:381] Registered at proc slot 1 + +# 已用 ~6150 MB 时再申请 2050 MB：被拒 +[2026-09-17T10:22:22Z WARN limiter::worker] [Worker PID:1139] Memory Quota Exceeded! + Request: 2050 MB, Used: 6150 MB, Limit: 8192 MB + +# 应用侧表现为容器内的 OOM（注意 total capacity 显示 8.00 GiB，不是整卡 21 GiB） +torch.OutOfMemoryError: NPU out of memory. Tried to allocate 2.00 GiB + (NPU 0; 8.00 GiB total capacity; 2.00 GiB already allocated; ...) +``` + +这段采集同时证明两件事：配额由 libvnpu 与 limiter 在用户态强制执行，超限请求被转成容器内 OOM；容器视角的设备已被虚拟化为 8 GiB，与 `-memory 8192` 请求一致，而不是物理的 21525 MB。节点侧，两个容器的进程落在同一块物理卡上： + +```bash +npu-smi info # 进程列表（节选） +``` + +```text +| NPU Chip | Process id | Process name | Process memory(MB) | +| 4 0 | 44533 | python3 | 2156 | +| 4 0 | 49806 | python3 | 2156 | +``` + +`NPU_PRIORITY=50` 同样被注入，通过时间片调度约束算力。本实验验证了注入，但没有运行长时间的满载对照 workload，因此不对 310P 上时间片隔离的实际强度下结论；算力维度在调度层的记账由下一步完整覆盖。 + +## 步骤 9：耗尽容量、释放与再分配 + +Pod A 与 B 此时占住 `npu-0-0` 的 16384 MiB / 100 cores；该设备只剩 5141 MiB / 0 cores。提交请求 8192 MiB / 50 cores 的 Pod C，使用同一 uuid 注解： + +```bash +kubectl apply -f pod-c.yaml +kubectl get pod ascend-share-c -n dra-ascend-e2e +kubectl describe pod ascend-share-c -n dra-ascend-e2e | tail -4 +kubectl get resourceclaim -n dra-ascend-e2e +``` + +```text +NAME READY STATUS RESTARTS AGE +ascend-share-c 0/1 Pending 0 133m +``` + +```text + Warning FailedScheduling 3m3s (x27 over 133m) default-scheduler + 0/1 nodes are available: 1 cannot allocate all claims. still not schedulable, + preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. +``` + +```text +NAME STATE AGE +dra-ascend-e2e-ascend-share-a-app-ascend310p allocated,reserved 157m +dra-ascend-e2e-ascend-share-b-app-ascend310p allocated,reserved 153m +dra-ascend-e2e-ascend-share-c-app-ascend310p pending 133m +``` + +Pod C 的 claim 停在 `pending`，没有分配结果。第二块卡 `npu-1-0` 空闲，但 uuid 注解把 Pod C 钉死在 `npu-0-0` 上，调度器如实拒绝。拒绝发生在调度器内部，不依赖节点上的 device plugin 汇报。 + +### 对照实验：单独验证 memory 维度 + +Pod C 同时超了两个维度（memory 8192 > 5141，cores 50 > 0），单靠它无法区分是哪个维度拒绝的。在第二块卡上做一组对照，单独隔离 memory 维度： + +| Pod | 请求 | npu-1-0 状态 | 结果 | +| :-- | :-- | :-- | :-- | +| ascend-ctrl-a | 8192 MiB / 50 cores（uuid D8496E64） | 初始 21525 / 100 | Running，分配后剩 13333 MiB / 50 cores | +| ascend-ctrl-b | 14000 MiB / 40 cores（同 uuid） | 剩余 13333 / 50 | Pending | + +ctrl-b 的算力请求满足（40 ≤ 50），显存请求不满足（14000 > 13333）： + +```text + Warning FailedScheduling 4m16s (x2 over 9m17s) default-scheduler + 0/1 nodes are available: 1 cannot allocate all claims. still not schedulable, ... +``` + +memory 与 cores 是两个独立记账维度：调度器逐维度对账所有已分配 claim 的 `consumedCapacity` 与 ResourceSlice 的 `capacity.value`。 + +### 释放与再分配 + +删除 Pod A，观察三个层面的恢复： + +```bash +kubectl delete pod ascend-share-a -n dra-ascend-e2e +kubectl get resourceclaim -n dra-ascend-e2e +``` + +```text +NAME STATE AGE +dra-ascend-e2e-ascend-ctrl-a-app-ascend310p allocated,reserved 10m +dra-ascend-e2e-ascend-ctrl-b-app-ascend310p pending 10m +dra-ascend-e2e-ascend-share-b-app-ascend310p allocated,reserved 164m +dra-ascend-e2e-ascend-share-c-app-ascend310p allocated,reserved 143m # 由 pending 转为已分配 +``` + +share-a 的 claim 消失了：HAMi-DRA 的 validating webhook 监听带 `hami.io/dra` 标签的 Pod 删除事件，主动删除对应 ResourceClaim。节点侧，driver 的 Unprepare 删除了 share-a 的 CDI spec，并为 share-c 生成新的： + +```text +# before +k8s.ascend.project-hami.io-npu_1b737f89-...yaml (share-b) +k8s.ascend.project-hami.io-npu_8ec6bbf9-...yaml (share-a，待回收) +# after +k8s.ascend.project-hami.io-npu_1b737f89-...yaml (share-b) +k8s.ascend.project-hami.io-npu_8a2fd648-...yaml (share-c，新分配) +``` + +容量恢复后（16384 - 8192 = 8192 MiB 已用，剩余 13333 ≥ 8192，cores 50 ≤ 50），Pod C 在 18 秒内被重新调度，并分配到同一设备： + +```text + Warning FailedScheduling 3m42s (x29 over 143m) default-scheduler (此前 29 次拒绝) + Normal Scheduled 18s default-scheduler + Successfully assigned dra-ascend-e2e/ascend-share-c to aio-node74-arm + Normal Started 16s kubelet Container started + +# share-c 的分配：同一个 npu-0-0，继承被释放的容量 +{"consumedCapacity":{"cores":"50","memory":"8Gi"},"device":"npu-0-0", + "driver":"ascend.project-hami.io","pool":"aio-node74-arm", + "shareID":"70cbda02-b70e-47d3-b9c8-c5c5c608a1fb"} +``` + +完整生命周期闭环：allocate(A)、share(A + B 共享 npu-0-0)、reject(C，容量不足)、release(A)、reallocate(C 落到同一 device)。 + +:::note claim 清理的边界 + +claim 的删除由 HAMi-DRA validating webhook 负责。如果 Pod 删除时 webhook 宕机，claim 可能残留（在记账中继续占住容量），需要手动清理或等 webhook 恢复；见故障排查。 + +::: + +## 故障排查 + +### 与 HAMi core webhook 冲突 + +- **症状**：集群中已有 HAMi core 或企业版时，DRA 的 Pod 被二次改写、携带非预期注解，或调度行为异常。 +- **原因**：HAMi core 的 mutating webhook 同样拦截 `huawei.com/Ascend310P*` 资源请求。 +- **检查**：`kubectl get mutatingwebhookconfigurations | grep hami`；对比 Pod 提交前后的 spec。 +- **修复**：namespace 与 Pod 双层打 `hami.io/webhook: ignore` 标签，与本实验一致。 + +### CEL 选择器与 ResourceSlice 不匹配 + +- **症状**：Pod 一直 Pending，事件为 `cannot allocate all claims`；claim 停在 `pending`。 +- **原因**：uuid 注解与实际发布的 uuid 不一致（例如凭印象写成“节点名-序号”）。早期 driver 版本的 uuid 生成也有 bug，本实验因此使用 `uuid-fix-20260909` 构建。 +- **检查**：`kubectl get resourceslice -o yaml | grep -A2 uuid`，对照 Pod 注解。 +- **修复**：uuid 永远从 ResourceSlice 抄写；`hami.io/use-nputype` 的值对 `productName`（如 `310P3`）。 + +### DRAConsumableCapacity 未开启 + +- **症状**：带 `capacity.requests` 的 ResourceClaim 被拒，或调度不记账（超额也能调度、共享行为异常）。 +- **原因**：容量请求与 `allowMultipleAllocations` 记账依赖该 gate。 +- **检查**：`ps -ef | grep -E "kube-apiserver|kube-scheduler|kubelet" | grep -o "feature-gates=.*"`。 +- **修复**：三个组件统一加 `DRAConsumableCapacity=true` 并重启。 + +### device-share 未开启 + +- **症状**：driver Prepare 失败，或容器无法正常使用软切分设备。 +- **原因**：HAMivNPUCore 要求芯片开启容器共享模式、昇腾驱动 25.5 或更高、ARM 平台。 +- **检查**：`npu-smi info -t device-share`；`/usr/local/Ascend/driver/version.info`。 +- **修复**：`npu-smi set -t device-share -i -d 1`（对指定 NPU 上所有 chip 生效）。 + +### Pod Running 但容器里看不到 NPU + +- **症状**：claim 已分配、Pod Running，但容器内没有 `/dev/davinci*`，也没有配额环境变量。 +- **原因**：缺 `runtimeClassName: ascend`，或镜像内缺少 CANN 与 npu-smi（设备注入靠 CDI 加 RuntimeClass；用户态工具来自镜像）。 +- **检查**：`kubectl get pod -o jsonpath='{.spec.runtimeClassName}'`；`kubectl exec ... -- ls /dev/davinci*`。 +- **修复**：补 RuntimeClass；换带 CANN 运行时的镜像（本实验用 `quay.io/ascend/vllm-ascend:v0.23.0-310p`）。 + +### ResourceClaim 泄漏 + +- **症状**：Pod 已删除但 ResourceClaim 残留（`allocated,reserved`），占住容量，导致后续 Pod 无故 Pending。 +- **原因**：Pod 删除时 validating webhook 宕机或漏配 RBAC。claim 自带 `resource.kubernetes.io/delete-protection` finalizer，不会被误删，反过来也意味着不会自动消失。 +- **检查**：`kubectl get resourceclaim -A` 对照存活的 `hami.io/dra=true` Pod。 +- **修复**：手动删除孤儿 claim；检查 hami-dra-webhook Pod 状态与日志。 + +## 清理 + +```bash +kubectl delete pod ascend-share-a ascend-share-b ascend-share-c \ + ascend-ctrl-a ascend-ctrl-b -n dra-ascend-e2e --ignore-not-found +kubectl get resourceclaim -A | grep ascend310p | awk '{print $1, $2}' | \ + xargs -n2 kubectl delete resourceclaim -n 2>/dev/null || true +kubectl delete ns dra-ascend-e2e --ignore-not-found + +helm uninstall ascend-dra-driver -n ascend-dra-driver +kubectl delete ns ascend-dra-driver --ignore-not-found + +helm uninstall hami-dra -n hami-system +``` + +如果其他 workload 还在用，feature gate 与 `ascend` RuntimeClass 可以保留。共享实验机上 `device-share` 也可以保持开启；关闭它会影响该卡上的所有 workload，包括 HAMi core 的。 + +## 本实验证明了什么 + +| 结论 | 证据 | +| :-- | :-- | +| 昇腾 310P 设备发现与 ResourceSlice 发布 | 2 个 device，携带 uuid、capacity、requestPolicy | +| HAMi 请求到 ResourceClaim 的转换 | 改写后的 Pod、claim YAML、MiB 转字节 `8589934592`、CEL 选择器 | +| 原生 DRA 调度分配 | `allocated,reserved` claim，含 shareID 的分配结果 | +| 通过 CDI 注入设备 | 容器内环境变量与 `/dev/davinci0`，`/var/run/cdi` 下的 spec | +| 两个 Pod 共享一块物理 NPU | 两个 claim 均在 `npu-0-0`，独立 shareID | +| 显存隔离 | limiter 在 8192 MB 处拒绝，容器 OOM，torch_npu 报告 8.00 GiB | +| 算力隔离 | `NPU_PRIORITY=50` 注入（注入已验证；时间片强度未量化） | +| memory 与 cores 独立记账 | Pod C Pending；对照组单独隔离出 memory 维度 | +| 释放与再分配 | claim 删除、CDI spec 更换、Pod C 18 秒内重新调度 | + +## 下一步 + +- 这条链路背后的概念与设计：[如何使用 HAMi DRA](/zh/docs/installation/how-to-use-hami-dra) +- NVIDIA GPU 上的同一套转换流程：[实验 4：用 DRA 切分 GPU](/zh/tutorials/labs/hami-dra) 与 [实验 11：KServe 推理与 HAMi DRA GPU 共享](/zh/tutorials/labs/kserve-hami-dra) +- 同一块硬件走 device-plugin 路径：[实验 13：用 Volcano + HAMi-core 软切分昇腾 310P3 vNPU](/zh/tutorials/labs/volcano-ascend-vnpu) +- 相关组件：[Project-HAMi/HAMi-DRA](https://github.com/Project-HAMi/HAMi-DRA) · [4pdOss/hami-dra-driver](https://github.com/4pdOss/hami-dra-driver) · [Project-HAMi/hami-vnpu-core](https://github.com/Project-HAMi/hami-vnpu-core) diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md index 10f83d7bd..aef67fb9f 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md @@ -31,3 +31,4 @@ import LabCardGridAuto from '@site/src/components/labs/LabCardGridAuto'; - **实验 13** 在昇腾 310P3 ARM 服务器上源码编译 Volcano 与 ascend-device-plugin，验证 hami-vnpu-core 软切分、binpack 共卡与容器级监控指标。 - **实验 14** 在挂载四块 T4 的 GKE 节点上安装 HAMi v2.10.0，并通过分配注解与调度器日志观察可组合的 `gpu-scheduler-policy` 策略链（`spread`、`binpack`、`mutex`、`mutex,binpack`）。 - **实验 16** 在配备 7 张 RTX PRO 6000 的服务器上安装 HAMi v2.10.0，并验证完整的 Dynamic MIG 生命周期：按 Pod 创建 MIG、混合规格、选择性回收、device plugin 重启恢复以及跨 GPU 调度。 +- **实验 17** 在昇腾 310P3 节点上安装 HAMi DRA 0.2.3 与 Ascend DRA Driver，验证请求到 ResourceClaim 的转换、双 Pod NPU 共享、显存配额隔离与调度器容量记账。 diff --git a/sidebars-tutorials.js b/sidebars-tutorials.js index d7953c00c..c019cec24 100644 --- a/sidebars-tutorials.js +++ b/sidebars-tutorials.js @@ -91,6 +91,11 @@ module.exports = { id: "labs/dynamic-mig-rtx-pro", customProps: { level: "Advanced", duration: "about 90 minutes" }, }, + { + type: "doc", + id: "labs/ascend-hami-dra", + customProps: { level: "Advanced", duration: "about 120 minutes" }, + }, ], }, ], diff --git a/tutorials/labs/ascend-hami-dra.md b/tutorials/labs/ascend-hami-dra.md new file mode 100644 index 000000000..b11c6216f --- /dev/null +++ b/tutorials/labs/ascend-hami-dra.md @@ -0,0 +1,790 @@ +--- +title: "Lab 17: Ascend NPU Sharing with HAMi DRA" +description: "Install HAMi DRA 0.2.3 and the Ascend DRA driver on an Ascend 310P3 node, watch the webhook turn huawei.com/Ascend310P requests into ResourceClaims, and verify two-Pod NPU sharing, memory quota enforcement, and scheduler capacity accounting." +sidebar_label: "Lab 17: Ascend + HAMi DRA" +lab: + level: Advanced + duration: about 120 minutes + environment: single-node Kubernetes 1.35.7 cluster on an aarch64 Ascend 310P3 server (Kylin V10, Ascend driver 25.5.1, containerd 1.7 with CDI) + cost: requires dedicated Ascend 310P3 aarch64 hardware; HAMivNPUCore soft slicing is ARM-only + authors: + - rootsongjc + verified: "2026-09-17" +tags: + - dra + - hami + - ascend + - resource-sharing +toc_max_heading_level: 2 +--- + +This lab walks the complete HAMi DRA path on Ascend hardware: you install HAMi DRA 0.2.3 and the Ascend DRA driver, submit a Pod written in ordinary HAMi syntax (`huawei.com/Ascend310P` plus `-memory` and `-core`), and watch the admission webhook convert that request into a Kubernetes-native ResourceClaim. The kube-scheduler then allocates a slice of one physical 310P3, two Pods share the same card with independent quotas, and over-capacity requests stay Pending until capacity is released. Every output block in this lab is a verbatim capture from a real run on the verification server. + +## What You'll Learn + +- How HAMi DRA converts HAMi-style extended resources into ResourceClaims with CEL selectors and capacity requests +- How to read a ResourceSlice: one device per NPU, with `uuid`, `productName`, memory and cores capacity, and `allowMultipleAllocations` +- How the kube-scheduler accounts consumed capacity per shareID and places multiple Pods on the same NPU +- How HAMivNPUCore enforces the memory quota inside the container and turns over-quota allocations into an in-container OOM +- How capacity is released and reclaimed when a Pod is deleted, and what breaks when the components coexist with an existing HAMi core installation + +## Lab Overview + +```mermaid +%% title: HAMi DRA path on Ascend 310P +flowchart TD + POD["Pod with HAMi syntax
Ascend310P + memory + core"] --> HOOK["HAMi-DRA webhook
creates ResourceClaim"] + HOOK --> CLAIM["ResourceClaim
count + capacity + CEL selectors"] + CLAIM --> SCHED["kube-scheduler
DRA + consumable capacity"] + SLICE["ResourceSlice
one device per 310P3"] --> SCHED + SCHED --> DRV["Ascend DRA driver
NodePrepareResources + CDI"] + DRV --> CORE["HAMivNPUCore
NPU_MEM_QUOTA / NPU_PRIORITY"] + CORE --> NPU["Ascend 310P
shared by multiple Pods"] +``` + +Three components, three responsibilities. Keep them apart throughout the lab: + +- **HAMi-DRA** decides _how the request is declared_. It is a set of admission webhooks, does no NPU virtualization, and deploys no kubelet plugin. +- **Ascend DRA driver** (driver name `ascend.project-hami.io`, from [4pdOss/hami-dra-driver](https://github.com/4pdOss/hami-dra-driver)) decides _how a device is allocated_: discovery, ResourceSlice publication, kubelet Prepare/Unprepare, CDI injection. +- **HAMivNPUCore** decides _how a device is shared_: `libvnpu.so` interception plus the in-container limiter enforce the memory quota (`NPU_MEM_QUOTA`) and the compute time slice (`NPU_PRIORITY`). + +## Prerequisites + +- Kubernetes 1.34 or newer (DRA core APIs are GA since 1.34). This lab was verified on 1.35.7, single node, control-plane and worker on the same machine. +- The `DRAConsumableCapacity` feature gate enabled on kube-apiserver, kube-scheduler, and kubelet. Capacity requests and `allowMultipleAllocations` accounting depend on it. +- containerd with CDI enabled: `enable_cdi = true` and `cdi_spec_dirs = ["/etc/cdi", "/var/run/cdi"]`. +- Ascend driver 25.5 or newer on an ARM (aarch64) host, with device-share mode enabled on each NPU used for soft slicing. +- The `ascend` RuntimeClass present (Pods need `runtimeClassName: ascend`). +- cert-manager (HAMi-DRA uses it to issue webhook certificates). + +### Verification environment + +| Component | Version / Value | +| :-- | :-- | +| Hardware | 2 × Ascend 310P3 (NPU ID 4 / 5, PCIe 0000:81:00.0 / 0000:85:00.0, 21525 MB memory per card) | +| OS | Kylin Linux Advanced Server V10 (Lance), aarch64, kernel 4.19.90-52.48.v2207.ky10 | +| Kubernetes | v1.35.7, single node | +| Container runtime | containerd 1.7.29, CDI enabled | +| Feature gates | `DRAConsumableCapacity=true` on kube-apiserver, kube-scheduler, kubelet | +| Ascend driver / firmware | 25.5.1 / 7.8.0.6.201 | +| HAMi-DRA | 0.2.3 (chart and image tag) | +| Ascend DRA driver | chart `ascend-dra-driver-0.1.1` (app 0.1.0), image `projecthami/ascend-dra-driver:uuid-fix-20260909` (development build, see Step 2) | +| Coexisting | HAMi enterprise 2.10.0-r1 (hami-scheduler, hami-ascend-device-plugin, hami-resource-pool-manager), cert-manager v1.21.1 | + +### Check the feature gate + +```bash +ps -ef | grep -E "kube-apiserver|kube-scheduler|kubelet" | grep -o "feature-gates=.*" +``` + +```text +feature-gates=DRAConsumableCapacity=true +feature-gates=DRAConsumableCapacity=true +``` + +You should see the gate on every control-plane component listed. If any of them misses it, capacity requests are rejected or accounting silently does not happen. + +### Check containerd CDI + +```bash +grep -E "enable_cdi|cdi_spec_dirs" /etc/containerd/config.toml +``` + +```text + cdi_spec_dirs = ["/etc/cdi", "/var/run/cdi"] + enable_cdi = true +``` + +### Enable device-share on the NPUs + +`-i` is the NPU ID from `npu-smi info -l` (4 and 5 on the verification server). The command applies to all chips on the given NPU: + +```bash +npu-smi set -t device-share -i 4 -d 1 +npu-smi set -t device-share -i 5 -d 1 +npu-smi info -t device-share +``` + +```text + NPU ID : 4 + Chip Count : 1 + + Device-share Status : True + Chip ID : 0 + + NPU ID : 5 + Chip Count : 1 + + Device-share Status : True + Chip ID : 0 +``` + +`Device-share Status: True` on every card you plan to share. Also confirm the RuntimeClass exists: + +```bash +kubectl get runtimeclass ascend +``` + +```text +NAME HANDLER AGE +ascend ascend ... +``` + +### Coexistence with an existing HAMi core installation + +If the cluster already runs HAMi core or HAMi enterprise (hami-scheduler, hami-webhook, hami-ascend-device-plugin), their mutating webhook also intercepts `huawei.com/Ascend310P*` resources and rewrites the same Pod a second time. DRA workloads must opt out on two levels: + +```bash +kubectl create ns dra-ascend-e2e +kubectl label ns dra-ascend-e2e hami.io/webhook=ignore +``` + +The Pods in this lab additionally carry the label `hami.io/webhook: ignore`. On a clean cluster without HAMi core this step is harmless but unnecessary. + +## Step 1: Install HAMi DRA 0.2.3 + +cert-manager is a dependency; the verification cluster already had v1.21.1. On a clean cluster install cert-manager first. + +From the official Helm repository: + +```bash +helm repo add hami-dra https://project-hami.github.io/HAMi-DRA +helm repo update +helm search repo hami-dra # confirm 0.2.3 +helm install hami-dra hami-dra/hami-dra -n hami-system \ + --version 0.2.3 -f ascend-values.yaml +``` + +`ascend-values.yaml` enables only the Ascend conversion: + +```yaml +deviceVendors: + - ascend # Ascend conversion only (910A/B2/B3/B4/B4-1/310P/910C) + +drivers: + nvidia: + enabled: false # no NVIDIA kubelet driver + fake: + enabled: false + +monitor: + enabled: false # optional Prometheus metrics component, not used here + +certs: + certManager: + enabled: true +``` + +Verify the webhook came up: + +```bash +kubectl get pods -n hami-system +kubectl get mutatingwebhookconfigurations,validatingwebhookconfigurations | grep hami-dra +``` + +```text +NAME READY STATUS RESTARTS AGE +hami-ascend-device-plugin-vp6tm 1/1 Running 0 5h22m +hami-dra-webhook-5b85c54c78-hbtjc 1/1 Running 0 13s +hami-resource-pool-manager-7b6bd856fb-phd7r 1/1 Running 0 5h23m +hami-scheduler-7dccfd9b96-7spsm 2/2 Running 0 5h28m +``` + +```text +mutatingwebhookconfiguration.admissionregistration.k8s.io/hami-dra-mutatingwebhookconfiguration +validatingwebhookconfiguration.admissionregistration.k8s.io/hami-dra-validatingwebhookconfiguration +``` + +The hami-scheduler and hami-ascend-device-plugin Pods in this listing belong to the pre-existing HAMi enterprise installation, not to HAMi-DRA. HAMi-DRA itself runs exactly one deployment, the webhook. It deploys no kubelet plugin: the node-side driver for Ascend comes from the next step. + +## Step 2: Install the Ascend DRA Driver + +```bash +git clone --recurse-submodules https://github.com/4pdOss/hami-dra-driver.git +cd hami-dra-driver +helm upgrade --install ascend-dra-driver \ + deployments/helm/ascend-dra-driver \ + -n ascend-dra-driver --create-namespace +``` + +The chart defaults to HAMivNPUCore mode. Do not set `kubeletPlugin.fullCardAndTraditionalVNPU.enabled=true`: that switches to the full-card / template-based vNPU path, which is still under development and disables the HAMivNPUCore gate. + +:::note About the image version used in this lab + +The verification run used chart `ascend-dra-driver-0.1.1` with the image overridden to the development build `projecthami/ascend-dra-driver:uuid-fix-20260909`, which fixes a ResourceSlice uuid generation bug in earlier builds. Behavior may change before an official release; pin what you tested against. + +::: + +Check the driver DaemonSet and the DRA objects it publishes: + +```bash +kubectl get pods -n ascend-dra-driver +kubectl get deviceclass +kubectl get resourceslice +``` + +Expected: all driver Pods Running, one DeviceClass named `hami-vnpu-core.project-hami.io`, and one ResourceSlice per node with driver `ascend.project-hami.io`. + +## Step 3: Inspect the DeviceClass and the ResourceSlice + +The DeviceClass filters devices by driver and type with a CEL expression: + +```bash +kubectl get deviceclass hami-vnpu-core.project-hami.io -o yaml +``` + +```yaml +apiVersion: resource.k8s.io/v1 +kind: DeviceClass +metadata: + name: hami-vnpu-core.project-hami.io +spec: + selectors: + - cel: + expression: |- + device.driver == "ascend.project-hami.io" && + device.attributes["ascend.project-hami.io"].type == "HAMivNPUCore" +``` + +The ResourceSlice is where Kubernetes sees the NPUs for the first time: + +```bash +kubectl get resourceslice +``` + +```text +NAME DRIVER NODE AGE +aio-node74-arm-ascend.project-hami.io-gktzs ascend.project-hami.io aio-node74-arm 3m +``` + +Expand it (abridged; `kubectl get resourceslice -o yaml` for the full object): + +```yaml +apiVersion: resource.k8s.io/v1 +kind: ResourceSlice +metadata: + name: aio-node74-arm-ascend.project-hami.io-gktzs + ownerReferences: + - kind: Node + name: aio-node74-arm +spec: + driver: ascend.project-hami.io + nodeName: aio-node74-arm + pool: + name: aio-node74-arm + resourceSliceCount: 1 + devices: + - allowMultipleAllocations: true # lets multiple claims consume the same device + name: npu-0-0 + attributes: + brand: { string: Huawei } + index: { int: 0 } + model: { string: 310P3 } + physicalID: { int: 0 } + productName: { string: 310P3 } + type: { string: HAMivNPUCore } + uuid: { string: 68496E64-20E05477-92C31323-6E78030A-BD003019 } + capacity: + cores: + value: "100" # full card counted as 100 + requestPolicy: { default: "100", validRange: { min: "0", max: "100", step: "1" } } + memory: + value: 21525Mi # matches npu-smi + requestPolicy: { default: 21525Mi, validRange: { min: 1Mi, max: 21525Mi, step: 1Mi } } + - allowMultipleAllocations: true + name: npu-1-0 + attributes: + index: { int: 1 } + physicalID: { int: 2 } + productName: { string: 310P3 } + type: { string: HAMivNPUCore } + uuid: { string: D8496E64-20C101B1-C0D42F23-AED8030A-40003039 } + capacity: + cores: { value: "100", ... } + memory: { value: 21525Mi, ... } +``` + +Four details matter for the rest of the lab: + +- Each 310P3 is one device (`npu-0-0`, `npu-1-0`), not a card count on the node. +- The `uuid` attribute is the chip UUID, for example `68496E64-20E05477-92C31323-6E78030A-BD003019`. It is **not** a node-name-plus-index string. Card-selection annotations must copy the value from here, never from memory. +- Capacity is published along two dimensions, `memory` (MiB) and `cores` (percent of the full card), each with a `requestPolicy` declaring the valid request range and step. +- `allowMultipleAllocations: true` permits several ResourceClaims on the same device. Without it there is no NPU sharing. + +The mapping between ResourceSlice devices and physical cards, cross-checked from the `npu-smi info` process list during the sharing experiments: + +| ResourceSlice device | uuid (first 8) | npu-smi NPU ID | PCIe Bus | +| :------------------- | :------------- | :------------- | :----------- | +| npu-0-0 | 68496E64 | 4 | 0000:81:00.0 | +| npu-1-0 | D8496E64 | 5 | 0000:85:00.0 | + +## Step 4: Run the First Ascend Workload + +The request syntax is unchanged HAMi syntax: + +- Count: `huawei.com/Ascend310P` (integer, cards) +- Memory: `huawei.com/Ascend310P-memory` (MiB) +- Compute: `huawei.com/Ascend310P-core` (percent) +- Card selection: `hami.io/use-Ascend310P-uuid`, the value copied from the ResourceSlice `uuid` +- Model selection: `hami.io/use-nputype`, matched against `productName` (not used in this lab) + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: ascend-share-a + namespace: dra-ascend-e2e + labels: + hami.io/webhook: ignore + annotations: + hami.io/use-Ascend310P-uuid: "68496E64-20E05477-92C31323-6E78030A-BD003019" # pins to npu-0-0 +spec: + runtimeClassName: ascend + containers: + - name: app + image: quay.io/ascend/vllm-ascend:v0.23.0-310p + command: ["sh", "-c", "sleep 3600"] + resources: + limits: + huawei.com/Ascend310P: 1 + huawei.com/Ascend310P-memory: "8192" + huawei.com/Ascend310P-core: "50" +``` + +Three choices in this manifest: the `hami.io/webhook: ignore` label keeps the coexisting HAMi core webhook away; `runtimeClassName: ascend` routes containerd through the Ascend Docker Runtime; the `vllm-ascend` image ships CANN, torch_npu, and npu-smi so the container can be inspected from inside later. + +```bash +kubectl apply -f pod.yaml +kubectl get pod ascend-share-a -n dra-ascend-e2e +``` + +```text +NAME READY STATUS RESTARTS AGE IP NODE +ascend-share-a 1/1 Running 0 3m45s 10.244.0.28 aio-node74-arm +``` + +Running only proves Kubernetes is satisfied. Before trusting it, look at what the webhook actually did. + +## Step 5: Observe the Request-to-ResourceClaim Conversion + +Save the mutated Pod and diff it against what you submitted: + +```bash +kubectl get pod ascend-share-a -n dra-ascend-e2e -o yaml > mutated-pod.yaml +``` + +The three `huawei.com/*` entries in `resources.limits` are gone. In their place: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: ascend-share-a + namespace: dra-ascend-e2e + labels: + hami.io/dra: "true" # added by the webhook + hami.io/webhook: ignore + annotations: + hami.io/use-Ascend310P-uuid: 68496E64-20E05477-92C31323-6E78030A-BD003019 +spec: + runtimeClassName: ascend + containers: + - name: app + image: quay.io/ascend/vllm-ascend:v0.23.0-310p + resources: + claims: # container-level reference + - name: dra-ascend-e2e-ascend-share-a-app-ascend310p + resourceClaims: # pod-level reference + - name: dra-ascend-e2e-ascend-share-a-app-ascend310p + resourceClaimName: dra-ascend-e2e-ascend-share-a-app-ascend310p +``` + +The generated ResourceClaim, after allocation: + +```bash +kubectl get resourceclaim -n dra-ascend-e2e +kubectl get resourceclaim dra-ascend-e2e-ascend-share-a-app-ascend310p -n dra-ascend-e2e -o yaml +``` + +```text +NAME STATE AGE +dra-ascend-e2e-ascend-share-a-app-ascend310p allocated,reserved 3m +``` + +```yaml +apiVersion: resource.k8s.io/v1 +kind: ResourceClaim +metadata: + name: dra-ascend-e2e-ascend-share-a-app-ascend310p + namespace: dra-ascend-e2e + finalizers: + - resource.kubernetes.io/delete-protection +spec: + devices: + requests: + - name: npu + exactly: + allocationMode: ExactCount + count: 1 + deviceClassName: hami-vnpu-core.project-hami.io + capacity: + requests: + cores: "50" # percent, passed through + memory: "8589934592" # 8192 MiB x 1024 x 1024 = 8 GiB in bytes + selectors: + - cel: + expression: device.driver == "ascend.project-hami.io" && device.attributes["ascend.project-hami.io"].type == "HAMivNPUCore" + - cel: + expression: device.attributes["ascend.project-hami.io"].uuid in ["68496E64-20E05477-92C31323-6E78030A-BD003019"] +status: + allocation: + devices: + results: + - consumedCapacity: + cores: "50" + memory: 8Gi + device: npu-0-0 # the scheduler's pick + driver: ascend.project-hami.io + pool: aio-node74-arm + request: npu + shareID: be06132e-0aff-4efc-bd8e-42a4e06ed385 + nodeSelector: + nodeSelectorTerms: + - matchFields: + - key: metadata.name + operator: In + values: [aio-node74-arm] + reservedFor: + - name: ascend-share-a + resource: pods +``` + +Field-by-field, this is where HAMi semantics enter DRA semantics: + +| ResourceClaim field | Comes from | Conversion | +| :-- | :-- | :-- | +| `count: 1` | `huawei.com/Ascend310P: 1` | integer passed through | +| `capacity.requests.memory: 8589934592` | `-memory: 8192` (MiB) | MiB to bytes, `value x 1024 x 1024` | +| `capacity.requests.cores: "50"` | `-core: 50` | percent passed through | +| selectors, entry 1 | fixed | matches driver plus `type == HAMivNPUCore`, same source as the DeviceClass | +| selectors, entry 2 | annotation `hami.io/use-Ascend310P-uuid` | `uuid in ["..."]`; `no-use-*` annotations invert the expression | +| (if used) selectors, entry 3 | annotation `hami.io/use-nputype` | matched against `productName` | +| claim name | namespace + pod + container | truncated to DNS-1123's 63 characters with a sha256 suffix when too long | + +Two details worth pausing on. The unit conversion: HAMi's `-memory` is MiB, DRA capacity is a Quantity in bytes, so 8192 MiB becomes `8589934592`; you write HAMi syntax, the accounting is DRA's. And the `shareID`: every allocated claim gets one, and it is the accounting voucher of the `DRAConsumableCapacity` model (KEP-5075). Each consumption on the same device is accounted independently, which the sharing step will show as two distinct shareIDs. + +## Step 6: Verify the NPU Allocation + +Three layers, from the cluster down to the node. + +**Kubernetes layer.** The claim is `allocated,reserved`, and `status.allocation.devices.results[0].device` is `npu-0-0`, as captured above. + +**Container layer.** The quota and device environment: + +```bash +kubectl exec -it ascend-share-a -n dra-ascend-e2e -- env | grep -E "NPU_|ASCEND_" +kubectl exec -it ascend-share-a -n dra-ascend-e2e -- ls /dev/davinci* /dev/davinci_manager /dev/devmm_svm +``` + +```text +NPU_PRIORITY=50 +ASCEND_VISIBLE_DEVICES=0 +NPU_GLOBAL_SHM_PATH=/hami-shared-region/0_global_registry +NPU_LOCAL_SHM_PATH=/hami-vnpu-shmem/vnpu_local_shmem +NPU_MEM_QUOTA=8192 +``` + +```text +/dev/davinci0 +/dev/davinci_manager +/dev/devmm_svm +``` + +`NPU_MEM_QUOTA=8192` and `NPU_PRIORITY=50` are the HAMivNPUCore memory quota (MiB) and compute time slice, matching the `-memory 8192` and `-core 50` requests. `ASCEND_VISIBLE_DEVICES=0` and `/dev/davinci0` come from CDI injection, as does the `libvnpu.so` referenced by `ld.so.preload` inside the container, the entry point of the interception. + +**Node layer.** The driver writes one CDI spec per claim under `CDI_ROOT=/var/run/cdi` during Prepare: + +```bash +ls /var/run/cdi/ +``` + +```text +k8s.ascend.project-hami.io-npu_8ec6bbf9-8ce0-4aa1-bc57-7078653d090c.yaml # this claim, named by uid +k8s.ascend.project-hami.io-npu_common.yaml +``` + +containerd, with `enable_cdi`, applies the spec at container creation. The DRA allocation lands in the container runtime through the standard CDI interface. The final device-level proof arrives in Step 8, when a real workload runs and the container's torch_npu reports an 8 GiB device instead of the 21 GiB card. + +## Step 7: Share One NPU Between Two Pods + +The core experiment: two Pods request parts of the same physical 310P3. The key is that both Pods carry the same uuid annotation. With two cards in the machine, leaving the scheduler free could spread the Pods across cards and prove nothing: + +```text +Physical Ascend 310P (uuid 68496E64-..., device npu-0-0, capacity 21525 MiB / 100 cores) + ++-- Pod A (ascend-share-a): memory 8192 MiB + core 50 ++-- Pod B (ascend-share-b): memory 8192 MiB + core 50 +``` + +Create the second Pod as a copy of the first: + +```bash +sed 's/ascend-share-a/ascend-share-b/' pod.yaml | kubectl apply -f - +kubectl get resourceclaim -n dra-ascend-e2e +``` + +```text +NAME STATE AGE +dra-ascend-e2e-ascend-share-a-app-ascend310p allocated,reserved 6m +dra-ascend-e2e-ascend-share-b-app-ascend310p allocated,reserved 2m +``` + +Compare the allocation results of both claims: + +```yaml +# claim-a +results: + - device: npu-0-0 # the same device + consumedCapacity: { cores: "50", memory: 8Gi } + shareID: be06132e-0aff-4efc-bd8e-42a4e06ed385 + +# claim-b +results: + - device: npu-0-0 # the same device + consumedCapacity: { cores: "50", memory: 8Gi } + shareID: 5554db84-1574-46bd-a3ea-0b7197fbbbda # an independent accounting voucher +``` + +Both claims landed on `npu-0-0`, each with its own shareID. Before allocating B, the scheduler had already debited A's 8 GiB and 50 cores against the device; that is exactly what `DRAConsumableCapacity` plus `allowMultipleAllocations: true` buys. On the Pod and device level: + +```bash +kubectl get pods -n dra-ascend-e2e -o wide +``` + +```text +NAME READY STATUS IP NODE +ascend-share-a 1/1 Running 10.244.0.28 aio-node74-arm +ascend-share-b 1/1 Running 10.244.0.29 aio-node74-arm +``` + +Both containers see the same device node and their own quotas: + +```text +pod a: /dev/davinci0 NPU_MEM_QUOTA=8192 NPU_PRIORITY=50 ASCEND_VISIBLE_DEVICES=0 +pod b: /dev/davinci0 NPU_MEM_QUOTA=8192 NPU_PRIORITY=50 ASCEND_VISIBLE_DEVICES=0 +``` + +In soft-slicing mode both containers see the same `/dev/davinci0`, not two vNPU device nodes; the isolation is done by libvnpu in user space. After a real workload runs in Step 8, the node's `npu-smi` process list shows both containers' processes on NPU 4. + +## Step 8: Verify Isolation and Quota Enforcement + +Run a PyTorch/torch_npu tensor allocation test inside Pod A (the image carries the full CANN runtime). Allocate toward the quota, then past it: + +```text +# the limiter (HAMivNPUCore's in-container manager) registers the process +[2026-09-17T10:18:46Z INFO limiter::worker] [Worker PID:381] Initialize SchedulerClient... +[2026-09-17T10:18:46Z INFO limiter::worker] [Worker PID:381] Registered at proc slot 1 + +# at ~6150 MB used, a 2050 MB request is refused +[2026-09-17T10:22:22Z WARN limiter::worker] [Worker PID:1139] Memory Quota Exceeded! + Request: 2050 MB, Used: 6150 MB, Limit: 8192 MB + +# the application sees an in-container OOM (note the 8.00 GiB total, not the card's 21 GiB) +torch.OutOfMemoryError: NPU out of memory. Tried to allocate 2.00 GiB + (NPU 0; 8.00 GiB total capacity; 2.00 GiB already allocated; ...) +``` + +This one capture proves two things at once: the quota is enforced in user space by libvnpu and the limiter, with over-quota requests converted to a container OOM; and the device as seen from inside the container has been virtualized to 8 GiB, matching the `-memory 8192` request rather than the physical 21525 MB. On the node, both containers' processes sit on the same physical card: + +```bash +npu-smi info # process list, abridged +``` + +```text +| NPU Chip | Process id | Process name | Process memory(MB) | +| 4 0 | 44533 | python3 | 2156 | +| 4 0 | 49806 | python3 | 2156 | +``` + +`NPU_PRIORITY=50` was injected the same way and constrains compute through time-slice scheduling. This lab verified the injection but did not run a long-running saturated comparison workload, so it draws no conclusion about the effective strength of time-slice isolation on the 310P; the scheduling-side accounting of the compute dimension is fully covered in the next step. + +## Step 9: Exhaust Capacity, Release, and Reallocate + +Pods A and B now hold 16384 MiB / 100 cores of `npu-0-0`; the device has 5141 MiB / 0 cores left. Submit Pod C requesting 8192 MiB / 50 cores with the same uuid annotation: + +```bash +kubectl apply -f pod-c.yaml +kubectl get pod ascend-share-c -n dra-ascend-e2e +kubectl describe pod ascend-share-c -n dra-ascend-e2e | tail -4 +kubectl get resourceclaim -n dra-ascend-e2e +``` + +```text +NAME READY STATUS RESTARTS AGE +ascend-share-c 0/1 Pending 0 133m +``` + +```text + Warning FailedScheduling 3m3s (x27 over 133m) default-scheduler + 0/1 nodes are available: 1 cannot allocate all claims. still not schedulable, + preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. +``` + +```text +NAME STATE AGE +dra-ascend-e2e-ascend-share-a-app-ascend310p allocated,reserved 157m +dra-ascend-e2e-ascend-share-b-app-ascend310p allocated,reserved 153m +dra-ascend-e2e-ascend-share-c-app-ascend310p pending 133m +``` + +Pod C's claim stays `pending` with no allocation result. The second card `npu-1-0` is idle, but the uuid annotation pins Pod C to `npu-0-0`, and the scheduler refuses honestly. The refusal happens inside the scheduler; no node-side device plugin reporting is involved. + +### Control experiment: the memory dimension alone + +Pod C oversubscribes both dimensions at once (8192 > 5141 memory, 50 > 0 cores), so by itself it cannot tell which dimension rejected it. A control pair on the second card isolates the memory dimension: + +| Pod | Request | npu-1-0 state | Result | +| :-- | :-- | :-- | :-- | +| ascend-ctrl-a | 8192 MiB / 50 cores (uuid D8496E64) | initial 21525 / 100 | Running, leaving 13333 MiB / 50 cores | +| ascend-ctrl-b | 14000 MiB / 40 cores (same uuid) | 13333 / 50 left | Pending | + +ctrl-b's compute request fits (40 ≤ 50) but the memory request does not (14000 > 13333): + +```text + Warning FailedScheduling 4m16s (x2 over 9m17s) default-scheduler + 0/1 nodes are available: 1 cannot allocate all claims. still not schedulable, ... +``` + +Memory and cores are accounted as two independent dimensions: the scheduler reconciles every allocated claim's `consumedCapacity` against the ResourceSlice's `capacity.value`, dimension by dimension. + +### Release and reallocation + +Delete Pod A and watch all three layers recover: + +```bash +kubectl delete pod ascend-share-a -n dra-ascend-e2e +kubectl get resourceclaim -n dra-ascend-e2e +``` + +```text +NAME STATE AGE +dra-ascend-e2e-ascend-ctrl-a-app-ascend310p allocated,reserved 10m +dra-ascend-e2e-ascend-ctrl-b-app-ascend310p pending 10m +dra-ascend-e2e-ascend-share-b-app-ascend310p allocated,reserved 164m +dra-ascend-e2e-ascend-share-c-app-ascend310p allocated,reserved 143m # pending to allocated +``` + +share-a's claim disappeared: the HAMi-DRA validating webhook watches Pod deletions for the `hami.io/dra` label and deletes the corresponding ResourceClaim. On the node, the driver's Unprepare removed share-a's CDI spec and a new one appeared for share-c: + +```text +# before +k8s.ascend.project-hami.io-npu_1b737f89-...yaml (share-b) +k8s.ascend.project-hami.io-npu_8ec6bbf9-...yaml (share-a, to be reclaimed) +# after +k8s.ascend.project-hami.io-npu_1b737f89-...yaml (share-b) +k8s.ascend.project-hami.io-npu_8a2fd648-...yaml (share-c, new allocation) +``` + +With the capacity restored (16384 - 8192 = 8192 MiB used, 13333 ≥ 8192 free, 50 ≤ 50 cores), Pod C was scheduled within 18 seconds and allocated the same device: + +```text + Warning FailedScheduling 3m42s (x29 over 143m) default-scheduler (29 prior refusals) + Normal Scheduled 18s default-scheduler + Successfully assigned dra-ascend-e2e/ascend-share-c to aio-node74-arm + Normal Started 16s kubelet Container started + +# share-c allocation: the same npu-0-0, inheriting the released capacity +{"consumedCapacity":{"cores":"50","memory":"8Gi"},"device":"npu-0-0", + "driver":"ascend.project-hami.io","pool":"aio-node74-arm", + "shareID":"70cbda02-b70e-47d3-b9c8-c5c5c608a1fb"} +``` + +The full lifecycle closes: allocate(A), share(A + B on npu-0-0), reject(C, insufficient capacity), release(A), reallocate(C onto the same device). + +:::note Boundary of the claim cleanup + +Claim deletion is the HAMi-DRA validating webhook's job. If the webhook is down when a Pod is deleted, the claim can linger (still holding capacity in the accounting) until it is cleaned up manually or the webhook recovers; see Troubleshooting. + +::: + +## Troubleshooting + +### Conflicts with the HAMi core webhook + +- **Symptom**: with HAMi core or enterprise already in the cluster, DRA Pods get rewritten twice, carry unexpected annotations, or schedule erratically. +- **Cause**: the HAMi core mutating webhook also intercepts `huawei.com/Ascend310P*` resources. +- **Check**: `kubectl get mutatingwebhookconfigurations | grep hami`; diff the Pod spec before and after submission. +- **Fix**: the `hami.io/webhook: ignore` label on both the namespace and the Pod, as in this lab. + +### CEL selectors that do not match the ResourceSlice + +- **Symptom**: the Pod stays Pending with `cannot allocate all claims`; the claim stays `pending`. +- **Cause**: the uuid annotation does not match the published uuid (for example a hand-written node-name-plus-index guess). Early driver builds also generated wrong uuids; this lab used the `uuid-fix-20260909` build for exactly this reason. +- **Check**: `kubectl get resourceslice -o yaml | grep -A2 uuid` against the Pod annotation. +- **Fix**: always copy the uuid from the ResourceSlice; match `hami.io/use-nputype` against `productName` (for example `310P3`). + +### DRAConsumableCapacity not enabled + +- **Symptom**: ResourceClaims with `capacity.requests` are rejected, or scheduling does not account capacity (oversubscription succeeds, sharing misbehaves). +- **Cause**: capacity requests and `allowMultipleAllocations` accounting depend on the gate. +- **Check**: `ps -ef | grep -E "kube-apiserver|kube-scheduler|kubelet" | grep -o "feature-gates=.*"`. +- **Fix**: add `DRAConsumableCapacity=true` to all three components and restart them. + +### device-share not enabled + +- **Symptom**: driver Prepare fails, or containers cannot use the soft-sliced device. +- **Cause**: HAMivNPUCore requires the chip's container-share mode, Ascend driver 25.5 or newer, and an ARM host. +- **Check**: `npu-smi info -t device-share`; `/usr/local/Ascend/driver/version.info`. +- **Fix**: `npu-smi set -t device-share -i -d 1` (applies to all chips on that NPU). + +### Pod Running but no NPU in the container + +- **Symptom**: the claim is allocated and the Pod Running, but the container has no `/dev/davinci*` and no quota environment variables. +- **Cause**: `runtimeClassName: ascend` missing, or the image lacks CANN and npu-smi (injection comes from CDI plus the RuntimeClass; the user-space tools come from the image). +- **Check**: `kubectl get pod -o jsonpath='{.spec.runtimeClassName}'`; `kubectl exec ... -- ls /dev/davinci*`. +- **Fix**: set the RuntimeClass; use an image with the CANN runtime (this lab used `quay.io/ascend/vllm-ascend:v0.23.0-310p`). + +### ResourceClaim leaks + +- **Symptom**: the Pod is gone but its ResourceClaim remains `allocated,reserved`, holding capacity and blocking later Pods. +- **Cause**: the validating webhook was down or missing RBAC when the Pod was deleted. The claim's `resource.kubernetes.io/delete-protection` finalizer prevents accidental deletion, which also means it never disappears on its own. +- **Check**: `kubectl get resourceclaim -A` against the live `hami.io/dra=true` Pods. +- **Fix**: delete orphaned claims manually; check the hami-dra-webhook Pod and its logs. + +## Cleanup + +```bash +kubectl delete pod ascend-share-a ascend-share-b ascend-share-c \ + ascend-ctrl-a ascend-ctrl-b -n dra-ascend-e2e --ignore-not-found +kubectl get resourceclaim -A | grep ascend310p | awk '{print $1, $2}' | \ + xargs -n2 kubectl delete resourceclaim -n 2>/dev/null || true +kubectl delete ns dra-ascend-e2e --ignore-not-found + +helm uninstall ascend-dra-driver -n ascend-dra-driver +kubectl delete ns ascend-dra-driver --ignore-not-found + +helm uninstall hami-dra -n hami-system +``` + +Leave the feature gates and the `ascend` RuntimeClass in place if other work uses them. On the shared lab machine, `device-share` can stay enabled; disabling it affects every workload on the card, HAMi core included. + +## What This Lab Proved + +| Claim | Evidence | +| :-- | :-- | +| Ascend 310P discovery and ResourceSlice publication | 2 devices with uuid, capacity, and requestPolicy | +| HAMi request to ResourceClaim conversion | mutated Pod, claim YAML, MiB-to-bytes `8589934592`, CEL selectors | +| Scheduler allocation via native DRA | `allocated,reserved` claim, allocation result with shareID | +| Device injection through CDI | in-container env and `/dev/davinci0`, CDI spec in `/var/run/cdi` | +| Two Pods share one physical NPU | both claims on `npu-0-0`, independent shareIDs | +| Memory isolation | limiter refusal at 8192 MB, container OOM, torch_npu reports 8.00 GiB total | +| Compute isolation | `NPU_PRIORITY=50` injected (injection verified; time-slice strength not quantified) | +| Independent two-dimension accounting | Pod C Pending; control pair isolates the memory dimension | +| Release and reallocation | claim deleted, CDI spec swapped, Pod C rescheduled in 18 seconds | + +## Next Steps + +- The concepts and design behind this path: [How to use HAMi DRA](/docs/installation/how-to-use-hami-dra) +- The same conversion flow on NVIDIA GPUs: [Lab 4: GPU Slicing with Dynamic Resource Allocation](/tutorials/labs/hami-dra) and [Lab 11: KServe Inference with HAMi DRA GPU Sharing](/tutorials/labs/kserve-hami-dra) +- The same hardware under the device-plugin path: [Lab 13: Soft-Slicing Ascend 310P3 vNPU with Volcano and HAMi-core](/tutorials/labs/volcano-ascend-vnpu) +- Components: [Project-HAMi/HAMi-DRA](https://github.com/Project-HAMi/HAMi-DRA) · [4pdOss/hami-dra-driver](https://github.com/4pdOss/hami-dra-driver) · [Project-HAMi/hami-vnpu-core](https://github.com/Project-HAMi/hami-vnpu-core) diff --git a/tutorials/overview.md b/tutorials/overview.md index 9d1daa898..4af3b3e76 100644 --- a/tutorials/overview.md +++ b/tutorials/overview.md @@ -31,3 +31,4 @@ Each lab lists its own prerequisites. - **Lab 13** builds Volcano and the ascend-device-plugin from source on an Ascend 310P3 ARM server and verifies hami-vnpu-core soft slicing, binpack card sharing, and per-container metrics. - **Lab 14** installs HAMi v2.10.0 on a four-T4 GKE node and observes the composable `gpu-scheduler-policy` chains (`spread`, `binpack`, `mutex`, `mutex,binpack`) through allocation annotations and scheduler logs. - **Lab 16** installs HAMi v2.10.0 on a seven-GPU RTX PRO 6000 server and verifies the complete Dynamic MIG lifecycle: per-Pod placement, mixed profiles, selective reclamation, device-plugin restart recovery, and spillover to a second GPU. +- **Lab 17** installs HAMi DRA 0.2.3 and the Ascend DRA driver on an Ascend 310P3 node and verifies the request-to-ResourceClaim conversion, two-Pod NPU sharing, memory quota enforcement, and scheduler capacity accounting. From c59a405890055845d2a00f0e0d3cb6f3f1355e10 Mon Sep 17 00:00:00 2001 From: Jimmy Song Date: Thu, 8 Oct 2026 18:35:56 +0800 Subject: [PATCH 02/13] Renumber Ascend DRA lab from Lab 18 to Lab 19 Lab 18 is taken upstream by hami-linkerd-noisy-neighbor, so the Ascend NPU Sharing lab becomes Lab 19 across tutorials, overview, and the companion blog post (en + zh). Adds blog diagrams. Signed-off-by: Jimmy Song --- blog/hami-dra-npu-sharing/index.md | 114 ++++++++++++----- .../hami-dra-npu-sharing/index.md | 116 +++++++++++++----- .../current/labs/ascend-hami-dra.md | 4 +- .../current/overview.md | 2 +- .../img/hami-dra-npu-sharing/dra-model-zh.png | 3 + static/img/hami-dra-npu-sharing/dra-model.png | 3 + .../hami-dra-design-zh.png | 3 + .../hami-dra-npu-sharing/hami-dra-design.png | 3 + .../webhook-conversion-zh.png | 3 + .../webhook-conversion.png | 3 + .../where-hami-stuck-zh.png | 3 + .../hami-dra-npu-sharing/where-hami-stuck.png | 3 + tutorials/labs/ascend-hami-dra.md | 4 +- tutorials/overview.md | 2 +- 14 files changed, 193 insertions(+), 73 deletions(-) create mode 100644 static/img/hami-dra-npu-sharing/dra-model-zh.png create mode 100644 static/img/hami-dra-npu-sharing/dra-model.png create mode 100644 static/img/hami-dra-npu-sharing/hami-dra-design-zh.png create mode 100644 static/img/hami-dra-npu-sharing/hami-dra-design.png create mode 100644 static/img/hami-dra-npu-sharing/webhook-conversion-zh.png create mode 100644 static/img/hami-dra-npu-sharing/webhook-conversion.png create mode 100644 static/img/hami-dra-npu-sharing/where-hami-stuck-zh.png create mode 100644 static/img/hami-dra-npu-sharing/where-hami-stuck.png diff --git a/blog/hami-dra-npu-sharing/index.md b/blog/hami-dra-npu-sharing/index.md index 7b8a002ad..41a3e590c 100644 --- a/blog/hami-dra-npu-sharing/index.md +++ b/blog/hami-dra-npu-sharing/index.md @@ -1,31 +1,53 @@ --- title: "From HAMi to HAMi-DRA: The Evolution of Heterogeneous Compute Management" -date: "2026-09-20" +date: "2026-09-24" description: "What HAMi DRA is, why HAMi needs it, and how it is designed: the four bottlenecks of the Device Plugin era (quota accounting, preemption compatibility, scheduling performance, deadlocks), the HAMi-DRA webhook conversion, the DRA driver ecosystem timeline, the role HAMi-core keeps, and adoption guidance with real-cluster examples." authors: [rootsongjc] tags: ["HAMi", "DRA", "Ascend", "NPU Sharing", "Kubernetes"] --- -Kubernetes 1.34 took the DRA (Dynamic Resource Allocation) core APIs to GA, and together with the Consumable Capacity model, accelerators became first-class objects in Kubernetes' native resource model: they have attributes, they have capacity, and multiple Pods may share them. For sharing stacks built on Device Plugin, like HAMi, this raises a practical question: requesting and scheduling devices can now be done natively by Kubernetes, but isolating resources inside a container is still something Kubernetes does not handle. +On September 15, 2026, the HAMi community released [HAMi DRA](https://github.com/Project-HAMi/HAMi-DRA) 0.2.3. The version itself is a routine iteration, but the timing is worth noting: in August the Ascend DRA driver entered development preview; before that, the HAMi 2.9 release webinar declared HAMi-DRA production-ready, with NPU DRA support planned for 2.10. For anyone tracking heterogeneous compute scheduling, HAMi DRA has moved from "experimental direction" to "an option worth evaluating seriously". -[HAMi DRA](https://github.com/Project-HAMi/HAMi-DRA) is the HAMi community's answer: a migration layer that automatically converts the HAMi-style resource requests in existing workloads into native ResourceClaims, hands scheduling and accounting back to kube-scheduler, keeps runtime isolation with HAMi-core, and leaves the business side without a single line to change. The current 0.2.3 release covers NVIDIA GPUs; with the [Ascend DRA driver](https://github.com/4pdOss/hami-dra-driver) entering preview, the chain has been verified end to end on a real 310P3 cluster, and the community shipped the companion [Lab 18: Ascend NPU Sharing with HAMi DRA](/tutorials/labs/ascend-hami-dra) with every command and real output. This post is a systematic introduction to HAMi DRA: its motivation, design, usage, and current boundaries. +The discussion around it has not stopped either. Ever since Kubernetes 1.34 took DRA (Dynamic Resource Allocation) to GA, "does DRA make HAMi obsolete?" has been a frequent question in the community channels; the answer given in [Does Kubernetes DRA Replace HAMi?](/blog/does-kubernetes-dra-replace-hami) is that DRA absorbs the request-and-scheduling half, while the enforce-inside-the-container half was never something DRA set out to do, and that is exactly where HAMi stays. + +HAMi DRA is the community's engineering answer to that debate: a migration layer that automatically converts the HAMi-style resource requests in existing workloads into native ResourceClaims, hands scheduling and accounting back to kube-scheduler, keeps runtime isolation with HAMi-core, and leaves the business side without a single line to change. The current 0.2.3 release covers NVIDIA GPUs; with the Ascend DRA driver entering preview, the chain has been verified end to end on a real 310P3 cluster, and the community shipped the companion [Lab 19: Ascend NPU Sharing with HAMi DRA](/tutorials/labs/ascend-hami-dra) with every command and real output. This post is a systematic introduction to HAMi DRA: its motivation, design, usage, and current boundaries. ## What HAMi solved, and where it hit the ceiling -HAMi (Heterogeneous AI Computing Virtualization Middleware), formerly k8s-vGPU-scheduler, is now a CNCF incubating project. The problem it solves is concrete: the Device Plugin's extended resources can only express whole cards (`nvidia.com/gpu: 1`, `huawei.com/Ascend310P: 1`), while sharing an accelerator means "part of one card's memory plus part of its compute". On top of the old API, HAMi built a complete stack: a mutating webhook rewrites Pods, a scheduler extender does shared scheduling, device plugins report and mount, and HAMi-core (`libvgpu` / `libvnpu`) enforces quotas inside containers. The stack has held up in production (case figures from a [community talk](https://aicon.infoq.cn/2026/shenzhen/presentation/7168)): SF Express achieved up to 57% GPU savings through GPU sharing, SNOW cut costs by 55%, and China Merchants Bank reached 100% hardware pool utilization with topology-aware scheduling. +The problem HAMi solves is direct: the Device Plugin's extended resources can only express whole cards (`nvidia.com/gpu: 1`, `huawei.com/Ascend310P: 1`), while sharing means "part of one card's memory plus part of its compute": + +```yaml +# Device Plugin can only ask for a whole card, no slicing +resources: + limits: + nvidia.com/gpu: 1 + +# Sharing actually needs this (HAMi syntax): +resources: + limits: + huawei.com/Ascend310P: 1 + huawei.com/Ascend310P-memory: "8192" # 8 GiB of memory + huawei.com/Ascend310P-core: "50" # 50% of the compute +``` + +To deliver that, HAMi built a complete stack on the old API: a mutating webhook rewrites Pods, a scheduler extender does shared scheduling, device plugins report and mount, and HAMi-core (`libvgpu` / `libvnpu`) enforces quotas inside containers. The stack has held up in production: SF Express saved up to 57% of its GPUs, SNOW cut costs by 55%, and China Merchants Bank reached 100% hardware pool utilization with topology-aware scheduling. -But precisely because the whole stack sits on Device Plugin and scheduler extension, the ceiling is structural. The real-world bottlenecks of the existing HAMi approach group into four (see the [community talk](https://aicon.infoq.cn/2026/shenzhen/presentation/7168)): +But every one of these components is a patch on the old API, and the more patches stack up, the more visible the structural problems become. The four pains HAMi users run into most often: -| Bottleneck | Root cause | +| Pain | In plain words | | :-- | :-- | -| ResourceQuota accounting | extended resources stay outside the native quota system and need a separate quota implementation | -| Preemption compatibility | Extender API limitations make Kubernetes scheduling features such as preemption awkward | -| Scheduling performance | node-lock exclusivity: scheduling several Pods onto one node degrades concurrency to serial | -| Deadlocks | the default lock window is 5 minutes; a failed request during creation can leave a node locked | +| Quota does not bite | resources like `huawei.com/*` are invisible to native ResourceQuota; limiting usage per team means building a separate quota system | +| Preemption does not apply | preemption and other scheduler features do not recognize decisions made by an out-of-tree scheduler, so a high-priority task cannot bump a low-priority one out of the way | +| Slower when busy | several Pods scheduled to the same node are processed one by one (the node lock); the busier the cluster, the slower Pod creation gets | +| Occasional stalls | one failed request can leave a whole node locked for 5 minutes, and every new Pod on that node waits | + +None of these are bugs in the code; they are the road itself: simulating fine-grained slicing on a "whole cards per node" allocation model cannot avoid these costs. -None of these are bugs in the code; they are architectural limits: doing fine-grained allocation on a node-centric allocation model comes with exactly these costs. +![Where HAMi gets stuck on Device Plugin](/img/hami-dra-npu-sharing/where-hami-stuck.png) + +The left side of the figure is the whole chain HAMi had to build itself to deliver sharing: every layer between the Pod and the accelerator is HAMi's own. The dashed lines show where each bottleneck lives: number 1 sits in the resource declaration itself (extended resources stay invisible to native quotas), while numbers 2, 3, and 4 all sit in the Scheduler Extender stage — preemption is limited by the extender API, the node lock serializes concurrent scheduling, and a failed request can lock a node for 5 minutes. ## What DRA changed @@ -38,21 +60,40 @@ Kubernetes 1.34 took the DRA (Dynamic Resource Allocation) core APIs to GA, and The key enabler for shared scheduling is **Consumable Capacity** (KEP-5075): devices publish capacity along two independent dimensions (memory and a compute percentage), `allowMultipleAllocations` lets several claims consume one device, and kube-scheduler reconciles consumed capacity dimension by dimension, with each allocation carrying an independent shareID. "Multiple workloads fairly consuming parts of one accelerator" becomes a native scheduler capability for the first time, with no out-of-tree component simulating it. Note the feature gate needs manual enabling on 1.34/1.35 and is on by default from 1.36. +What the device and the allocation look like in a real cluster (abridged): + +```yaml +# ResourceSlice: one device per 310P3, capacity in two dimensions +devices: + - name: npu-0-0 + attributes: + productName: { string: 310P3 } + uuid: { string: 68496E64-20E05477-92C31323-6E78030A-BD003019 } + capacity: + cores: { value: "100" } + memory: { value: 21525Mi } + allowMultipleAllocations: true + +# ResourceClaim: asking for a part of it +capacity: + requests: + cores: "50" + memory: "8589934592" +``` + +![The DRA object model and per-dimension accounting](/img/hami-dra-npu-sharing/dra-model.png) + +In the figure, Claim A and Claim B each ask for a part of the same device (npu-0-0), and kube-scheduler keeps the books against the ResourceSlice, dimension by dimension; once the capacity is fully consumed, Claim C stays pending until a Pod releases its share. + The traditional Device Plugin interface can only expose simple capacity, and the scheduler discovers "not enough resource" only after landing on a node; with DRA, device attributes are matched precisely at scheduling time. The migration logic here is straightforward: the bottleneck was never HAMi's scheduling algorithm, but the API layer beneath it. ## The design of HAMi DRA HAMi DRA does not start over; it replaces only the upper part: scheduling and accounting move to the Kubernetes-native implementation, and runtime isolation stays with HAMi-core. -```mermaid -%% title: Component split of the HAMi DRA path -flowchart TD - POD["User Pod
HAMi-style resource request"] --> HOOK["HAMi-DRA Webhook
creates and reclaims ResourceClaims"] - HOOK --> DRA["Kubernetes DRA
ResourceClaim / DeviceClass
ResourceSlice / kube-scheduler"] - DRA --> DRV["DRA driver
kubelet plugin + CDI
discovery and injection"] - DRV --> CORE["HAMi-core
memory quota / compute time slice
libvgpu / libvnpu interception"] - CORE --> ACC["physical accelerator
shared by multiple Pods"] -``` +![The HAMi DRA request path, from YAML to accelerator](/img/hami-dra-npu-sharing/hami-dra-design.png) + +The whole journey starts from a single YAML. A compatible-mode user keeps writing resource names like `huawei.com/Ascend310P`, and HAMi DRA's mutating webhook converts them into a ResourceClaim at admission; a native-mode user submits the ResourceClaim directly and skips that step. From there, kube-scheduler schedules and accounts against the ResourceSlice and binds the Pod to a node. On the node, kubelet calls the DRA driver's Prepare interface, which writes a CDI spec; containerd then creates the container from the spec, with HAMi-core (`libvgpu` / `libvnpu`) injected inside to enforce the quotas. The workload finally runs on GPUs, NPUs, DCUs, and the other supported accelerators, and the numbers ① through ⑧ in the figure follow that order. Three components, three responsibilities: the **HAMi-DRA Webhook** decides "how the request is declared", the **DRA driver** decides "how a device is allocated", and **HAMi-core** decides "how a device is shared". @@ -62,13 +103,13 @@ The **DRA driver** is the node-side implementation: it discovers devices and pub **HAMi-core** stays as it is: DRA only addresses the request-and-scheduling half; the enforce-inside-the-container half (memory quotas, compute time slices) remains with `libvgpu` / `libvnpu`. That is exactly where the conclusion of [Does Kubernetes DRA Replace HAMi?](/blog/does-kubernetes-dra-replace-hami) lands architecturally. -The scheduling-performance gain comes from the same replacement (the [AICon](https://aicon.infoq.cn/2026/shenzhen/presentation/7168) and [KCD Beijing](https://dynamia.ai/zh/blog/kcd-beijing-2026-dra-gpu-scheduling) talks give mutually confirming comparisons): by riding DRA's own scheduling, HAMi-DRA avoids the node-lock performance decay; when several Pods land on the same node concurrently, the original approach's Pod creation time degrades nearly linearly (peaking in the 42-second range), while HAMi-DRA stays clearly lower (by more than 30%). This owes mainly to DRA's resource pre-binding: the allocation is settled at scheduling time, which removes scheduling conflicts and retries. The overall change summarizes as four mappings: NodeLock to No Locks, ResourceName to ResourceClaim, Device Plugin to DRA Driver, and Annotation to Attributes. +The scheduling-performance gain comes from the same replacement: by riding DRA's own scheduling, HAMi-DRA avoids the node-lock performance decay; when several Pods land on the same node concurrently, the original approach's Pod creation time degrades nearly linearly (peaking in the 42-second range), while HAMi-DRA stays clearly lower (by more than 30%). This owes mainly to DRA's resource pre-binding: the allocation is settled at scheduling time, which removes scheduling conflicts and retries. The overall change summarizes as four mappings: NodeLock to No Locks, ResourceName to ResourceClaim, Device Plugin to DRA Driver, and Annotation to Attributes. -Another easily underrated change is observability. In the traditional model, resource information comes from Nodes and usage from Pods, and a complete resource view has to be assembled by inference; in the DRA model, ResourceSlice describes the device inventory and ResourceClaim the allocations, so the resource view is itself a first-class citizen. Observability moves from "inferring" to "directly modeling" ([community talk](https://dynamia.ai/zh/blog/kcd-beijing-2026-dra-gpu-scheduling)): operators can read from a ResourceClaim who occupies which accelerator, how much memory was allocated, and how much is left, instead of reverse-engineering it from node status and Pod specs. +Another easily underrated change is observability. In the traditional model, resource information comes from Nodes and usage from Pods, and a complete resource view has to be assembled by inference; in the DRA model, ResourceSlice describes the device inventory and ResourceClaim the allocations, so the resource view is itself a first-class citizen. Observability moves from "inferring" to "directly modeling": operators can read from a ResourceClaim who occupies which accelerator, how much memory was allocated, and how much is left, instead of reverse-engineering it from node status and Pod specs. ## Two usage modes -One cost of DRA's capability upgrade is often overlooked: the writing complexity, the often-mentioned "UX regression" ([community talk](https://dynamia.ai/zh/blog/kcd-beijing-2026-dra-gpu-scheduling)). The Device Plugin style is one line: +One cost of DRA's capability upgrade is often overlooked: the writing complexity, the often-mentioned "UX regression". The Device Plugin style is one line: ```yaml resources: @@ -92,10 +133,16 @@ spec: For an enterprise already on Device Plugin, the migration cost is not just rewriting YAML; the whole team has to learn a new resource-declaration paradigm. HAMi-DRA's answer is two submission styles with the same scheduling and slicing underneath: -- **DRA native mode**: create the ResourceClaim by hand declaring memory and compute, and reference it from the Pod via `resourceClaims`. Suited to new workloads; CEL selectors express device-attribute constraints directly (topology awareness, NUMA affinity; the same mechanism extends to NICs and other resources, and `firstAvailable` expresses multiple candidate allocation plans, see the [community talk](https://aicon.infoq.cn/2026/shenzhen/presentation/7168)). -- **DevicePlugin-compatible mode**: keep the traditional `nvidia.com/gpu` / `huawei.com/Ascend310P` resource syntax; the webhook intercepts and converts it into a ResourceClaim. Existing workloads migrate with zero changes, which is where "seamless migration" comes from: the design turns DRA from an "expert interface" into an "ordinary user interface" ([community talk](https://dynamia.ai/zh/blog/kcd-beijing-2026-dra-gpu-scheduling)). +- **DRA native mode**: create the ResourceClaim by hand declaring memory and compute, and reference it from the Pod via `resourceClaims`. Suited to new workloads; CEL selectors express device-attribute constraints directly (topology awareness, NUMA affinity; the same mechanism extends to NICs and other resources, and `firstAvailable` expresses multiple candidate allocation plans). +- **DevicePlugin-compatible mode**: keep the traditional `nvidia.com/gpu` / `huawei.com/Ascend310P` resource syntax; the webhook intercepts and converts it into a ResourceClaim. Existing workloads migrate with zero changes, which is where "seamless migration" comes from: the design turns DRA from an "expert interface" into an "ordinary user interface". + +In compatible mode the conversion happens at admission: + +![Compatible mode: the webhook converts HAMi syntax at admission](/img/hami-dra-npu-sharing/webhook-conversion.png) + +The webhook turns the three HAMi resource entries into the ResourceClaim's `count`, `capacity.requests` (8192 MiB converted to bytes), and CEL selectors; the claim's lifecycle is bound to the Pod and is reclaimed when the Pod is deleted. -In compatible mode the conversion happens at admission. For a Pod submitted to a real Ascend 310P cluster, the three `huawei.com/*` entries in `resources.limits` are replaced by `resources.claims` and `spec.resourceClaims`, and in the generated ResourceClaim HAMi semantics enter DRA semantics field by field: +For a Pod submitted to a real Ascend 310P cluster, the three `huawei.com/*` entries in `resources.limits` are replaced by `resources.claims` and `spec.resourceClaims`, and in the generated ResourceClaim HAMi semantics enter DRA semantics field by field: | ResourceClaim field | Comes from | Conversion | | :-- | :-- | :-- | @@ -104,18 +151,18 @@ In compatible mode the conversion happens at admission. For a Pod submitted to a | `capacity.requests.cores: "50"` | `-core: 50` | percent passed through | | CEL selectors | `use-Ascend310P-uuid` annotation | `uuid in ["..."]`, the value taken from the ResourceSlice | -When two Pods each request 8192 MiB plus 50 cores against the same card, both claims land on the same device with independent shareIDs, and the scheduler debits the first one's consumption before allocating the second. Once capacity is fully debited, the next request is refused by the scheduler itself (the claim stays pending); when a Pod is deleted, its claim is reclaimed automatically and the accounting recovers. In-container enforcement is still HAMi-core: `libvnpu.so` intercepts over-quota memory requests and turns them into an in-container OOM, and the device as seen from the container is virtualized to the requested 8 GiB rather than the card's 21 GiB. The full walkthrough is [Lab 18](/tutorials/labs/ascend-hami-dra). +When two Pods each request 8192 MiB plus 50 cores against the same card, both claims land on the same device with independent shareIDs, and the scheduler debits the first one's consumption before allocating the second. Once capacity is fully debited, the next request is refused by the scheduler itself (the claim stays pending); when a Pod is deleted, its claim is reclaimed automatically and the accounting recovers. In-container enforcement is still HAMi-core: `libvnpu.so` intercepts over-quota memory requests and turns them into an in-container OOM, and the device as seen from the container is virtualized to the requested 8 GiB rather than the card's 21 GiB. The full walkthrough is [Lab 19](/tutorials/labs/ascend-hami-dra). ## The device support timeline, and "why now" -What HAMi DRA can do is bounded by each vendor's DRA driver progress. The timeline shared by the community ([AICon talk](https://aicon.infoq.cn/2026/shenzhen/presentation/7168)): +What HAMi DRA can do is bounded by each vendor's DRA driver progress: - 2025.09: NVIDIA DRA driver, Consumable Capacity ready - 2026.03: Hygon DCU DRA driver integrated with HAMi DRA - 2026.04: Enflame DRA driver, integration under way -- 2026.08: the **Ascend DRA driver entered development and preview** ([4pdOss/hami-dra-driver](https://github.com/4pdOss/hami-dra-driver), Paradigm's open-source repository) +- 2026.08: the **Ascend DRA driver entered development and preview** (4pdOss/hami-dra-driver) -The Ascend path entering preview means HAMi DRA's reach has just extended to NPUs: [Lab 18](/tutorials/labs/ascend-hami-dra) has already run the complete chain from HAMi request to NPU sharing on a 310P3 server, and the Ascend DRA driver is currently chart 0.1.1 (the lab used the development build fixing uuid generation; behavior may change before an official release). The HAMi 2.9 release webinar mentioned that NPU DRA support is planned to ship with 2.10, with test builds released to the community beforehand; the window for trying it and giving feedback is now. +The Ascend path entering preview means HAMi DRA's reach has just extended to NPUs: [Lab 19](/tutorials/labs/ascend-hami-dra) has already run the complete chain from HAMi request to NPU sharing on a 310P3 server, and the Ascend DRA driver is currently chart 0.1.1 (the lab used the development build fixing uuid generation; behavior may change before an official release). The HAMi 2.9 release webinar mentioned that NPU DRA support is planned to ship with 2.10, with test builds released to the community beforehand; the window for trying it and giving feedback is now. ## The practical constraints before adoption @@ -126,20 +173,21 @@ The adoption friction groups into four: - **Missing scoring**: DRA lacks scoring at the device-scheduling layer, limiting support for specialized scheduling requirements. - **Learning cost**: DRA's new concepts carry a threshold for the people deploying and maintaining clusters; the HAMi community's countermeasure is precisely the webhook-compatible mode keeping that cost out of the user's sight. -On choosing between them (guidance from the [HAMi 2.9 release webinar](https://dynamia.ai/zh/blog/hami-2.9-webinar-recap)): for standardized clusters chasing the best scheduling outcome (topology-aware scheduling, for example), plain HAMi is recommended; for highly customized clusters (already running their own scheduler) or plug-and-play device reuse, HAMi DRA is recommended. Two operational facts are also worth remembering: DRA mode and the traditional mode are mutually exclusive, do not enable both; and DRA mode still goes through HAMi-core. +On choosing between them, the community's guidance: for standardized clusters chasing the best scheduling outcome (topology-aware scheduling, for example), plain HAMi is recommended; for highly customized clusters (already running their own scheduler) or plug-and-play device reuse, HAMi DRA is recommended. Two operational facts are also worth remembering: DRA mode and the traditional mode are mutually exclusive, do not enable both; and DRA mode still goes through HAMi-core. ## Outlook On the HAMi-DRA roadmap: more heterogeneous devices (MetaX, Iluvatar CoreX), more DRA feature adoption (List Attributes, Partitionable Devices), compatibility with the Kubernetes-standard device attribute `resource.kubernetes.io/pcieRoot`, and scheduler integrations with Volcano and Kueue. -Zooming out ([community talk](https://dynamia.ai/zh/blog/kcd-beijing-2026-dra-gpu-scheduling)): Kubernetes is evolving into the control plane of AI infrastructure, and HAMi's position is the accelerator resource layer for Kubernetes, adapting heterogeneous devices downward, supporting training, inference, and agentic workloads upward, and providing scheduling, virtualization, and resource abstraction in between. HAMi-DRA is the step that aligns this resource layer with the Kubernetes-native model: HAMi's value converges onto runtime enforcement and heterogeneous ecosystem adaptation, while resource expression and scheduling move to the native model. If you are evaluating a migration from the Device Plugin model to DRA, or looking for a unified resource layer over a heterogeneous cluster, try [HAMi DRA](https://github.com/Project-HAMi/HAMi-DRA) and tell the community what you find. +Zooming out: Kubernetes is evolving into the control plane of AI infrastructure, and HAMi's position is the accelerator resource layer for Kubernetes, adapting heterogeneous devices downward, supporting training, inference, and agentic workloads upward, and providing scheduling, virtualization, and resource abstraction in between. HAMi-DRA is the step that aligns this resource layer with the Kubernetes-native model: HAMi's value converges onto runtime enforcement and heterogeneous ecosystem adaptation, while resource expression and scheduling move to the native model. If you are evaluating a migration from the Device Plugin model to DRA, or looking for a unified resource layer over a heterogeneous cluster, try HAMi DRA and tell the community what you find. ## Further reading +- HAMi DRA 0.2.3 release: [Release hami-dra-0.2.3](https://github.com/Project-HAMi/HAMi-DRA/releases/tag/hami-dra-0.2.3) - Yang Shouren at AICon Shenzhen 2026: [From HAMi to HAMi-DRA (talk, in Chinese)](https://aicon.infoq.cn/2026/shenzhen/presentation/7168) - Wang Jifei and James Deng at KCD Beijing 2026: [From Device Plugin to DRA: GPU Scheduling Paradigm Shift and HAMi-DRA in Practice (recap, in Chinese)](https://dynamia.ai/zh/blog/kcd-beijing-2026-dra-gpu-scheduling) - HAMi 2.9 release webinar recap by Li Mengxuan (in Chinese): [HAMi 2.9 Ascend soft slicing and DRA in depth](https://dynamia.ai/zh/blog/hami-2.9-webinar-recap) - Community getting-started guide (in Chinese): [HAMi meets Kubernetes DRA: a practical guide](https://dynamia.ai/zh/blog/hami-dra-quickstart) - Mesut Oezdil: [Does Kubernetes DRA Replace HAMi?](/blog/does-kubernetes-dra-replace-hami) ([CNCF blog original](https://www.cncf.io/blog/2026/08/07/does-kubernetes-dra-replace-hami/)) -- User guide: [How to use HAMi DRA](/docs/installation/how-to-use-hami-dra); hands-on: [Lab 18: Ascend NPU Sharing with HAMi DRA](/tutorials/labs/ascend-hami-dra) and [Lab 4: GPU Slicing with Dynamic Resource Allocation](/tutorials/labs/hami-dra) +- User guide: [How to use HAMi DRA](/docs/installation/how-to-use-hami-dra); hands-on: [Lab 19: Ascend NPU Sharing with HAMi DRA](/tutorials/labs/ascend-hami-dra) and [Lab 4: GPU Slicing with Dynamic Resource Allocation](/tutorials/labs/hami-dra) - Components: [Project-HAMi/HAMi-DRA](https://github.com/Project-HAMi/HAMi-DRA) · [4pdOss/hami-dra-driver](https://github.com/4pdOss/hami-dra-driver) · [Project-HAMi/hami-vnpu-core](https://github.com/Project-HAMi/hami-vnpu-core) diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-dra-npu-sharing/index.md b/i18n/zh/docusaurus-plugin-content-blog/hami-dra-npu-sharing/index.md index 69785a1ec..9fd3ed359 100644 --- a/i18n/zh/docusaurus-plugin-content-blog/hami-dra-npu-sharing/index.md +++ b/i18n/zh/docusaurus-plugin-content-blog/hami-dra-npu-sharing/index.md @@ -1,31 +1,53 @@ --- title: "从 HAMi 到 HAMi-DRA：异构算力资源管理的演进" -date: "2026-09-20" +date: "2026-09-24" description: "HAMi DRA 是什么、为什么需要它、如何设计：回顾 Device Plugin 时代 HAMi 的四大瓶颈（配额计量、抢占兼容、调度性能、死锁），解析 HAMi-DRA 的 Webhook 转换、DRA Driver 生态时间线与 HAMi-core 的保留角色，并给出选型建议与真实集群中的运行示例。" authors: [rootsongjc] tags: ["HAMi", "DRA", "Ascend", "NPU 共享", "Kubernetes"] --- -Kubernetes 1.34 将 DRA（Dynamic Resource Allocation）核心 API 推进至 GA，配合 Consumable Capacity（可消费容量）模型，加速器第一次成为 Kubernetes 原生资源模型里的“一等公民”：有属性、有容量、允许多个 Pod 共享。这对 HAMi 这类基于 Device Plugin 的共享方案提出了一个现实问题：设备的申请和调度可以交给 Kubernetes 原生完成，但容器内的资源隔离，Kubernetes 并不负责。 +2026 年 9 月 15 日，HAMi 社区发布了 [HAMi DRA](https://github.com/Project-HAMi/HAMi-DRA) 0.2.3。这个版本本身是一次常规迭代，但时间点值得注意：8 月，昇腾 DRA Driver 刚进入开发预览；再往前，HAMi 2.9 版本直播宣布 HAMi-DRA 进入生产可用，NPU 的 DRA 支持计划随 2.10 发布。对关注异构算力调度的用户来说，HAMi DRA 已经从“实验性方向”变成了“可以认真评估的选项”。 -[HAMi DRA](https://github.com/Project-HAMi/HAMi-DRA) 是 HAMi 社区交出的答案：一个迁移层，把存量 workload 里的 HAMi 风格资源请求自动转换成原生 ResourceClaim，调度与记账交还 kube-scheduler，运行时隔离继续由 HAMi-core 完成，业务侧一行代码不用改。当前 0.2.3 版本覆盖 NVIDIA GPU；随着[昇腾 DRA Driver](https://github.com/4pdOss/hami-dra-driver) 进入预览，这条链路已在真实的 310P3 集群上完成端到端验证，社区同步发布了配套的 [实验 18：用 HAMi DRA 共享昇腾 NPU](/zh/tutorials/labs/ascend-hami-dra)，提供每一步的命令与真实输出。本文系统介绍 HAMi DRA 的动机、设计、使用方式与当前边界。 +围绕它的讨论也一直没有停。自从 Kubernetes 1.34 将 DRA（Dynamic Resource Allocation）推进至 GA，“DRA 会不会让 HAMi 过时”就成了社区频道里的高频问题；[《Kubernetes DRA 会取代 HAMi 吗？》](/zh/blog/does-kubernetes-dra-replace-hami)给出的回答是：DRA 吸收的是“请求与调度”的那一半，而“容器内强制执行”的那一半，DRA 从未设计去做，这正是 HAMi 保留的位置。 + +HAMi DRA 就是社区对这个争论的工程回应：一个迁移层，把存量 workload 里的 HAMi 风格资源请求自动转换成原生 ResourceClaim，调度与记账交还 kube-scheduler，运行时隔离继续由 HAMi-core 完成，业务侧一行代码不用改。当前 0.2.3 版本覆盖 NVIDIA GPU；随着昇腾 DRA Driver 进入预览，这条链路已在真实的 310P3 集群上完成端到端验证，社区同步发布了配套的 [实验 19：用 HAMi DRA 共享昇腾 NPU](/zh/tutorials/labs/ascend-hami-dra)，提供每一步的命令与真实输出。本文系统介绍 HAMi DRA 的动机、设计、使用方式与当前边界。 ## HAMi 解决了什么，又卡在哪里 -HAMi（异构 AI 计算虚拟化中间件）的前身是 k8s-vGPU-scheduler，现在是 CNCF 孵化项目。它解决的问题很具体：Device Plugin 的 Extended Resource 只能表达整数张卡（`nvidia.com/gpu: 1`、`huawei.com/Ascend310P: 1`），而加速器共享需要的是“一块卡的一部分显存加一部分算力”。HAMi 在旧 API 之上搭起了完整体系：mutating webhook 改写 Pod、调度器扩展（extender）做共享调度、device plugin 上报与挂载、HAMi-core（`libvgpu` / `libvnpu`）在容器内强制执行配额。这套体系在生产中经受了检验（案例数据来自[社区分享](https://aicon.infoq.cn/2026/shenzhen/presentation/7168)）：顺丰通过 GPU 共享实现最高 57% 的 GPU 节省，SNOW 降低 55% 成本，招商银行借助拓扑感知调度实现硬件资源池 100% 利用。 +HAMi 解决的问题很直接：Device Plugin 的 Extended Resource 只能表达整数张卡（`nvidia.com/gpu: 1`、`huawei.com/Ascend310P: 1`），而共享要的是“一块卡的一部分显存加一部分算力”： + +```yaml +# Device Plugin 只能这样申请：整卡，无法切分 +resources: + limits: + nvidia.com/gpu: 1 + +# 共享真正需要的是这样（HAMi 语法）： +resources: + limits: + huawei.com/Ascend310P: 1 + huawei.com/Ascend310P-memory: "8192" # 8 GiB 显存 + huawei.com/Ascend310P-core: "50" # 50% 算力 +``` + +为此 HAMi 在旧 API 之上搭起了完整体系：mutating webhook 改写 Pod、调度器扩展（extender）做共享调度、device plugin 上报与挂载、HAMi-core（`libvgpu` / `libvnpu`）在容器内强制执行配额。这套体系经受了生产检验：顺丰最高节省 57% GPU，SNOW 降本 55%，招商银行借拓扑感知调度实现硬件资源池 100% 利用。 + +但这些组件都是补在旧 API 上的，补得越多，结构性问题越明显。HAMi 现有方案最常见的四个痛点： -但正因为整个体系建在 Device Plugin 和调度扩展之上，天花板也是结构性的。HAMi 现有方案的现实瓶颈可以归纳为四点（详见[社区分享](https://aicon.infoq.cn/2026/shenzhen/presentation/7168)）： +| 痛点 | 说白了 | +| :-- | :-- | +| 配额管不住 | `huawei.com/*` 这类资源不在 Kubernetes 原生配额（ResourceQuota）的统计范围内，想按团队限制用量，就得自己再造一套配额系统 | +| 抢占用不上 | 抢占（Preemption）等调度功能不认识树外调度器替它做的决定，高优先级任务来了，也没法让低优先级任务让位 | +| 越忙越慢 | 多个 Pod 同时调度到同一台节点时只能排队一个个来（节点锁），集群越忙，创建 Pod 越慢 | +| 偶尔卡死 | 一次失败的请求可能让整台节点被锁 5 分钟，期间这台节点上的新 Pod 全部干等 | -| 瓶颈 | 根源 | -| :--------------------- | :-------------------------------------------------------------- | -| ResourceQuota 计量问题 | Extended Resource 不进原生配额体系，需要额外的 Quota 实现 | -| Preemption 兼容问题 | Extender API 的限制，在支持抢占等调度功能时捉襟见肘 | -| 调度性能劣化 | 节点锁的排它性：单节点同时调度多个 Pod 时，并发退化为串行 | -| 异常死锁 | 系统默认锁定时长 5 分钟，创建过程中的请求失败可能让节点无法解锁 | +这四点都不是代码写错了，而是路本身的问题：在“按节点整卡分配”的模型上模拟精细切分，这些代价躲不开。 -这四点都不是代码层面的 bug，而是架构限制：要在以节点为中心的分配模型上做精细化分配，就得付出这些代价。 +![HAMi 在 Device Plugin 之上卡住的位置](/img/hami-dra-npu-sharing/where-hami-stuck-zh.png) + +图中左侧是 HAMi 为实现共享而自建的整条链路：从 Pod 到加速器之间的每一层都由 HAMi 自己实现。四个瓶颈的位置如图中虚线所示：第 1 个出在资源声明本身（Extended Resource 不进原生配额），第 2、3、4 个都出在 Scheduler Extender 这一环——抢占受 Extender API 限制、节点锁把并发退化为串行、失败请求可能锁住节点 5 分钟。 ## DRA 改变了什么 @@ -38,21 +60,40 @@ Kubernetes 1.34 将 DRA（Dynamic Resource Allocation）核心 API 推进至 GA 对共享调度最关键的是 **Consumable Capacity**（KEP-5075）：设备以两个独立维度（显存、算力百分比）发布容量，`allowMultipleAllocations` 允许同一设备被多个 claim 消费，kube-scheduler 逐维度对账已消费容量，每次分配带独立 shareID。这意味着“多个 workload 合理消费同一加速器的部分容量”第一次成为调度器的原生能力，不再需要树外组件模拟。注意该 feature gate 在 1.34/1.35 需手动开启，1.36 起默认开启。 +设备与分配在真实集群里的样子（节选）： + +```yaml +# ResourceSlice：每块 310P3 一个 device，发布两个维度的容量 +devices: + - name: npu-0-0 + attributes: + productName: { string: 310P3 } + uuid: { string: 68496E64-20E05477-92C31323-6E78030A-BD003019 } + capacity: + cores: { value: "100" } + memory: { value: 21525Mi } + allowMultipleAllocations: true + +# ResourceClaim：申请其中一部分 +capacity: + requests: + cores: "50" + memory: "8589934592" +``` + +![DRA 对象模型与逐维度记账](/img/hami-dra-npu-sharing/dra-model-zh.png) + +图中 Claim A 与 Claim B 各自申请同一块设备（npu-0-0）的一部分容量，kube-scheduler 对照 ResourceSlice 逐维度记账；容量记满后，Claim C 只能停在 pending，直到有 Pod 释放容量。 + 传统 Device Plugin 接口只能暴露简单容量信息，调度器“调度到节点后才发现资源不够”；DRA 让设备属性在调度阶段就可精确匹配。这里的迁移逻辑很清楚：瓶颈不在 HAMi 的调度算法，而在它脚下那层 API。 ## HAMi DRA 的设计 HAMi DRA 没有推倒重来，而是只替换了上半部分：调度和记账改用 Kubernetes 原生实现，运行时隔离继续用 HAMi-core。 -```mermaid -%% title: HAMi DRA 的组件分工 -flowchart TD - POD["用户 Pod
HAMi 风格资源请求"] --> HOOK["HAMi-DRA Webhook
创建与回收 ResourceClaim"] - HOOK --> DRA["Kubernetes DRA
ResourceClaim / DeviceClass
ResourceSlice / kube-scheduler"] - DRA --> DRV["DRA driver
kubelet plugin + CDI
设备发现与注入"] - DRV --> CORE["HAMi-core
显存配额 / 算力时间片
libvgpu / libvnpu 拦截"] - CORE --> ACC["物理加速器
多个 Pod 共享"] -``` +![HAMi DRA 请求路径：从 YAML 到加速器](/img/hami-dra-npu-sharing/hami-dra-design-zh.png) + +整个过程从一份 YAML 开始。兼容模式的用户继续写 `huawei.com/Ascend310P` 这类资源名，HAMi DRA 的 Mutating Webhook 会在准入阶段把它转换成 ResourceClaim；原生模式的用户直接提交 ResourceClaim，跳过这一步。接下来 kube-scheduler 依据 ResourceSlice 完成调度和记账，把 Pod 绑定到节点。节点上的 kubelet 随即调用 DRA Driver 的 Prepare 接口，由它生成 CDI spec；containerd 按照 spec 创建容器，HAMi-core（`libvgpu` / `libvnpu`）也在这一步被注入容器、负责执行配额。workload 最终运行在 GPU、NPU、DCU 等加速器上，图中的编号 ① 到 ⑧ 对应这条链路的先后顺序。 三个组件，各管一段：**HAMi-DRA Webhook** 负责“怎么声明”，**DRA driver** 负责“怎么分配设备”，**HAMi-core** 负责“怎么共享设备”。 @@ -62,13 +103,13 @@ flowchart TD **HAMi-core** 原样保留：DRA 只解决“请求与调度”那一半，“容器内强制执行”那一半（显存配额、算力时间片）仍由 `libvgpu` / `libvnpu` 完成。这正是 [《Kubernetes DRA 会取代 HAMi 吗？》](/zh/blog/does-kubernetes-dra-replace-hami)的结论在架构上的落点。 -调度性能的提升同样来自这次替换（[AICon](https://aicon.infoq.cn/2026/shenzhen/presentation/7168) 与 [KCD Beijing](https://dynamia.ai/zh/blog/kcd-beijing-2026-dra-gpu-scheduling) 两场分享给出的对比相互印证）：借助 DRA 自身的调度系统，HAMi-DRA 避免了节点锁带来的性能衰退，同节点并发调度多个 Pod 时，原方案的 Pod 创建时间近似线性劣化（峰值在 42 秒量级），HAMi-DRA 则明显更低（幅度在 30% 以上）。这主要得益于 DRA 的资源预绑定机制：资源分配在调度阶段就已确定，减少了调度冲突与重试。整体可以概括为四组对照：NodeLock 到 No Locks，ResourceName 到 ResourceClaim，Device Plugin 到 DRA Driver，Annotation 到 Attributes。 +调度性能的提升同样来自这次替换：借助 DRA 自身的调度系统，HAMi-DRA 避免了节点锁带来的性能衰退，同节点并发调度多个 Pod 时，原方案的 Pod 创建时间近似线性劣化（峰值在 42 秒量级），HAMi-DRA 则明显更低（幅度在 30% 以上）。这主要得益于 DRA 的资源预绑定机制：资源分配在调度阶段就已确定，减少了调度冲突与重试。整体可以概括为四组对照：NodeLock 到 No Locks，ResourceName 到 ResourceClaim，Device Plugin 到 DRA Driver，Annotation 到 Attributes。 -还有一个容易被低估的变化是可观测性。传统模型里，资源信息来自 Node、使用情况来自 Pod，完整的资源视图需要聚合推断才能得到；DRA 模型里 ResourceSlice 描述设备清单、ResourceClaim 描述分配，资源视角本身就是一等公民。可观测性从“推断”变成了“直接建模”（[社区分享](https://dynamia.ai/zh/blog/kcd-beijing-2026-dra-gpu-scheduling)）：运维可以直接从 ResourceClaim 看到每块加速器被谁占用、分配了多少显存、还剩多少余量，而不必从节点状态和 Pod 配置里反推。 +还有一个容易被低估的变化是可观测性。传统模型里，资源信息来自 Node、使用情况来自 Pod，完整的资源视图需要聚合推断才能得到；DRA 模型里 ResourceSlice 描述设备清单、ResourceClaim 描述分配，资源视角本身就是一等公民。可观测性从“推断”变成了“直接建模”：运维可以直接从 ResourceClaim 看到每块加速器被谁占用、分配了多少显存、还剩多少余量，而不必从节点状态和 Pod 配置里反推。 ## 两种使用模式 -DRA 的能力升级有一面经常被忽视的代价：写法复杂度，也就是常被提到的“UX 退化”（[社区分享](https://dynamia.ai/zh/blog/kcd-beijing-2026-dra-gpu-scheduling)）。Device Plugin 的写法是一行： +DRA 的能力升级有一面经常被忽视的代价：写法复杂度，也就是常被提到的“UX 退化”。Device Plugin 的写法是一行： ```yaml resources: @@ -92,10 +133,16 @@ spec: 对已在使用 Device Plugin 的企业，迁移成本不只是改 YAML，而是整个团队要学一套新的资源声明范式。HAMi-DRA 对此的回答是两种提交方式，底层调度与切分逻辑一致： -- **DRA 原生模式**：手动创建 ResourceClaim 声明显存与算力，Pod 通过 `resourceClaims` 引用。适合新业务，能直接使用 CEL 选择器表达设备属性约束（例如拓扑感知、NUMA 亲和；相同机制已扩展到网卡等资源，配合 `firstAvailable` 还能表达多种候选分配方案，见[社区分享](https://aicon.infoq.cn/2026/shenzhen/presentation/7168)）。 -- **DevicePlugin 兼容模式**：沿用传统 `nvidia.com/gpu` / `huawei.com/Ascend310P` 资源申请语法，Webhook 自动拦截并转换为 ResourceClaim。存量业务零改造迁移，这就是“无感迁移”的由来：这个设计把 DRA 从“专家接口”变成了“普通用户接口”（[社区分享](https://dynamia.ai/zh/blog/kcd-beijing-2026-dra-gpu-scheduling)）。 +- **DRA 原生模式**：手动创建 ResourceClaim 声明显存与算力，Pod 通过 `resourceClaims` 引用。适合新业务，能直接使用 CEL 选择器表达设备属性约束（例如拓扑感知、NUMA 亲和；相同机制已扩展到网卡等资源，配合 `firstAvailable` 还能表达多种候选分配方案）。 +- **DevicePlugin 兼容模式**：沿用传统 `nvidia.com/gpu` / `huawei.com/Ascend310P` 资源申请语法，Webhook 自动拦截并转换为 ResourceClaim。存量业务零改造迁移，这就是“无感迁移”的由来：这个设计把 DRA 从“专家接口”变成了“普通用户接口”。 + +兼容模式的转换发生在准入阶段： + +![兼容模式：Webhook 在准入阶段完成转换](/img/hami-dra-npu-sharing/webhook-conversion-zh.png) + +Webhook 在准入阶段把三个 HAMi 资源条目换成 ResourceClaim 的 `count`、`capacity.requests`（8192 MiB 换算为字节数）与 CEL 选择器；claim 的生命周期与 Pod 绑定，Pod 删除时一并回收。 -兼容模式的转换发生在准入阶段。以一台昇腾 310P 真实集群里提交的 Pod 为例，`resources.limits` 里的三个 `huawei.com/*` 条目被替换为 `resources.claims` 与 `spec.resourceClaims`，生成的 ResourceClaim 中 HAMi 语义逐项进入 DRA 语义： +以一台昇腾 310P 真实集群里提交的 Pod 为例，`resources.limits` 里的三个 `huawei.com/*` 条目被替换为 `resources.claims` 与 `spec.resourceClaims`，生成的 ResourceClaim 中 HAMi 语义逐项进入 DRA 语义： | ResourceClaim 字段 | 来源 | 转换规则 | | :-- | :-- | :-- | @@ -104,18 +151,18 @@ spec: | `capacity.requests.cores: "50"` | `-core: 50` | 百分比直传 | | CEL 选择器 | `use-Ascend310P-uuid` 注解 | `uuid in ["..."]`，值来自 ResourceSlice | -两个 Pod 各请求 8192 MiB 显存加 50 算力、指向同一块卡时，两个 claim 落到同一个 device 上，各自带独立 shareID，调度器在分配第二个之前已把第一个的消费记账。容量记满后新请求被调度器直接拒绝（claim 停在 pending），Pod 删除后 claim 自动回收、记账恢复。容器内的强制执行依旧是 HAMi-core：`libvnpu.so` 拦截超限的显存申请并转为容器内 OOM，容器视角的设备被虚拟化为申请的 8 GiB 而非整卡 21 GiB。完整过程见[实验 18](/zh/tutorials/labs/ascend-hami-dra)。 +两个 Pod 各请求 8192 MiB 显存加 50 算力、指向同一块卡时，两个 claim 落到同一个 device 上，各自带独立 shareID，调度器在分配第二个之前已把第一个的消费记账。容量记满后新请求被调度器直接拒绝（claim 停在 pending），Pod 删除后 claim 自动回收、记账恢复。容器内的强制执行依旧是 HAMi-core：`libvnpu.so` 拦截超限的显存申请并转为容器内 OOM，容器视角的设备被虚拟化为申请的 8 GiB 而非整卡 21 GiB。完整过程见[实验 19](/zh/tutorials/labs/ascend-hami-dra)。 ## 设备支持的时间线，与“为什么是现在” -HAMi DRA 的能力边界很大程度上取决于各厂商 DRA driver 的适配进度。社区分享中给出的时间线（[AICon](https://aicon.infoq.cn/2026/shenzhen/presentation/7168)）： +HAMi DRA 的能力边界很大程度上取决于各厂商 DRA driver 的适配进度： - 2025.09：NVIDIA DRA Driver，Consumable Capacity 就绪 - 2026.03：海光 DCU DRA Driver 完成与 HAMi DRA 的集成 - 2026.04：Enflame（燧原）DRA Driver 集成推进中 -- 2026.08：**Ascend DRA Driver 进入开发与预览**（[4pdOss/hami-dra-driver](https://github.com/4pdOss/hami-dra-driver)，即范式的开源仓库） +- 2026.08：**Ascend DRA Driver 进入开发与预览**（4pdOss/hami-dra-driver） -昇腾链路进入预览，意味着 HAMi DRA 的能力边界刚刚扩展到 NPU：[实验 18](/zh/tutorials/labs/ascend-hami-dra) 已在一台 310P3 服务器上把 HAMi 请求到 NPU 共享的完整链路跑通，昇腾 DRA driver 当前为 chart 0.1.1（实验所用镜像为修复 uuid 生成问题的开发构建，正式发布前行为可能变化）。HAMi 2.9 版本直播中提到 NPU 的 DRA 支持计划随 2.10 发布，在那之前会先放出测试版本供社区试用，现在正是尝鲜与反馈的窗口。 +昇腾链路进入预览，意味着 HAMi DRA 的能力边界刚刚扩展到 NPU：[实验 19](/zh/tutorials/labs/ascend-hami-dra) 已在一台 310P3 服务器上把 HAMi 请求到 NPU 共享的完整链路跑通，昇腾 DRA driver 当前为 chart 0.1.1（实验所用镜像为修复 uuid 生成问题的开发构建，正式发布前行为可能变化）。HAMi 2.9 版本直播中提到 NPU 的 DRA 支持计划随 2.10 发布，在那之前会先放出测试版本供社区试用，现在正是尝鲜与反馈的窗口。 ## 落地前的现实约束 @@ -126,20 +173,21 @@ HAMi DRA 的能力边界很大程度上取决于各厂商 DRA driver 的适配 - **Scoring 缺失**：设备调度层面 DRA 的打分能力尚缺，对特性化调度要求的支持有限。 - **学习成本**：DRA 引入的新概念对集群部署与维护同学有门槛，HAMi 社区的对策正是用 Webhook 兼容模式把这份成本挡在用户视线之外。 -选型上（[HAMi 2.9 版本直播](https://dynamia.ai/zh/blog/hami-2.9-webinar-recap)给出的建议）：标准化集群、追求最优调度效果（如拓扑感知），推荐原生 HAMi；高度定制化（已有自己的调度器）或需要即插即用设备复用的集群，推荐 HAMi DRA。此外两条运维事实值得记住：DRA 模式与传统模式不兼容，不要同时启用；使用 DRA 模式仍然会经过 HAMi-core。 +选型上，社区给出的建议：标准化集群、追求最优调度效果（如拓扑感知），推荐原生 HAMi；高度定制化（已有自己的调度器）或需要即插即用设备复用的集群，推荐 HAMi DRA。此外两条运维事实值得记住：DRA 模式与传统模式不兼容，不要同时启用；使用 DRA 模式仍然会经过 HAMi-core。 ## 展望 HAMi-DRA 的 roadmap 上排着：更多异构设备（MetaX、天数智芯 Iluvatar CoreX）、更多 DRA 特性适配（List Attributes、Partitionable Devices）、兼容 Kubernetes 标准设备属性（`resource.kubernetes.io/pcieRoot`），以及与 Volcano、Kueue 的调度器集成。 -再往大处看（[社区分享](https://dynamia.ai/zh/blog/kcd-beijing-2026-dra-gpu-scheduling)）：Kubernetes 正在演进为 AI 基础设施的控制平面，而 HAMi 的定位是面向 Kubernetes 的加速器资源层，向下适配异构设备，向上支撑训练、推理与 Agent 等 workload，中间提供调度、虚拟化与资源抽象。HAMi-DRA 正是把这层资源能力与 Kubernetes 原生模型对齐的关键一步：HAMi 的价值收敛到运行时强制执行与异构生态适配，资源表达与调度则交给原生模型。如果你正在评估从 Device Plugin 模式迁移到 DRA，或在为异构集群寻找统一的资源管理层，欢迎试用 [HAMi DRA](https://github.com/Project-HAMi/HAMi-DRA) 并向社区反馈。 +再往大处看：Kubernetes 正在演进为 AI 基础设施的控制平面，而 HAMi 的定位是面向 Kubernetes 的加速器资源层，向下适配异构设备，向上支撑训练、推理与 Agent 等 workload，中间提供调度、虚拟化与资源抽象。HAMi-DRA 正是把这层资源能力与 Kubernetes 原生模型对齐的关键一步：HAMi 的价值收敛到运行时强制执行与异构生态适配，资源表达与调度则交给原生模型。如果你正在评估从 Device Plugin 模式迁移到 DRA，或在为异构集群寻找统一的资源管理层，欢迎试用 HAMi DRA 并向社区反馈。 ## 延伸阅读 +- HAMi DRA 0.2.3 发布：[Release hami-dra-0.2.3](https://github.com/Project-HAMi/HAMi-DRA/releases/tag/hami-dra-0.2.3) - 杨守仁 AICon 深圳 2026 分享：[从 HAMi 到 HAMi-DRA：异构环境的算力资源管理实践](https://aicon.infoq.cn/2026/shenzhen/presentation/7168) - 王纪飞、James Deng KCD Beijing 2026 分享回顾：[从 Device Plugin 到 DRA：GPU 调度范式升级与 HAMi-DRA 实践](https://dynamia.ai/zh/blog/kcd-beijing-2026-dra-gpu-scheduling) - HAMi 2.9 版本直播回顾（李孟轩）：[HAMi 2.9 昇腾软切分与 DRA 实战详解](https://dynamia.ai/zh/blog/hami-2.9-webinar-recap) - 社区上手指南：[HAMi 正式接入 Kubernetes DRA：下一代 GPU 资源模型实践指南](https://dynamia.ai/zh/blog/hami-dra-quickstart) - Mesut Oezdil：[Kubernetes DRA 会取代 HAMi 吗？](/zh/blog/does-kubernetes-dra-replace-hami)（[英文原文](https://www.cncf.io/blog/2026/08/07/does-kubernetes-dra-replace-hami/)，CNCF 博客） -- 用户指南：[如何使用 HAMi DRA](/zh/docs/installation/how-to-use-hami-dra)；动手实验：[实验 18：用 HAMi DRA 共享昇腾 NPU](/zh/tutorials/labs/ascend-hami-dra)、[实验 4：用 DRA 切分 GPU](/zh/tutorials/labs/hami-dra) +- 用户指南：[如何使用 HAMi DRA](/zh/docs/installation/how-to-use-hami-dra)；动手实验：[实验 19：用 HAMi DRA 共享昇腾 NPU](/zh/tutorials/labs/ascend-hami-dra)、[实验 4：用 DRA 切分 GPU](/zh/tutorials/labs/hami-dra) - 相关组件：[Project-HAMi/HAMi-DRA](https://github.com/Project-HAMi/HAMi-DRA) · [4pdOss/hami-dra-driver](https://github.com/4pdOss/hami-dra-driver) · [Project-HAMi/hami-vnpu-core](https://github.com/Project-HAMi/hami-vnpu-core) diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md index a20446e2d..a9ea14855 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md @@ -1,7 +1,7 @@ --- -title: "实验 18：用 HAMi DRA 共享昇腾 NPU" +title: "实验 19：用 HAMi DRA 共享昇腾 NPU" description: "在昇腾 310P3 节点上安装 HAMi DRA 0.2.3 与 Ascend DRA Driver，观察 webhook 如何把 huawei.com/Ascend310P 请求转换成 ResourceClaim，并验证双 Pod NPU 共享、显存配额隔离与调度器容量记账。" -sidebar_label: "实验 18：昇腾 + HAMi DRA" +sidebar_label: "实验 19：昇腾 + HAMi DRA" lab: level: Advanced duration: 约 120 分钟 diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md index 3ef20fe27..4d45b33ca 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md @@ -32,4 +32,4 @@ import LabCardGridAuto from '@site/src/components/labs/LabCardGridAuto'; - **实验 14** 在挂载四块 T4 的 GKE 节点上安装 HAMi v2.10.0，并通过分配注解与调度器日志观察可组合的 `gpu-scheduler-policy` 策略链（`spread`、`binpack`、`mutex`、`mutex,binpack`）。 - **实验 16** 在配备 7 张 RTX PRO 6000 的服务器上安装 HAMi v2.10.0，并验证完整的 Dynamic MIG 生命周期：按 Pod 创建 MIG、混合规格、选择性回收、device plugin 重启恢复以及跨 GPU 调度。 - **实验 17** 将模型打包为 KitOps ModelKit，通过 initContainer 从 Jozu Hub 拉取，并使用 SGLang（可选 vLLM）在 HAMi GPU 共享资源上提供服务。 -- **实验 18** 在昇腾 310P3 节点上安装 HAMi DRA 0.2.3 与 Ascend DRA Driver，验证请求到 ResourceClaim 的转换、双 Pod NPU 共享、显存配额隔离与调度器容量记账。 +- **实验 19** 在昇腾 310P3 节点上安装 HAMi DRA 0.2.3 与 Ascend DRA Driver，验证请求到 ResourceClaim 的转换、双 Pod NPU 共享、显存配额隔离与调度器容量记账。 diff --git a/static/img/hami-dra-npu-sharing/dra-model-zh.png b/static/img/hami-dra-npu-sharing/dra-model-zh.png new file mode 100644 index 000000000..b4e9ecf5b --- /dev/null +++ b/static/img/hami-dra-npu-sharing/dra-model-zh.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:0908a56e4482d1beac550fefcc266b4bee5a2dee470880aad3b2a025f54864c1 +size 612805 diff --git a/static/img/hami-dra-npu-sharing/dra-model.png b/static/img/hami-dra-npu-sharing/dra-model.png new file mode 100644 index 000000000..0be395bd5 --- /dev/null +++ b/static/img/hami-dra-npu-sharing/dra-model.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:8ea6c1382f45c17129a8a445cc4a277c1870556d835a93a8cdb3c7a442dc641d +size 619687 diff --git a/static/img/hami-dra-npu-sharing/hami-dra-design-zh.png b/static/img/hami-dra-npu-sharing/hami-dra-design-zh.png new file mode 100644 index 000000000..9d88dc296 --- /dev/null +++ b/static/img/hami-dra-npu-sharing/hami-dra-design-zh.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:d91db4f30c2144a2a873f27ca3d71747cdebecff6684ca3d2732396eb3b67ab0 +size 534578 diff --git a/static/img/hami-dra-npu-sharing/hami-dra-design.png b/static/img/hami-dra-npu-sharing/hami-dra-design.png new file mode 100644 index 000000000..baf24e2e1 --- /dev/null +++ b/static/img/hami-dra-npu-sharing/hami-dra-design.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ee755bd96339a95190a2695b594a567d271c5610b215d77c51254441f3f51226 +size 518341 diff --git a/static/img/hami-dra-npu-sharing/webhook-conversion-zh.png b/static/img/hami-dra-npu-sharing/webhook-conversion-zh.png new file mode 100644 index 000000000..58ff842c2 --- /dev/null +++ b/static/img/hami-dra-npu-sharing/webhook-conversion-zh.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:13caad87e66c23146b2ad8c25619f0bcc0db84fa6b152efbb99e9f9879ec565b +size 224213 diff --git a/static/img/hami-dra-npu-sharing/webhook-conversion.png b/static/img/hami-dra-npu-sharing/webhook-conversion.png new file mode 100644 index 000000000..c2df82e42 --- /dev/null +++ b/static/img/hami-dra-npu-sharing/webhook-conversion.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c720aecd12b06f9e25caf9c8e00dbb594ff52387fc02354c50a12a6c8ff17a91 +size 226118 diff --git a/static/img/hami-dra-npu-sharing/where-hami-stuck-zh.png b/static/img/hami-dra-npu-sharing/where-hami-stuck-zh.png new file mode 100644 index 000000000..424f4f8db --- /dev/null +++ b/static/img/hami-dra-npu-sharing/where-hami-stuck-zh.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:1acfc362c3786f8039ea1a9e67499fc34bc701e2a88da8525ac3659dfe60d5ae +size 289032 diff --git a/static/img/hami-dra-npu-sharing/where-hami-stuck.png b/static/img/hami-dra-npu-sharing/where-hami-stuck.png new file mode 100644 index 000000000..203f63b7a --- /dev/null +++ b/static/img/hami-dra-npu-sharing/where-hami-stuck.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:78ba580f225f77474569a45a706009a47693ce3458b1276c874817a4e3b7e1d3 +size 283916 diff --git a/tutorials/labs/ascend-hami-dra.md b/tutorials/labs/ascend-hami-dra.md index cc437a0b1..9e33e1f18 100644 --- a/tutorials/labs/ascend-hami-dra.md +++ b/tutorials/labs/ascend-hami-dra.md @@ -1,7 +1,7 @@ --- -title: "Lab 18: Ascend NPU Sharing with HAMi DRA" +title: "Lab 19: Ascend NPU Sharing with HAMi DRA" description: "Install HAMi DRA 0.2.3 and the Ascend DRA driver on an Ascend 310P3 node, watch the webhook turn huawei.com/Ascend310P requests into ResourceClaims, and verify two-Pod NPU sharing, memory quota enforcement, and scheduler capacity accounting." -sidebar_label: "Lab 18: Ascend + HAMi DRA" +sidebar_label: "Lab 19: Ascend + HAMi DRA" lab: level: Advanced duration: about 120 minutes diff --git a/tutorials/overview.md b/tutorials/overview.md index a226716a9..9e28224ee 100644 --- a/tutorials/overview.md +++ b/tutorials/overview.md @@ -32,4 +32,4 @@ Each lab lists its own prerequisites. - **Lab 14** installs HAMi v2.10.0 on a four-T4 GKE node and observes the composable `gpu-scheduler-policy` chains (`spread`, `binpack`, `mutex`, `mutex,binpack`) through allocation annotations and scheduler logs. - **Lab 16** installs HAMi v2.10.0 on a seven-GPU RTX PRO 6000 server and verifies the complete Dynamic MIG lifecycle: per-Pod placement, mixed profiles, selective reclamation, device-plugin restart recovery, and spillover to a second GPU. - **Lab 17** packages a model as a KitOps ModelKit, pulls it from Jozu Hub through an initContainer, and serves it with SGLang and optionally vLLM on HAMi GPU shares. -- **Lab 18** installs HAMi DRA 0.2.3 and the Ascend DRA driver on an Ascend 310P3 node and verifies the request-to-ResourceClaim conversion, two-Pod NPU sharing, memory quota enforcement, and scheduler capacity accounting. +- **Lab 19** installs HAMi DRA 0.2.3 and the Ascend DRA driver on an Ascend 310P3 node and verifies the request-to-ResourceClaim conversion, two-Pod NPU sharing, memory quota enforcement, and scheduler capacity accounting. From 7c8a792b9040cc11cda5cadcbd916cac1202f640 Mon Sep 17 00:00:00 2001 From: Jimmy Song Date: Thu, 8 Oct 2026 19:56:52 +0800 Subject: [PATCH 03/13] Update documentation and images for Lab 19: Ascend NPU Sharing with HAMi DRA Signed-off-by: Jimmy Song --- blog/hami-dra-npu-sharing/index.md | 11 +++++------ .../hami-dra-npu-sharing/index.md | 13 ++++++------- .../current/labs/ascend-hami-dra.md | 16 ++++++++-------- static/img/hami-dra-npu-sharing/dra-model-zh.png | 4 ++-- static/img/hami-dra-npu-sharing/dra-model.png | 4 ++-- .../webhook-conversion-zh.png | 4 ++-- .../hami-dra-npu-sharing/webhook-conversion.png | 4 ++-- tutorials/labs/ascend-hami-dra.md | 16 ++++++++-------- 8 files changed, 35 insertions(+), 37 deletions(-) diff --git a/blog/hami-dra-npu-sharing/index.md b/blog/hami-dra-npu-sharing/index.md index 41a3e590c..623bcf153 100644 --- a/blog/hami-dra-npu-sharing/index.md +++ b/blog/hami-dra-npu-sharing/index.md @@ -6,11 +6,11 @@ authors: [rootsongjc] tags: ["HAMi", "DRA", "Ascend", "NPU Sharing", "Kubernetes"] --- -On September 15, 2026, the HAMi community released [HAMi DRA](https://github.com/Project-HAMi/HAMi-DRA) 0.2.3. The version itself is a routine iteration, but the timing is worth noting: in August the Ascend DRA driver entered development preview; before that, the HAMi 2.9 release webinar declared HAMi-DRA production-ready, with NPU DRA support planned for 2.10. For anyone tracking heterogeneous compute scheduling, HAMi DRA has moved from "experimental direction" to "an option worth evaluating seriously". +On September 15, 2026, the HAMi community released [HAMi DRA](https://github.com/Project-HAMi/HAMi-DRA) 0.2.3. The version itself is a routine iteration, but the timing is worth noting: the HAMi 2.9 release webinar declared HAMi-DRA production-ready, with NPU DRA support planned for 2.10. For anyone tracking heterogeneous compute scheduling, HAMi DRA has moved from "experimental direction" to "an option worth evaluating seriously". The discussion around it has not stopped either. Ever since Kubernetes 1.34 took DRA (Dynamic Resource Allocation) to GA, "does DRA make HAMi obsolete?" has been a frequent question in the community channels; the answer given in [Does Kubernetes DRA Replace HAMi?](/blog/does-kubernetes-dra-replace-hami) is that DRA absorbs the request-and-scheduling half, while the enforce-inside-the-container half was never something DRA set out to do, and that is exactly where HAMi stays. -HAMi DRA is the community's engineering answer to that debate: a migration layer that automatically converts the HAMi-style resource requests in existing workloads into native ResourceClaims, hands scheduling and accounting back to kube-scheduler, keeps runtime isolation with HAMi-core, and leaves the business side without a single line to change. The current 0.2.3 release covers NVIDIA GPUs; with the Ascend DRA driver entering preview, the chain has been verified end to end on a real 310P3 cluster, and the community shipped the companion [Lab 19: Ascend NPU Sharing with HAMi DRA](/tutorials/labs/ascend-hami-dra) with every command and real output. This post is a systematic introduction to HAMi DRA: its motivation, design, usage, and current boundaries. +HAMi DRA is the community's engineering answer to that debate: a migration layer that automatically converts the HAMi-style resource requests in existing workloads into native ResourceClaims, hands scheduling and accounting back to kube-scheduler, keeps runtime isolation with HAMi-core, and leaves the business side without a single line to change. The current 0.2.3 release covers NVIDIA GPUs; with the Ascend DRA driver, the chain has been verified end to end on a real 310P3 cluster, and the community shipped the companion [Lab 19: Ascend NPU Sharing with HAMi DRA](/tutorials/labs/ascend-hami-dra) with every command and real output. This post is a systematic introduction to HAMi DRA: its motivation, design, usage, and current boundaries. @@ -47,7 +47,7 @@ None of these are bugs in the code; they are the road itself: simulating fine-gr ![Where HAMi gets stuck on Device Plugin](/img/hami-dra-npu-sharing/where-hami-stuck.png) -The left side of the figure is the whole chain HAMi had to build itself to deliver sharing: every layer between the Pod and the accelerator is HAMi's own. The dashed lines show where each bottleneck lives: number 1 sits in the resource declaration itself (extended resources stay invisible to native quotas), while numbers 2, 3, and 4 all sit in the Scheduler Extender stage — preemption is limited by the extender API, the node lock serializes concurrent scheduling, and a failed request can lock a node for 5 minutes. +The left side of the figure is the whole chain HAMi had to build itself to deliver sharing: every layer between the Pod and the accelerator is HAMi's own. The dashed lines show where each bottleneck lives: number 1 sits in the resource declaration itself (extended resources stay invisible to native quotas), while numbers 2, 3, and 4 all sit in the Scheduler Extender stage: preemption is limited by the extender API, the node lock serializes concurrent scheduling, and a failed request can lock a node for 5 minutes. ## What DRA changed @@ -160,9 +160,8 @@ What HAMi DRA can do is bounded by each vendor's DRA driver progress: - 2025.09: NVIDIA DRA driver, Consumable Capacity ready - 2026.03: Hygon DCU DRA driver integrated with HAMi DRA - 2026.04: Enflame DRA driver, integration under way -- 2026.08: the **Ascend DRA driver entered development and preview** (4pdOss/hami-dra-driver) -The Ascend path entering preview means HAMi DRA's reach has just extended to NPUs: [Lab 19](/tutorials/labs/ascend-hami-dra) has already run the complete chain from HAMi request to NPU sharing on a 310P3 server, and the Ascend DRA driver is currently chart 0.1.1 (the lab used the development build fixing uuid generation; behavior may change before an official release). The HAMi 2.9 release webinar mentioned that NPU DRA support is planned to ship with 2.10, with test builds released to the community beforehand; the window for trying it and giving feedback is now. +The Ascend path extends HAMi DRA's reach to NPUs: [Lab 19](/tutorials/labs/ascend-hami-dra) has already run the complete chain from HAMi request to NPU sharing on a 310P3 server with the official [Project-HAMi/ascend-dra-driver](https://github.com/Project-HAMi/ascend-dra-driver) at chart 0.1.1 (the lab used the development build fixing uuid generation; behavior may change before an official release). The HAMi 2.9 release webinar mentioned that NPU DRA support is planned to ship with 2.10, with test builds released to the community beforehand; the window for trying it and giving feedback is now. ## The practical constraints before adoption @@ -190,4 +189,4 @@ Zooming out: Kubernetes is evolving into the control plane of AI infrastructure, - Community getting-started guide (in Chinese): [HAMi meets Kubernetes DRA: a practical guide](https://dynamia.ai/zh/blog/hami-dra-quickstart) - Mesut Oezdil: [Does Kubernetes DRA Replace HAMi?](/blog/does-kubernetes-dra-replace-hami) ([CNCF blog original](https://www.cncf.io/blog/2026/08/07/does-kubernetes-dra-replace-hami/)) - User guide: [How to use HAMi DRA](/docs/installation/how-to-use-hami-dra); hands-on: [Lab 19: Ascend NPU Sharing with HAMi DRA](/tutorials/labs/ascend-hami-dra) and [Lab 4: GPU Slicing with Dynamic Resource Allocation](/tutorials/labs/hami-dra) -- Components: [Project-HAMi/HAMi-DRA](https://github.com/Project-HAMi/HAMi-DRA) · [4pdOss/hami-dra-driver](https://github.com/4pdOss/hami-dra-driver) · [Project-HAMi/hami-vnpu-core](https://github.com/Project-HAMi/hami-vnpu-core) +- Components: [Project-HAMi/HAMi-DRA](https://github.com/Project-HAMi/HAMi-DRA) · [Project-HAMi/ascend-dra-driver](https://github.com/Project-HAMi/ascend-dra-driver) · [Project-HAMi/hami-vnpu-core](https://github.com/Project-HAMi/hami-vnpu-core) diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-dra-npu-sharing/index.md b/i18n/zh/docusaurus-plugin-content-blog/hami-dra-npu-sharing/index.md index 9fd3ed359..3abf64b66 100644 --- a/i18n/zh/docusaurus-plugin-content-blog/hami-dra-npu-sharing/index.md +++ b/i18n/zh/docusaurus-plugin-content-blog/hami-dra-npu-sharing/index.md @@ -6,11 +6,11 @@ authors: [rootsongjc] tags: ["HAMi", "DRA", "Ascend", "NPU 共享", "Kubernetes"] --- -2026 年 9 月 15 日，HAMi 社区发布了 [HAMi DRA](https://github.com/Project-HAMi/HAMi-DRA) 0.2.3。这个版本本身是一次常规迭代，但时间点值得注意：8 月，昇腾 DRA Driver 刚进入开发预览；再往前，HAMi 2.9 版本直播宣布 HAMi-DRA 进入生产可用，NPU 的 DRA 支持计划随 2.10 发布。对关注异构算力调度的用户来说，HAMi DRA 已经从“实验性方向”变成了“可以认真评估的选项”。 +2026 年 9 月 15 日，HAMi 社区发布了 [HAMi DRA](https://github.com/Project-HAMi/HAMi-DRA) 0.2.3。这个版本本身是一次常规迭代，但时间点值得注意：HAMi 2.9 版本直播宣布 HAMi-DRA 进入生产可用，NPU 的 DRA 支持计划随 2.10 发布。对关注异构算力调度的用户来说，HAMi DRA 已经从“实验性方向”变成了“可以认真评估的选项”。 围绕它的讨论也一直没有停。自从 Kubernetes 1.34 将 DRA（Dynamic Resource Allocation）推进至 GA，“DRA 会不会让 HAMi 过时”就成了社区频道里的高频问题；[《Kubernetes DRA 会取代 HAMi 吗？》](/zh/blog/does-kubernetes-dra-replace-hami)给出的回答是：DRA 吸收的是“请求与调度”的那一半，而“容器内强制执行”的那一半，DRA 从未设计去做，这正是 HAMi 保留的位置。 -HAMi DRA 就是社区对这个争论的工程回应：一个迁移层，把存量 workload 里的 HAMi 风格资源请求自动转换成原生 ResourceClaim，调度与记账交还 kube-scheduler，运行时隔离继续由 HAMi-core 完成，业务侧一行代码不用改。当前 0.2.3 版本覆盖 NVIDIA GPU；随着昇腾 DRA Driver 进入预览，这条链路已在真实的 310P3 集群上完成端到端验证，社区同步发布了配套的 [实验 19：用 HAMi DRA 共享昇腾 NPU](/zh/tutorials/labs/ascend-hami-dra)，提供每一步的命令与真实输出。本文系统介绍 HAMi DRA 的动机、设计、使用方式与当前边界。 +HAMi DRA 就是社区对这个争论的工程回应：一个迁移层，把存量 workload 里的 HAMi 风格资源请求自动转换成原生 ResourceClaim，调度与记账交还 kube-scheduler，运行时隔离继续由 HAMi-core 完成，业务侧一行代码不用改。当前 0.2.3 版本覆盖 NVIDIA GPU；在昇腾 DRA Driver 的配合下，这条链路已在真实的 310P3 集群上完成端到端验证，社区同步发布了配套的 [实验 19：用 HAMi DRA 共享昇腾 NPU](/zh/tutorials/labs/ascend-hami-dra)，提供每一步的命令与真实输出。本文系统介绍 HAMi DRA 的动机、设计、使用方式与当前边界。 @@ -36,7 +36,7 @@ resources: 但这些组件都是补在旧 API 上的，补得越多，结构性问题越明显。HAMi 现有方案最常见的四个痛点： -| 痛点 | 说白了 | +| 痛点 | 解释 | | :-- | :-- | | 配额管不住 | `huawei.com/*` 这类资源不在 Kubernetes 原生配额（ResourceQuota）的统计范围内，想按团队限制用量，就得自己再造一套配额系统 | | 抢占用不上 | 抢占（Preemption）等调度功能不认识树外调度器替它做的决定，高优先级任务来了，也没法让低优先级任务让位 | @@ -47,7 +47,7 @@ resources: ![HAMi 在 Device Plugin 之上卡住的位置](/img/hami-dra-npu-sharing/where-hami-stuck-zh.png) -图中左侧是 HAMi 为实现共享而自建的整条链路：从 Pod 到加速器之间的每一层都由 HAMi 自己实现。四个瓶颈的位置如图中虚线所示：第 1 个出在资源声明本身（Extended Resource 不进原生配额），第 2、3、4 个都出在 Scheduler Extender 这一环——抢占受 Extender API 限制、节点锁把并发退化为串行、失败请求可能锁住节点 5 分钟。 +图中左侧是 HAMi 为实现共享而自建的整条链路：从 Pod 到加速器之间的每一层都由 HAMi 自己实现。四个瓶颈的位置如图中虚线所示：第 1 个出在资源声明本身（Extended Resource 不进原生配额），第 2、3、4 个都出在 Scheduler Extender 这一环。抢占受 Extender API 限制、节点锁把并发退化为串行、失败请求可能锁住节点 5 分钟。 ## DRA 改变了什么 @@ -160,9 +160,8 @@ HAMi DRA 的能力边界很大程度上取决于各厂商 DRA driver 的适配 - 2025.09：NVIDIA DRA Driver，Consumable Capacity 就绪 - 2026.03：海光 DCU DRA Driver 完成与 HAMi DRA 的集成 - 2026.04：Enflame（燧原）DRA Driver 集成推进中 -- 2026.08：**Ascend DRA Driver 进入开发与预览**（4pdOss/hami-dra-driver） -昇腾链路进入预览，意味着 HAMi DRA 的能力边界刚刚扩展到 NPU：[实验 19](/zh/tutorials/labs/ascend-hami-dra) 已在一台 310P3 服务器上把 HAMi 请求到 NPU 共享的完整链路跑通，昇腾 DRA driver 当前为 chart 0.1.1（实验所用镜像为修复 uuid 生成问题的开发构建，正式发布前行为可能变化）。HAMi 2.9 版本直播中提到 NPU 的 DRA 支持计划随 2.10 发布，在那之前会先放出测试版本供社区试用，现在正是尝鲜与反馈的窗口。 +昇腾链路的打通，意味着 HAMi DRA 的能力边界扩展到了 NPU：[实验 19](/zh/tutorials/labs/ascend-hami-dra) 已在一台 310P3 服务器上，用官方 [Project-HAMi/ascend-dra-driver](https://github.com/Project-HAMi/ascend-dra-driver)（chart 0.1.1）把 HAMi 请求到 NPU 共享的完整链路跑通（实验所用镜像为修复 uuid 生成问题的开发构建，正式发布前行为可能变化）。HAMi 2.9 版本直播中提到 NPU 的 DRA 支持计划随 2.10 发布，在那之前会先放出测试版本供社区试用，现在正是尝鲜与反馈的窗口。 ## 落地前的现实约束 @@ -190,4 +189,4 @@ HAMi-DRA 的 roadmap 上排着：更多异构设备（MetaX、天数智芯 Iluva - 社区上手指南：[HAMi 正式接入 Kubernetes DRA：下一代 GPU 资源模型实践指南](https://dynamia.ai/zh/blog/hami-dra-quickstart) - Mesut Oezdil：[Kubernetes DRA 会取代 HAMi 吗？](/zh/blog/does-kubernetes-dra-replace-hami)（[英文原文](https://www.cncf.io/blog/2026/08/07/does-kubernetes-dra-replace-hami/)，CNCF 博客） - 用户指南：[如何使用 HAMi DRA](/zh/docs/installation/how-to-use-hami-dra)；动手实验：[实验 19：用 HAMi DRA 共享昇腾 NPU](/zh/tutorials/labs/ascend-hami-dra)、[实验 4：用 DRA 切分 GPU](/zh/tutorials/labs/hami-dra) -- 相关组件：[Project-HAMi/HAMi-DRA](https://github.com/Project-HAMi/HAMi-DRA) · [4pdOss/hami-dra-driver](https://github.com/4pdOss/hami-dra-driver) · [Project-HAMi/hami-vnpu-core](https://github.com/Project-HAMi/hami-vnpu-core) +- 相关组件：[Project-HAMi/HAMi-DRA](https://github.com/Project-HAMi/HAMi-DRA) · [Project-HAMi/ascend-dra-driver](https://github.com/Project-HAMi/ascend-dra-driver) · [Project-HAMi/hami-vnpu-core](https://github.com/Project-HAMi/hami-vnpu-core) diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md index a9ea14855..853bd1be2 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md @@ -45,7 +45,7 @@ flowchart TD 三个组件，三段职责，整个实验期间要分清： - **HAMi-DRA** 负责“请求怎么声明”。它是一组 admission webhook，不做任何 NPU 虚拟化，也不部署 kubelet plugin。 -- **Ascend DRA driver**（driver 名 `ascend.project-hami.io`，来自 [4pdOss/hami-dra-driver](https://github.com/4pdOss/hami-dra-driver)）负责“设备怎么分配”：设备发现、ResourceSlice 发布、kubelet Prepare/Unprepare、CDI 注入。 +- **Ascend DRA driver**（driver 名 `ascend.project-hami.io`，来自 [Project-HAMi/ascend-dra-driver](https://github.com/Project-HAMi/ascend-dra-driver)）负责“设备怎么分配”：设备发现、ResourceSlice 发布、kubelet Prepare/Unprepare、CDI 注入。 - **HAMivNPUCore** 负责“设备怎么共享”：`libvnpu.so` 拦截加容器内 limiter，强制显存配额（`NPU_MEM_QUOTA`）与算力时间片（`NPU_PRIORITY`）。 ## 前置条件 @@ -69,7 +69,7 @@ flowchart TD | 昇腾驱动 / 固件 | 25.5.1 / 7.8.0.6.201 | | HAMi-DRA | 0.2.3（chart 与镜像 tag 一致） | | Ascend DRA driver | chart `ascend-dra-driver-0.1.1`（app 0.1.0），镜像 `projecthami/ascend-dra-driver:uuid-fix-20260909`（开发构建，见步骤 2） | -| 共存组件 | HAMi 企业版 2.10.0-r1（hami-scheduler、hami-ascend-device-plugin、hami-resource-pool-manager）、cert-manager v1.21.1 | +| 共存组件 | HAMi v2.10.0（hami-scheduler、hami-ascend-device-plugin）、cert-manager v1.21.1 | ### 检查 feature gate @@ -132,7 +132,7 @@ ascend ascend ... ### 与存量 HAMi core 部署共存 -如果集群已在运行 HAMi core 或 HAMi 企业版（hami-scheduler、hami-webhook、hami-ascend-device-plugin），它们的 mutating webhook 同样会拦截 `huawei.com/Ascend310P*` 资源请求，对同一个 Pod 二次改写。DRA 的 workload 必须在两层上同时豁免： +如果集群已在运行 HAMi core（hami-scheduler、hami-webhook、hami-ascend-device-plugin），它们的 mutating webhook 同样会拦截 `huawei.com/Ascend310P*` 资源请求，对同一个 Pod 二次改写。DRA 的 workload 必须在两层上同时豁免： ```bash kubectl create ns dra-ascend-e2e @@ -195,13 +195,13 @@ mutatingwebhookconfiguration.admissionregistration.k8s.io/hami-dra-mutatingwebho validatingwebhookconfiguration.admissionregistration.k8s.io/hami-dra-validatingwebhookconfiguration ``` -这份列表里的 hami-scheduler 与 hami-ascend-device-plugin 属于先前已存在的 HAMi 企业版安装，不属于 HAMi-DRA。HAMi-DRA 自身只运行一个 deployment，即 webhook。它不部署任何 kubelet plugin：昇腾的节点侧 driver 来自下一步。 +这份列表里的 hami-scheduler 与 hami-ascend-device-plugin 属于先前已存在的 HAMi 安装，不属于 HAMi-DRA。HAMi-DRA 自身只运行一个 deployment，即 webhook。它不部署任何 kubelet plugin：昇腾的节点侧 driver 来自下一步。 ## 步骤 2：安装 Ascend DRA Driver ```bash -git clone --recurse-submodules https://github.com/4pdOss/hami-dra-driver.git -cd hami-dra-driver +git clone --recurse-submodules https://github.com/Project-HAMi/ascend-dra-driver.git +cd ascend-dra-driver helm upgrade --install ascend-dra-driver \ deployments/helm/ascend-dra-driver \ -n ascend-dra-driver --create-namespace @@ -711,7 +711,7 @@ claim 的删除由 HAMi-DRA validating webhook 负责。如果 Pod 删除时 web ### 与 HAMi core webhook 冲突 -- **症状**：集群中已有 HAMi core 或企业版时，DRA 的 Pod 被二次改写、携带非预期注解，或调度行为异常。 +- **症状**：集群中已有 HAMi core 时，DRA 的 Pod 被二次改写、携带非预期注解，或调度行为异常。 - **原因**：HAMi core 的 mutating webhook 同样拦截 `huawei.com/Ascend310P*` 资源请求。 - **检查**：`kubectl get mutatingwebhookconfigurations | grep hami`；对比 Pod 提交前后的 spec。 - **修复**：namespace 与 Pod 双层打 `hami.io/webhook: ignore` 标签，与本实验一致。 @@ -787,4 +787,4 @@ helm uninstall hami-dra -n hami-system - 这条链路背后的概念与设计：[如何使用 HAMi DRA](/zh/docs/installation/how-to-use-hami-dra) - NVIDIA GPU 上的同一套转换流程：[实验 4：用 DRA 切分 GPU](/zh/tutorials/labs/hami-dra) 与 [实验 11：KServe 推理与 HAMi DRA GPU 共享](/zh/tutorials/labs/kserve-hami-dra) - 同一块硬件走 device-plugin 路径：[实验 13：用 Volcano + HAMi-core 软切分昇腾 310P3 vNPU](/zh/tutorials/labs/volcano-ascend-vnpu) -- 相关组件：[Project-HAMi/HAMi-DRA](https://github.com/Project-HAMi/HAMi-DRA) · [4pdOss/hami-dra-driver](https://github.com/4pdOss/hami-dra-driver) · [Project-HAMi/hami-vnpu-core](https://github.com/Project-HAMi/hami-vnpu-core) +- 相关组件：[Project-HAMi/HAMi-DRA](https://github.com/Project-HAMi/HAMi-DRA) · [Project-HAMi/ascend-dra-driver](https://github.com/Project-HAMi/ascend-dra-driver) · [Project-HAMi/hami-vnpu-core](https://github.com/Project-HAMi/hami-vnpu-core) diff --git a/static/img/hami-dra-npu-sharing/dra-model-zh.png b/static/img/hami-dra-npu-sharing/dra-model-zh.png index b4e9ecf5b..80f941c03 100644 --- a/static/img/hami-dra-npu-sharing/dra-model-zh.png +++ b/static/img/hami-dra-npu-sharing/dra-model-zh.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0908a56e4482d1beac550fefcc266b4bee5a2dee470880aad3b2a025f54864c1 -size 612805 +oid sha256:a446850973533ccad017819e18d32293bb0928215db54e88fb4b6e8a465160cd +size 614268 diff --git a/static/img/hami-dra-npu-sharing/dra-model.png b/static/img/hami-dra-npu-sharing/dra-model.png index 0be395bd5..068b63c41 100644 --- a/static/img/hami-dra-npu-sharing/dra-model.png +++ b/static/img/hami-dra-npu-sharing/dra-model.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8ea6c1382f45c17129a8a445cc4a277c1870556d835a93a8cdb3c7a442dc641d -size 619687 +oid sha256:e26ae2609f05c1f9f92c4c42d3b479e4c02500977971f7db9983c67a2875e489 +size 621472 diff --git a/static/img/hami-dra-npu-sharing/webhook-conversion-zh.png b/static/img/hami-dra-npu-sharing/webhook-conversion-zh.png index 58ff842c2..4df5226ee 100644 --- a/static/img/hami-dra-npu-sharing/webhook-conversion-zh.png +++ b/static/img/hami-dra-npu-sharing/webhook-conversion-zh.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:13caad87e66c23146b2ad8c25619f0bcc0db84fa6b152efbb99e9f9879ec565b -size 224213 +oid sha256:6fd65312f23c5184911deec73dd5e883cfc5ec7f1fbe371b576cc26597f11caf +size 224246 diff --git a/static/img/hami-dra-npu-sharing/webhook-conversion.png b/static/img/hami-dra-npu-sharing/webhook-conversion.png index c2df82e42..a18aad276 100644 --- a/static/img/hami-dra-npu-sharing/webhook-conversion.png +++ b/static/img/hami-dra-npu-sharing/webhook-conversion.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c720aecd12b06f9e25caf9c8e00dbb594ff52387fc02354c50a12a6c8ff17a91 -size 226118 +oid sha256:d0c6b22e24b17bd7087773a9cca22a1a9c026324ff0ec1fd24cf8663c0d3ab23 +size 226285 diff --git a/tutorials/labs/ascend-hami-dra.md b/tutorials/labs/ascend-hami-dra.md index 9e33e1f18..1dbdb20ff 100644 --- a/tutorials/labs/ascend-hami-dra.md +++ b/tutorials/labs/ascend-hami-dra.md @@ -45,7 +45,7 @@ flowchart TD Three components, three responsibilities. Keep them apart throughout the lab: - **HAMi-DRA** decides _how the request is declared_. It is a set of admission webhooks, does no NPU virtualization, and deploys no kubelet plugin. -- **Ascend DRA driver** (driver name `ascend.project-hami.io`, from [4pdOss/hami-dra-driver](https://github.com/4pdOss/hami-dra-driver)) decides _how a device is allocated_: discovery, ResourceSlice publication, kubelet Prepare/Unprepare, CDI injection. +- **Ascend DRA driver** (driver name `ascend.project-hami.io`, from [Project-HAMi/ascend-dra-driver](https://github.com/Project-HAMi/ascend-dra-driver)) decides _how a device is allocated_: discovery, ResourceSlice publication, kubelet Prepare/Unprepare, CDI injection. - **HAMivNPUCore** decides _how a device is shared_: `libvnpu.so` interception plus the in-container limiter enforce the memory quota (`NPU_MEM_QUOTA`) and the compute time slice (`NPU_PRIORITY`). ## Prerequisites @@ -69,7 +69,7 @@ Three components, three responsibilities. Keep them apart throughout the lab: | Ascend driver / firmware | 25.5.1 / 7.8.0.6.201 | | HAMi-DRA | 0.2.3 (chart and image tag) | | Ascend DRA driver | chart `ascend-dra-driver-0.1.1` (app 0.1.0), image `projecthami/ascend-dra-driver:uuid-fix-20260909` (development build, see Step 2) | -| Coexisting | HAMi enterprise 2.10.0-r1 (hami-scheduler, hami-ascend-device-plugin, hami-resource-pool-manager), cert-manager v1.21.1 | +| Coexisting | HAMi v2.10.0 (hami-scheduler, hami-ascend-device-plugin), cert-manager v1.21.1 | ### Check the feature gate @@ -132,7 +132,7 @@ ascend ascend ... ### Coexistence with an existing HAMi core installation -If the cluster already runs HAMi core or HAMi enterprise (hami-scheduler, hami-webhook, hami-ascend-device-plugin), their mutating webhook also intercepts `huawei.com/Ascend310P*` resources and rewrites the same Pod a second time. DRA workloads must opt out on two levels: +If the cluster already runs HAMi core (hami-scheduler, hami-webhook, hami-ascend-device-plugin), their mutating webhook also intercepts `huawei.com/Ascend310P*` resources and rewrites the same Pod a second time. DRA workloads must opt out on two levels: ```bash kubectl create ns dra-ascend-e2e @@ -195,13 +195,13 @@ mutatingwebhookconfiguration.admissionregistration.k8s.io/hami-dra-mutatingwebho validatingwebhookconfiguration.admissionregistration.k8s.io/hami-dra-validatingwebhookconfiguration ``` -The hami-scheduler and hami-ascend-device-plugin Pods in this listing belong to the pre-existing HAMi enterprise installation, not to HAMi-DRA. HAMi-DRA itself runs exactly one deployment, the webhook. It deploys no kubelet plugin: the node-side driver for Ascend comes from the next step. +The hami-scheduler and hami-ascend-device-plugin Pods in this listing belong to the pre-existing HAMi installation, not to HAMi-DRA. HAMi-DRA itself runs exactly one deployment, the webhook. It deploys no kubelet plugin: the node-side driver for Ascend comes from the next step. ## Step 2: Install the Ascend DRA Driver ```bash -git clone --recurse-submodules https://github.com/4pdOss/hami-dra-driver.git -cd hami-dra-driver +git clone --recurse-submodules https://github.com/Project-HAMi/ascend-dra-driver.git +cd ascend-dra-driver helm upgrade --install ascend-dra-driver \ deployments/helm/ascend-dra-driver \ -n ascend-dra-driver --create-namespace @@ -711,7 +711,7 @@ Claim deletion is the HAMi-DRA validating webhook's job. If the webhook is down ### Conflicts with the HAMi core webhook -- **Symptom**: with HAMi core or enterprise already in the cluster, DRA Pods get rewritten twice, carry unexpected annotations, or schedule erratically. +- **Symptom**: with HAMi core already in the cluster, DRA Pods get rewritten twice, carry unexpected annotations, or schedule erratically. - **Cause**: the HAMi core mutating webhook also intercepts `huawei.com/Ascend310P*` resources. - **Check**: `kubectl get mutatingwebhookconfigurations | grep hami`; diff the Pod spec before and after submission. - **Fix**: the `hami.io/webhook: ignore` label on both the namespace and the Pod, as in this lab. @@ -787,4 +787,4 @@ Leave the feature gates and the `ascend` RuntimeClass in place if other work use - The concepts and design behind this path: [How to use HAMi DRA](/docs/installation/how-to-use-hami-dra) - The same conversion flow on NVIDIA GPUs: [Lab 4: GPU Slicing with Dynamic Resource Allocation](/tutorials/labs/hami-dra) and [Lab 11: KServe Inference with HAMi DRA GPU Sharing](/tutorials/labs/kserve-hami-dra) - The same hardware under the device-plugin path: [Lab 13: Soft-Slicing Ascend 310P3 vNPU with Volcano and HAMi-core](/tutorials/labs/volcano-ascend-vnpu) -- Components: [Project-HAMi/HAMi-DRA](https://github.com/Project-HAMi/HAMi-DRA) · [4pdOss/hami-dra-driver](https://github.com/4pdOss/hami-dra-driver) · [Project-HAMi/hami-vnpu-core](https://github.com/Project-HAMi/hami-vnpu-core) +- Components: [Project-HAMi/HAMi-DRA](https://github.com/Project-HAMi/HAMi-DRA) · [Project-HAMi/ascend-dra-driver](https://github.com/Project-HAMi/ascend-dra-driver) · [Project-HAMi/hami-vnpu-core](https://github.com/Project-HAMi/hami-vnpu-core) From 6eb5e00759d8fba79ecd6c67902ba1c3cee87234 Mon Sep 17 00:00:00 2001 From: Jimmy Song Date: Fri, 9 Oct 2026 10:44:39 +0800 Subject: [PATCH 04/13] Update documentation for Lab 19: clarify feature gate requirements and UUID handling for Ascend NPU sharing Signed-off-by: Jimmy Song --- blog/hami-dra-npu-sharing/index.md | 4 ++-- .../hami-dra-npu-sharing/index.md | 4 ++-- .../current/labs/ascend-hami-dra.md | 18 ++++++++++-------- tutorials/labs/ascend-hami-dra.md | 14 ++++++++------ 4 files changed, 22 insertions(+), 18 deletions(-) diff --git a/blog/hami-dra-npu-sharing/index.md b/blog/hami-dra-npu-sharing/index.md index 623bcf153..2d4313efe 100644 --- a/blog/hami-dra-npu-sharing/index.md +++ b/blog/hami-dra-npu-sharing/index.md @@ -38,7 +38,7 @@ But every one of these components is a patch on the old API, and the more patche | Pain | In plain words | | :-- | :-- | -| Quota does not bite | resources like `huawei.com/*` are invisible to native ResourceQuota; limiting usage per team means building a separate quota system | +| Quota does not bite | native ResourceQuota can count `huawei.com/*` extended-resource requests, but only as flat totals per namespace; it cannot see the per-device memory-and-core slicing HAMi does, so limiting real usage per team means building a separate quota system | | Preemption does not apply | preemption and other scheduler features do not recognize decisions made by an out-of-tree scheduler, so a high-priority task cannot bump a low-priority one out of the way | | Slower when busy | several Pods scheduled to the same node are processed one by one (the node lock); the busier the cluster, the slower Pod creation gets | | Occasional stalls | one failed request can leave a whole node locked for 5 minutes, and every new Pod on that node waits | @@ -47,7 +47,7 @@ None of these are bugs in the code; they are the road itself: simulating fine-gr ![Where HAMi gets stuck on Device Plugin](/img/hami-dra-npu-sharing/where-hami-stuck.png) -The left side of the figure is the whole chain HAMi had to build itself to deliver sharing: every layer between the Pod and the accelerator is HAMi's own. The dashed lines show where each bottleneck lives: number 1 sits in the resource declaration itself (extended resources stay invisible to native quotas), while numbers 2, 3, and 4 all sit in the Scheduler Extender stage: preemption is limited by the extender API, the node lock serializes concurrent scheduling, and a failed request can lock a node for 5 minutes. +The left side of the figure is the whole chain HAMi had to build itself to deliver sharing: every layer between the Pod and the accelerator is HAMi's own. The dashed lines show where each bottleneck lives: number 1 sits in the resource declaration itself (extended resources enter native quotas only as flat card counts, not the fine-grained slicing HAMi performs), while numbers 2, 3, and 4 all sit in the Scheduler Extender stage: preemption is limited because HAMi does not implement the optional preempt verb that the extender API defines, the node lock serializes concurrent scheduling, and a failed request can lock a node for 5 minutes. ## What DRA changed diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-dra-npu-sharing/index.md b/i18n/zh/docusaurus-plugin-content-blog/hami-dra-npu-sharing/index.md index 3abf64b66..09144742a 100644 --- a/i18n/zh/docusaurus-plugin-content-blog/hami-dra-npu-sharing/index.md +++ b/i18n/zh/docusaurus-plugin-content-blog/hami-dra-npu-sharing/index.md @@ -38,7 +38,7 @@ resources: | 痛点 | 解释 | | :-- | :-- | -| 配额管不住 | `huawei.com/*` 这类资源不在 Kubernetes 原生配额（ResourceQuota）的统计范围内，想按团队限制用量，就得自己再造一套配额系统 | +| 配额管不住 | 原生 ResourceQuota 能统计 `huawei.com/*` 这类 Extended Resource 的请求量，但只能按命名空间记总账，看不到 HAMi 在每块卡上做的内存加算力切分，想按团队限制真实用量，就得自己再造一套配额系统 | | 抢占用不上 | 抢占（Preemption）等调度功能不认识树外调度器替它做的决定，高优先级任务来了，也没法让低优先级任务让位 | | 越忙越慢 | 多个 Pod 同时调度到同一台节点时只能排队一个个来（节点锁），集群越忙，创建 Pod 越慢 | | 偶尔卡死 | 一次失败的请求可能让整台节点被锁 5 分钟，期间这台节点上的新 Pod 全部干等 | @@ -47,7 +47,7 @@ resources: ![HAMi 在 Device Plugin 之上卡住的位置](/img/hami-dra-npu-sharing/where-hami-stuck-zh.png) -图中左侧是 HAMi 为实现共享而自建的整条链路：从 Pod 到加速器之间的每一层都由 HAMi 自己实现。四个瓶颈的位置如图中虚线所示：第 1 个出在资源声明本身（Extended Resource 不进原生配额），第 2、3、4 个都出在 Scheduler Extender 这一环。抢占受 Extender API 限制、节点锁把并发退化为串行、失败请求可能锁住节点 5 分钟。 +图中左侧是 HAMi 为实现共享而自建的整条链路：从 Pod 到加速器之间的每一层都由 HAMi 自己实现。四个瓶颈的位置如图中虚线所示：第 1 个出在资源声明本身（Extended Resource 进原生配额的只有整卡计数，没有 HAMi 的精细切分），第 2、3、4 个都出在 Scheduler Extender 这一环。抢占受限是因为 HAMi 未实现 Extender API 中的可选抢占接口（preempt verb）、节点锁把并发退化为串行、失败请求可能锁住节点 5 分钟。 ## DRA 改变了什么 diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md index 853bd1be2..2936b96ad 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md @@ -51,7 +51,7 @@ flowchart TD ## 前置条件 - Kubernetes 1.34 或更高版本（DRA 核心 API 自 1.34 起 GA）。本实验在 1.35.7 上验证，单节点，control-plane 与 worker 同机。 -- kube-apiserver、kube-scheduler、kubelet 三处均开启 `DRAConsumableCapacity` feature gate。容量请求与 `allowMultipleAllocations` 记账都依赖它。 +- kube-apiserver、kube-controller-manager、kube-scheduler、kubelet 四处均开启 `DRAConsumableCapacity` feature gate（该 gate 在 1.36 之前为 alpha 且默认关闭，需显式开启）。容量请求与 `allowMultipleAllocations` 记账都依赖它。 - containerd 开启 CDI：`enable_cdi = true` 且 `cdi_spec_dirs = ["/etc/cdi", "/var/run/cdi"]`。 - ARM（aarch64）宿主机，昇腾驱动 25.5 或更高，用于软切分的每块 NPU 都开启 device-share 模式。 - `ascend` RuntimeClass 存在（Pod 需要 `runtimeClassName: ascend`）。 @@ -74,7 +74,7 @@ flowchart TD ### 检查 feature gate ```bash -ps -ef | grep -E "kube-apiserver|kube-scheduler|kubelet" | grep -o "feature-gates=.*" +ps -ef | grep -E "kube-apiserver|kube-controller-manager|kube-scheduler|kubelet" | grep -o "feature-gates=.*" ``` ```text @@ -202,8 +202,10 @@ validatingwebhookconfiguration.admissionregistration.k8s.io/hami-dra-validatingw ```bash git clone --recurse-submodules https://github.com/Project-HAMi/ascend-dra-driver.git cd ascend-dra-driver +git checkout 91d82a28 # 本实验验证时使用的 commit helm upgrade --install ascend-dra-driver \ deployments/helm/ascend-dra-driver \ + --set image.tag=uuid-fix-20260909 \ -n ascend-dra-driver --create-namespace ``` @@ -307,7 +309,7 @@ spec: 四处细节对后续步骤至关重要： - 每块 310P3 是一个 device（`npu-0-0`、`npu-1-0`），不是节点上的卡数计数。 -- `uuid` attribute 是芯片 UUID，例如 `68496E64-20E05477-92C31323-6E78030A-BD003019`，**不是**“节点名-序号”格式的字符串。选卡注解的值必须从这里抄，不能凭记忆手写。 +- `uuid` attribute 是芯片 UUID，例如 `68496E64-20E05477-92C31323-6E78030A-BD003019`，**不是**“节点名 - 序号”格式的字符串。选卡注解的值必须从这里抄，不能凭记忆手写。 - 容量沿两个维度发布：`memory`（MiB）与 `cores`（整卡百分比），各自的 `requestPolicy` 声明合法请求范围与步长。 - `allowMultipleAllocations: true` 允许同一设备被多个 ResourceClaim 分配。没有它就没有 NPU 共享。 @@ -610,10 +612,10 @@ npu-smi info # 进程列表（节选） ## 步骤 9：耗尽容量、释放与再分配 -Pod A 与 B 此时占住 `npu-0-0` 的 16384 MiB / 100 cores；该设备只剩 5141 MiB / 0 cores。提交请求 8192 MiB / 50 cores 的 Pod C，使用同一 uuid 注解： +Pod A 与 B 此时占住 `npu-0-0` 的 16384 MiB / 100 cores；该设备只剩 5141 MiB / 0 cores。将 `pod.yaml` 改名派生出 Pod C（仍为 8192 MiB / 50 cores，uuid 注解相同）： ```bash -kubectl apply -f pod-c.yaml +sed 's/ascend-share-a/ascend-share-c/' pod.yaml | kubectl apply -f - kubectl get pod ascend-share-c -n dra-ascend-e2e kubectl describe pod ascend-share-c -n dra-ascend-e2e | tail -4 kubectl get resourceclaim -n dra-ascend-e2e @@ -719,7 +721,7 @@ claim 的删除由 HAMi-DRA validating webhook 负责。如果 Pod 删除时 web ### CEL 选择器与 ResourceSlice 不匹配 - **症状**：Pod 一直 Pending，事件为 `cannot allocate all claims`；claim 停在 `pending`。 -- **原因**：uuid 注解与实际发布的 uuid 不一致（例如凭印象写成“节点名-序号”）。早期 driver 版本的 uuid 生成也有 bug，本实验因此使用 `uuid-fix-20260909` 构建。 +- **原因**：uuid 注解与实际发布的 uuid 不一致（例如凭印象写成“节点名 - 序号”）。早期 driver 版本的 uuid 生成也有 bug，本实验因此使用 `uuid-fix-20260909` 构建。 - **检查**：`kubectl get resourceslice -o yaml | grep -A2 uuid`，对照 Pod 注解。 - **修复**：uuid 永远从 ResourceSlice 抄写；`hami.io/use-nputype` 的值对 `productName`（如 `310P3`）。 @@ -727,8 +729,8 @@ claim 的删除由 HAMi-DRA validating webhook 负责。如果 Pod 删除时 web - **症状**：带 `capacity.requests` 的 ResourceClaim 被拒，或调度不记账（超额也能调度、共享行为异常）。 - **原因**：容量请求与 `allowMultipleAllocations` 记账依赖该 gate。 -- **检查**：`ps -ef | grep -E "kube-apiserver|kube-scheduler|kubelet" | grep -o "feature-gates=.*"`。 -- **修复**：三个组件统一加 `DRAConsumableCapacity=true` 并重启。 +- **检查**：`ps -ef | grep -E "kube-apiserver|kube-controller-manager|kube-scheduler|kubelet" | grep -o "feature-gates=.*"`。 +- **修复**：四个组件统一加 `DRAConsumableCapacity=true` 并重启。 ### device-share 未开启 diff --git a/tutorials/labs/ascend-hami-dra.md b/tutorials/labs/ascend-hami-dra.md index 1dbdb20ff..e1dd47192 100644 --- a/tutorials/labs/ascend-hami-dra.md +++ b/tutorials/labs/ascend-hami-dra.md @@ -51,7 +51,7 @@ Three components, three responsibilities. Keep them apart throughout the lab: ## Prerequisites - Kubernetes 1.34 or newer (DRA core APIs are GA since 1.34). This lab was verified on 1.35.7, single node, control-plane and worker on the same machine. -- The `DRAConsumableCapacity` feature gate enabled on kube-apiserver, kube-scheduler, and kubelet. Capacity requests and `allowMultipleAllocations` accounting depend on it. +- The `DRAConsumableCapacity` feature gate enabled on kube-apiserver, kube-controller-manager, kube-scheduler, and kubelet; the gate is alpha and off by default before 1.36, so set it explicitly. Capacity requests and `allowMultipleAllocations` accounting depend on it. - containerd with CDI enabled: `enable_cdi = true` and `cdi_spec_dirs = ["/etc/cdi", "/var/run/cdi"]`. - Ascend driver 25.5 or newer on an ARM (aarch64) host, with device-share mode enabled on each NPU used for soft slicing. - The `ascend` RuntimeClass present (Pods need `runtimeClassName: ascend`). @@ -74,7 +74,7 @@ Three components, three responsibilities. Keep them apart throughout the lab: ### Check the feature gate ```bash -ps -ef | grep -E "kube-apiserver|kube-scheduler|kubelet" | grep -o "feature-gates=.*" +ps -ef | grep -E "kube-apiserver|kube-controller-manager|kube-scheduler|kubelet" | grep -o "feature-gates=.*" ``` ```text @@ -202,8 +202,10 @@ The hami-scheduler and hami-ascend-device-plugin Pods in this listing belong to ```bash git clone --recurse-submodules https://github.com/Project-HAMi/ascend-dra-driver.git cd ascend-dra-driver +git checkout 91d82a28 # the commit this lab was verified against helm upgrade --install ascend-dra-driver \ deployments/helm/ascend-dra-driver \ + --set image.tag=uuid-fix-20260909 \ -n ascend-dra-driver --create-namespace ``` @@ -610,10 +612,10 @@ npu-smi info # process list, abridged ## Step 9: Exhaust Capacity, Release, and Reallocate -Pods A and B now hold 16384 MiB / 100 cores of `npu-0-0`; the device has 5141 MiB / 0 cores left. Submit Pod C requesting 8192 MiB / 50 cores with the same uuid annotation: +Pods A and B now hold 16384 MiB / 100 cores of `npu-0-0`; the device has 5141 MiB / 0 cores left. Submit Pod C as a renamed copy of `pod.yaml` (still 8192 MiB / 50 cores, same uuid annotation): ```bash -kubectl apply -f pod-c.yaml +sed 's/ascend-share-a/ascend-share-c/' pod.yaml | kubectl apply -f - kubectl get pod ascend-share-c -n dra-ascend-e2e kubectl describe pod ascend-share-c -n dra-ascend-e2e | tail -4 kubectl get resourceclaim -n dra-ascend-e2e @@ -727,8 +729,8 @@ Claim deletion is the HAMi-DRA validating webhook's job. If the webhook is down - **Symptom**: ResourceClaims with `capacity.requests` are rejected, or scheduling does not account capacity (oversubscription succeeds, sharing misbehaves). - **Cause**: capacity requests and `allowMultipleAllocations` accounting depend on the gate. -- **Check**: `ps -ef | grep -E "kube-apiserver|kube-scheduler|kubelet" | grep -o "feature-gates=.*"`. -- **Fix**: add `DRAConsumableCapacity=true` to all three components and restart them. +- **Check**: `ps -ef | grep -E "kube-apiserver|kube-controller-manager|kube-scheduler|kubelet" | grep -o "feature-gates=.*"`. +- **Fix**: add `DRAConsumableCapacity=true` to all four components and restart them. ### device-share not enabled From d41222a94434daf0a10f81c5747211de0b810534 Mon Sep 17 00:00:00 2001 From: Jimmy Song Date: Fri, 9 Oct 2026 19:53:19 +0800 Subject: [PATCH 05/13] Rename Lab 19 to Lab 20 and update references in documentation for Ascend NPU Sharing with HAMi DRA Signed-off-by: Jimmy Song --- blog/hami-dra-npu-sharing/index.md | 14 ++++++------- .../hami-dra-npu-sharing/index.md | 14 ++++++------- .../current/labs/ascend-hami-dra.md | 11 +++++----- .../current/overview.md | 2 +- tutorials/labs/ascend-hami-dra.md | 11 +++++----- .../20-ascend-hami-dra/ascend-values.yaml | 14 +++++++++++++ .../labs/examples/20-ascend-hami-dra/pod.yaml | 20 +++++++++++++++++++ tutorials/overview.md | 2 +- 8 files changed, 62 insertions(+), 26 deletions(-) create mode 100644 tutorials/labs/examples/20-ascend-hami-dra/ascend-values.yaml create mode 100644 tutorials/labs/examples/20-ascend-hami-dra/pod.yaml diff --git a/blog/hami-dra-npu-sharing/index.md b/blog/hami-dra-npu-sharing/index.md index 2d4313efe..dc2bbbf3b 100644 --- a/blog/hami-dra-npu-sharing/index.md +++ b/blog/hami-dra-npu-sharing/index.md @@ -1,6 +1,6 @@ --- title: "From HAMi to HAMi-DRA: The Evolution of Heterogeneous Compute Management" -date: "2026-09-24" +date: "2026-10-09" description: "What HAMi DRA is, why HAMi needs it, and how it is designed: the four bottlenecks of the Device Plugin era (quota accounting, preemption compatibility, scheduling performance, deadlocks), the HAMi-DRA webhook conversion, the DRA driver ecosystem timeline, the role HAMi-core keeps, and adoption guidance with real-cluster examples." authors: [rootsongjc] tags: ["HAMi", "DRA", "Ascend", "NPU Sharing", "Kubernetes"] @@ -10,7 +10,7 @@ On September 15, 2026, the HAMi community released [HAMi DRA](https://github.com The discussion around it has not stopped either. Ever since Kubernetes 1.34 took DRA (Dynamic Resource Allocation) to GA, "does DRA make HAMi obsolete?" has been a frequent question in the community channels; the answer given in [Does Kubernetes DRA Replace HAMi?](/blog/does-kubernetes-dra-replace-hami) is that DRA absorbs the request-and-scheduling half, while the enforce-inside-the-container half was never something DRA set out to do, and that is exactly where HAMi stays. -HAMi DRA is the community's engineering answer to that debate: a migration layer that automatically converts the HAMi-style resource requests in existing workloads into native ResourceClaims, hands scheduling and accounting back to kube-scheduler, keeps runtime isolation with HAMi-core, and leaves the business side without a single line to change. The current 0.2.3 release covers NVIDIA GPUs; with the Ascend DRA driver, the chain has been verified end to end on a real 310P3 cluster, and the community shipped the companion [Lab 19: Ascend NPU Sharing with HAMi DRA](/tutorials/labs/ascend-hami-dra) with every command and real output. This post is a systematic introduction to HAMi DRA: its motivation, design, usage, and current boundaries. +HAMi DRA is the community's engineering answer to that debate: a migration layer that automatically converts the HAMi-style resource requests in existing workloads into native ResourceClaims, hands scheduling and accounting back to kube-scheduler, keeps runtime isolation with HAMi-core, and leaves the business side without a single line to change. The current 0.2.3 release covers NVIDIA GPUs; with the Ascend DRA driver, the chain has been verified end to end on a real 310P3 cluster, and the community shipped the companion [Lab 20: Ascend NPU Sharing with HAMi DRA](/tutorials/labs/ascend-hami-dra) with commands and real captured output. This post is a systematic introduction to HAMi DRA: its motivation, design, usage, and current boundaries. @@ -103,7 +103,7 @@ The **DRA driver** is the node-side implementation: it discovers devices and pub **HAMi-core** stays as it is: DRA only addresses the request-and-scheduling half; the enforce-inside-the-container half (memory quotas, compute time slices) remains with `libvgpu` / `libvnpu`. That is exactly where the conclusion of [Does Kubernetes DRA Replace HAMi?](/blog/does-kubernetes-dra-replace-hami) lands architecturally. -The scheduling-performance gain comes from the same replacement: by riding DRA's own scheduling, HAMi-DRA avoids the node-lock performance decay; when several Pods land on the same node concurrently, the original approach's Pod creation time degrades nearly linearly (peaking in the 42-second range), while HAMi-DRA stays clearly lower (by more than 30%). This owes mainly to DRA's resource pre-binding: the allocation is settled at scheduling time, which removes scheduling conflicts and retries. The overall change summarizes as four mappings: NodeLock to No Locks, ResourceName to ResourceClaim, Device Plugin to DRA Driver, and Annotation to Attributes. +The scheduling-performance gain comes from the same replacement: by riding DRA's own scheduling, HAMi-DRA avoids the node-lock performance decay; when several Pods land on the same node concurrently, the original approach's Pod creation time degrades roughly linearly as concurrency grows, while HAMi-DRA stays clearly lower in internal community testing. This owes mainly to DRA's resource pre-binding: the allocation is settled at scheduling time, which removes scheduling conflicts and retries. The overall change summarizes as four mappings: NodeLock to No Locks, ResourceName to ResourceClaim, Device Plugin to DRA Driver, and Annotation to Attributes. Another easily underrated change is observability. In the traditional model, resource information comes from Nodes and usage from Pods, and a complete resource view has to be assembled by inference; in the DRA model, ResourceSlice describes the device inventory and ResourceClaim the allocations, so the resource view is itself a first-class citizen. Observability moves from "inferring" to "directly modeling": operators can read from a ResourceClaim who occupies which accelerator, how much memory was allocated, and how much is left, instead of reverse-engineering it from node status and Pod specs. @@ -151,7 +151,7 @@ For a Pod submitted to a real Ascend 310P cluster, the three `huawei.com/*` entr | `capacity.requests.cores: "50"` | `-core: 50` | percent passed through | | CEL selectors | `use-Ascend310P-uuid` annotation | `uuid in ["..."]`, the value taken from the ResourceSlice | -When two Pods each request 8192 MiB plus 50 cores against the same card, both claims land on the same device with independent shareIDs, and the scheduler debits the first one's consumption before allocating the second. Once capacity is fully debited, the next request is refused by the scheduler itself (the claim stays pending); when a Pod is deleted, its claim is reclaimed automatically and the accounting recovers. In-container enforcement is still HAMi-core: `libvnpu.so` intercepts over-quota memory requests and turns them into an in-container OOM, and the device as seen from the container is virtualized to the requested 8 GiB rather than the card's 21 GiB. The full walkthrough is [Lab 19](/tutorials/labs/ascend-hami-dra). +When two Pods each request 8192 MiB plus 50 cores against the same card, both claims land on the same device with independent shareIDs, and the scheduler debits the first one's consumption before allocating the second. Once capacity is fully debited, the next request is refused by the scheduler itself (the claim stays pending); when a Pod is deleted, its claim is reclaimed automatically and the accounting recovers. In-container enforcement is still HAMi-core: `libvnpu.so` intercepts over-quota memory requests and turns them into an in-container OOM, and the device as seen from the container is virtualized to the requested 8 GiB rather than the card's 21 GiB. The full walkthrough is [Lab 20](/tutorials/labs/ascend-hami-dra). ## The device support timeline, and "why now" @@ -161,7 +161,7 @@ What HAMi DRA can do is bounded by each vendor's DRA driver progress: - 2026.03: Hygon DCU DRA driver integrated with HAMi DRA - 2026.04: Enflame DRA driver, integration under way -The Ascend path extends HAMi DRA's reach to NPUs: [Lab 19](/tutorials/labs/ascend-hami-dra) has already run the complete chain from HAMi request to NPU sharing on a 310P3 server with the official [Project-HAMi/ascend-dra-driver](https://github.com/Project-HAMi/ascend-dra-driver) at chart 0.1.1 (the lab used the development build fixing uuid generation; behavior may change before an official release). The HAMi 2.9 release webinar mentioned that NPU DRA support is planned to ship with 2.10, with test builds released to the community beforehand; the window for trying it and giving feedback is now. +The Ascend path extends HAMi DRA's reach to NPUs: [Lab 20](/tutorials/labs/ascend-hami-dra) has already run the complete chain from HAMi request to NPU sharing on a 310P3 server with the official [Project-HAMi/ascend-dra-driver](https://github.com/Project-HAMi/ascend-dra-driver) at chart 0.1.1 (the lab used the development build fixing uuid generation; behavior may change before an official release). HAMi 2.10.0 was released in August 2026 and its release notes now point the community to the Ascend DRA driver repository; the driver itself is still pre-release, so the window for trying it and giving feedback remains open. ## The practical constraints before adoption @@ -172,7 +172,7 @@ The adoption friction groups into four: - **Missing scoring**: DRA lacks scoring at the device-scheduling layer, limiting support for specialized scheduling requirements. - **Learning cost**: DRA's new concepts carry a threshold for the people deploying and maintaining clusters; the HAMi community's countermeasure is precisely the webhook-compatible mode keeping that cost out of the user's sight. -On choosing between them, the community's guidance: for standardized clusters chasing the best scheduling outcome (topology-aware scheduling, for example), plain HAMi is recommended; for highly customized clusters (already running their own scheduler) or plug-and-play device reuse, HAMi DRA is recommended. Two operational facts are also worth remembering: DRA mode and the traditional mode are mutually exclusive, do not enable both; and DRA mode still goes through HAMi-core. +On choosing between them, the community's guidance: for standardized clusters chasing the best scheduling outcome (topology-aware scheduling, for example), plain HAMi is recommended; for highly customized clusters (already running their own scheduler) or plug-and-play device reuse, HAMi DRA is recommended. Two operational facts are also worth remembering: DRA mode and the traditional mode are two different paths for the same request, and one request must not go through both; when HAMi core is still installed, the webhook exemptions used in Lab 20 let the two coexist in one cluster. And DRA mode still goes through HAMi-core. ## Outlook @@ -188,5 +188,5 @@ Zooming out: Kubernetes is evolving into the control plane of AI infrastructure, - HAMi 2.9 release webinar recap by Li Mengxuan (in Chinese): [HAMi 2.9 Ascend soft slicing and DRA in depth](https://dynamia.ai/zh/blog/hami-2.9-webinar-recap) - Community getting-started guide (in Chinese): [HAMi meets Kubernetes DRA: a practical guide](https://dynamia.ai/zh/blog/hami-dra-quickstart) - Mesut Oezdil: [Does Kubernetes DRA Replace HAMi?](/blog/does-kubernetes-dra-replace-hami) ([CNCF blog original](https://www.cncf.io/blog/2026/08/07/does-kubernetes-dra-replace-hami/)) -- User guide: [How to use HAMi DRA](/docs/installation/how-to-use-hami-dra); hands-on: [Lab 19: Ascend NPU Sharing with HAMi DRA](/tutorials/labs/ascend-hami-dra) and [Lab 4: GPU Slicing with Dynamic Resource Allocation](/tutorials/labs/hami-dra) +- User guide: [How to use HAMi DRA](/docs/installation/how-to-use-hami-dra); hands-on: [Lab 20: Ascend NPU Sharing with HAMi DRA](/tutorials/labs/ascend-hami-dra) and [Lab 4: GPU Slicing with Dynamic Resource Allocation](/tutorials/labs/hami-dra) - Components: [Project-HAMi/HAMi-DRA](https://github.com/Project-HAMi/HAMi-DRA) · [Project-HAMi/ascend-dra-driver](https://github.com/Project-HAMi/ascend-dra-driver) · [Project-HAMi/hami-vnpu-core](https://github.com/Project-HAMi/hami-vnpu-core) diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-dra-npu-sharing/index.md b/i18n/zh/docusaurus-plugin-content-blog/hami-dra-npu-sharing/index.md index 09144742a..aaa5911f3 100644 --- a/i18n/zh/docusaurus-plugin-content-blog/hami-dra-npu-sharing/index.md +++ b/i18n/zh/docusaurus-plugin-content-blog/hami-dra-npu-sharing/index.md @@ -1,6 +1,6 @@ --- title: "从 HAMi 到 HAMi-DRA：异构算力资源管理的演进" -date: "2026-09-24" +date: "2026-10-09" description: "HAMi DRA 是什么、为什么需要它、如何设计：回顾 Device Plugin 时代 HAMi 的四大瓶颈（配额计量、抢占兼容、调度性能、死锁），解析 HAMi-DRA 的 Webhook 转换、DRA Driver 生态时间线与 HAMi-core 的保留角色，并给出选型建议与真实集群中的运行示例。" authors: [rootsongjc] tags: ["HAMi", "DRA", "Ascend", "NPU 共享", "Kubernetes"] @@ -10,7 +10,7 @@ tags: ["HAMi", "DRA", "Ascend", "NPU 共享", "Kubernetes"] 围绕它的讨论也一直没有停。自从 Kubernetes 1.34 将 DRA（Dynamic Resource Allocation）推进至 GA，“DRA 会不会让 HAMi 过时”就成了社区频道里的高频问题；[《Kubernetes DRA 会取代 HAMi 吗？》](/zh/blog/does-kubernetes-dra-replace-hami)给出的回答是：DRA 吸收的是“请求与调度”的那一半，而“容器内强制执行”的那一半，DRA 从未设计去做，这正是 HAMi 保留的位置。 -HAMi DRA 就是社区对这个争论的工程回应：一个迁移层，把存量 workload 里的 HAMi 风格资源请求自动转换成原生 ResourceClaim，调度与记账交还 kube-scheduler，运行时隔离继续由 HAMi-core 完成，业务侧一行代码不用改。当前 0.2.3 版本覆盖 NVIDIA GPU；在昇腾 DRA Driver 的配合下，这条链路已在真实的 310P3 集群上完成端到端验证，社区同步发布了配套的 [实验 19：用 HAMi DRA 共享昇腾 NPU](/zh/tutorials/labs/ascend-hami-dra)，提供每一步的命令与真实输出。本文系统介绍 HAMi DRA 的动机、设计、使用方式与当前边界。 +HAMi DRA 就是社区对这个争论的工程回应：一个迁移层，把存量 workload 里的 HAMi 风格资源请求自动转换成原生 ResourceClaim，调度与记账交还 kube-scheduler，运行时隔离继续由 HAMi-core 完成，业务侧一行代码不用改。当前 0.2.3 版本覆盖 NVIDIA GPU；在昇腾 DRA Driver 的配合下，这条链路已在真实的 310P3 集群上完成端到端验证，社区同步发布了配套的 [实验 20：用 HAMi DRA 共享昇腾 NPU](/zh/tutorials/labs/ascend-hami-dra)，提供命令与真实采集输出。本文系统介绍 HAMi DRA 的动机、设计、使用方式与当前边界。 @@ -103,7 +103,7 @@ HAMi DRA 没有推倒重来，而是只替换了上半部分：调度和记账 **HAMi-core** 原样保留：DRA 只解决“请求与调度”那一半，“容器内强制执行”那一半（显存配额、算力时间片）仍由 `libvgpu` / `libvnpu` 完成。这正是 [《Kubernetes DRA 会取代 HAMi 吗？》](/zh/blog/does-kubernetes-dra-replace-hami)的结论在架构上的落点。 -调度性能的提升同样来自这次替换：借助 DRA 自身的调度系统，HAMi-DRA 避免了节点锁带来的性能衰退，同节点并发调度多个 Pod 时，原方案的 Pod 创建时间近似线性劣化（峰值在 42 秒量级），HAMi-DRA 则明显更低（幅度在 30% 以上）。这主要得益于 DRA 的资源预绑定机制：资源分配在调度阶段就已确定，减少了调度冲突与重试。整体可以概括为四组对照：NodeLock 到 No Locks，ResourceName 到 ResourceClaim，Device Plugin 到 DRA Driver，Annotation 到 Attributes。 +调度性能的提升同样来自这次替换：借助 DRA 自身的调度系统，HAMi-DRA 避免了节点锁带来的性能衰退，同节点并发调度多个 Pod 时，原方案的 Pod 创建时间随并发近似线性劣化，社区内部测试中 HAMi-DRA 明显更低。这主要得益于 DRA 的资源预绑定机制：资源分配在调度阶段就已确定，减少了调度冲突与重试。整体可以概括为四组对照：NodeLock 到 No Locks，ResourceName 到 ResourceClaim，Device Plugin 到 DRA Driver，Annotation 到 Attributes。 还有一个容易被低估的变化是可观测性。传统模型里，资源信息来自 Node、使用情况来自 Pod，完整的资源视图需要聚合推断才能得到；DRA 模型里 ResourceSlice 描述设备清单、ResourceClaim 描述分配，资源视角本身就是一等公民。可观测性从“推断”变成了“直接建模”：运维可以直接从 ResourceClaim 看到每块加速器被谁占用、分配了多少显存、还剩多少余量，而不必从节点状态和 Pod 配置里反推。 @@ -151,7 +151,7 @@ Webhook 在准入阶段把三个 HAMi 资源条目换成 ResourceClaim 的 `coun | `capacity.requests.cores: "50"` | `-core: 50` | 百分比直传 | | CEL 选择器 | `use-Ascend310P-uuid` 注解 | `uuid in ["..."]`，值来自 ResourceSlice | -两个 Pod 各请求 8192 MiB 显存加 50 算力、指向同一块卡时，两个 claim 落到同一个 device 上，各自带独立 shareID，调度器在分配第二个之前已把第一个的消费记账。容量记满后新请求被调度器直接拒绝（claim 停在 pending），Pod 删除后 claim 自动回收、记账恢复。容器内的强制执行依旧是 HAMi-core：`libvnpu.so` 拦截超限的显存申请并转为容器内 OOM，容器视角的设备被虚拟化为申请的 8 GiB 而非整卡 21 GiB。完整过程见[实验 19](/zh/tutorials/labs/ascend-hami-dra)。 +两个 Pod 各请求 8192 MiB 显存加 50 算力、指向同一块卡时，两个 claim 落到同一个 device 上，各自带独立 shareID，调度器在分配第二个之前已把第一个的消费记账。容量记满后新请求被调度器直接拒绝（claim 停在 pending），Pod 删除后 claim 自动回收、记账恢复。容器内的强制执行依旧是 HAMi-core：`libvnpu.so` 拦截超限的显存申请并转为容器内 OOM，容器视角的设备被虚拟化为申请的 8 GiB 而非整卡 21 GiB。完整过程见[实验 20](/zh/tutorials/labs/ascend-hami-dra)。 ## 设备支持的时间线，与“为什么是现在” @@ -161,7 +161,7 @@ HAMi DRA 的能力边界很大程度上取决于各厂商 DRA driver 的适配 - 2026.03：海光 DCU DRA Driver 完成与 HAMi DRA 的集成 - 2026.04：Enflame（燧原）DRA Driver 集成推进中 -昇腾链路的打通，意味着 HAMi DRA 的能力边界扩展到了 NPU：[实验 19](/zh/tutorials/labs/ascend-hami-dra) 已在一台 310P3 服务器上，用官方 [Project-HAMi/ascend-dra-driver](https://github.com/Project-HAMi/ascend-dra-driver)（chart 0.1.1）把 HAMi 请求到 NPU 共享的完整链路跑通（实验所用镜像为修复 uuid 生成问题的开发构建，正式发布前行为可能变化）。HAMi 2.9 版本直播中提到 NPU 的 DRA 支持计划随 2.10 发布，在那之前会先放出测试版本供社区试用，现在正是尝鲜与反馈的窗口。 +昇腾链路的打通，意味着 HAMi DRA 的能力边界扩展到了 NPU：[实验 20](/zh/tutorials/labs/ascend-hami-dra) 已在一台 310P3 服务器上，用官方 [Project-HAMi/ascend-dra-driver](https://github.com/Project-HAMi/ascend-dra-driver)（chart 0.1.1）把 HAMi 请求到 NPU 共享的完整链路跑通（实验所用镜像为修复 uuid 生成问题的开发构建，正式发布前行为可能变化）。HAMi 2.10.0 已于 2026 年 8 月发布，release notes 中已把昇腾 DRA 指向该 driver 仓库；driver 本身仍是预发布状态，现在仍是尝鲜与反馈的窗口。 ## 落地前的现实约束 @@ -172,7 +172,7 @@ HAMi DRA 的能力边界很大程度上取决于各厂商 DRA driver 的适配 - **Scoring 缺失**：设备调度层面 DRA 的打分能力尚缺，对特性化调度要求的支持有限。 - **学习成本**：DRA 引入的新概念对集群部署与维护同学有门槛，HAMi 社区的对策正是用 Webhook 兼容模式把这份成本挡在用户视线之外。 -选型上，社区给出的建议：标准化集群、追求最优调度效果（如拓扑感知），推荐原生 HAMi；高度定制化（已有自己的调度器）或需要即插即用设备复用的集群，推荐 HAMi DRA。此外两条运维事实值得记住：DRA 模式与传统模式不兼容，不要同时启用；使用 DRA 模式仍然会经过 HAMi-core。 +选型上，社区给出的建议：标准化集群、追求最优调度效果（如拓扑感知），推荐原生 HAMi；高度定制化（已有自己的调度器）或需要即插即用设备复用的集群，推荐 HAMi DRA。此外两条运维事实值得记住：DRA 模式与传统模式是同一请求的两条不同路径，一个请求不能同时走两条路；集群里仍装有 HAMi core 时，用实验 20 中的 webhook 豁免可以让两者共存于同一集群。使用 DRA 模式仍然会经过 HAMi-core。 ## 展望 @@ -188,5 +188,5 @@ HAMi-DRA 的 roadmap 上排着：更多异构设备（MetaX、天数智芯 Iluva - HAMi 2.9 版本直播回顾（李孟轩）：[HAMi 2.9 昇腾软切分与 DRA 实战详解](https://dynamia.ai/zh/blog/hami-2.9-webinar-recap) - 社区上手指南：[HAMi 正式接入 Kubernetes DRA：下一代 GPU 资源模型实践指南](https://dynamia.ai/zh/blog/hami-dra-quickstart) - Mesut Oezdil：[Kubernetes DRA 会取代 HAMi 吗？](/zh/blog/does-kubernetes-dra-replace-hami)（[英文原文](https://www.cncf.io/blog/2026/08/07/does-kubernetes-dra-replace-hami/)，CNCF 博客） -- 用户指南：[如何使用 HAMi DRA](/zh/docs/installation/how-to-use-hami-dra)；动手实验：[实验 19：用 HAMi DRA 共享昇腾 NPU](/zh/tutorials/labs/ascend-hami-dra)、[实验 4：用 DRA 切分 GPU](/zh/tutorials/labs/hami-dra) +- 用户指南：[如何使用 HAMi DRA](/zh/docs/installation/how-to-use-hami-dra)；动手实验：[实验 20：用 HAMi DRA 共享昇腾 NPU](/zh/tutorials/labs/ascend-hami-dra)、[实验 4：用 DRA 切分 GPU](/zh/tutorials/labs/hami-dra) - 相关组件：[Project-HAMi/HAMi-DRA](https://github.com/Project-HAMi/HAMi-DRA) · [Project-HAMi/ascend-dra-driver](https://github.com/Project-HAMi/ascend-dra-driver) · [Project-HAMi/hami-vnpu-core](https://github.com/Project-HAMi/hami-vnpu-core) diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md index 2936b96ad..a0d74f6c9 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md @@ -1,7 +1,7 @@ --- -title: "实验 19：用 HAMi DRA 共享昇腾 NPU" +title: "实验 20：用 HAMi DRA 共享昇腾 NPU" description: "在昇腾 310P3 节点上安装 HAMi DRA 0.2.3 与 Ascend DRA Driver，观察 webhook 如何把 huawei.com/Ascend310P 请求转换成 ResourceClaim，并验证双 Pod NPU 共享、显存配额隔离与调度器容量记账。" -sidebar_label: "实验 19：昇腾 + HAMi DRA" +sidebar_label: "实验 20：昇腾 + HAMi DRA" lab: level: Advanced duration: 约 120 分钟 @@ -9,7 +9,7 @@ lab: cost: 需要专属的昇腾 310P3 aarch64 硬件；HAMivNPUCore 软切分仅支持 ARM authors: - rootsongjc - verified: "2026-09-17" + verified: "2026-10-09" tags: - DRA - hami @@ -56,6 +56,7 @@ flowchart TD - ARM（aarch64）宿主机，昇腾驱动 25.5 或更高，用于软切分的每块 NPU 都开启 device-share 模式。 - `ascend` RuntimeClass 存在（Pod 需要 `runtimeClassName: ascend`）。 - cert-manager（HAMi-DRA 用它签发 webhook 证书）。 +- 清单文件来自 [`tutorials/labs/examples/20-ascend-hami-dra/`](https://github.com/Project-HAMi/website/tree/master/tutorials/labs/examples/20-ascend-hami-dra)（`ascend-values.yaml` 与 `pod.yaml`），下文步骤中均已内联，无需克隆仓库。 ### 验证环境 @@ -205,7 +206,7 @@ cd ascend-dra-driver git checkout 91d82a28 # 本实验验证时使用的 commit helm upgrade --install ascend-dra-driver \ deployments/helm/ascend-dra-driver \ - --set image.tag=uuid-fix-20260909 \ + --set image.tag=latest \ -n ascend-dra-driver --create-namespace ``` @@ -213,7 +214,7 @@ chart 默认开启 HAMivNPUCore 模式。不要设置 `kubeletPlugin.fullCardAnd :::note 关于本实验使用的镜像版本 -验证运行使用 chart `ascend-dra-driver-0.1.1`，镜像覆盖为开发构建 `projecthami/ascend-dra-driver:uuid-fix-20260909`，该构建修复了早期版本 ResourceSlice uuid 生成的 bug。正式发布前行为可能变化；请固定你实际测试过的版本。 +验证运行使用 chart `ascend-dra-driver-0.1.1`，镜像为开发构建 `projecthami/ascend-dra-driver:uuid-fix-20260909`，该构建修复了早期版本 ResourceSlice uuid 生成的 bug。该 tag 未发布到任何公开 registry。uuid 修复已合入 driver 的 master 分支，Docker Hub 上的 `latest` 镜像于 2026-09-14（合入之后）推送，因此上面的安装命令改用 `image.tag=latest`。不要省略该覆盖：chart 默认 tag 会解析为未发布的 `0.1.0`。`latest` 跟随 master 移动；请固定你实际测试过的 tag 或 digest。 ::: diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md index c9fe613f9..e28d7c6f9 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md @@ -33,4 +33,4 @@ import LabCardGridAuto from '@site/src/components/labs/LabCardGridAuto'; - **实验 16** 在配备 7 张 RTX PRO 6000 的服务器上安装 HAMi v2.10.0，并验证完整的 Dynamic MIG 生命周期：按 Pod 创建 MIG、混合规格、选择性回收、device plugin 重启恢复以及跨 GPU 调度。 - **实验 17** 将模型打包为 KitOps ModelKit，通过 initContainer 从 Jozu Hub 拉取，并使用 SGLang（可选 vLLM）在 HAMi GPU 共享资源上提供服务。 - **实验 18** 在两张共享 GPU 上同时运行吵闹邻居与安静租户负载，验证 HAMi 隔离能覆盖与不能覆盖的边界，并使用 Linkerd 让敏感流量绕开竞争。 -- **实验 19** 在昇腾 310P3 节点上安装 HAMi DRA 0.2.3 与 Ascend DRA Driver，验证请求到 ResourceClaim 的转换、双 Pod NPU 共享、显存配额隔离与调度器容量记账。 +- **实验 20** 在昇腾 310P3 节点上安装 HAMi DRA 0.2.3 与 Ascend DRA Driver，验证请求到 ResourceClaim 的转换、双 Pod NPU 共享、显存配额隔离与调度器容量记账。 diff --git a/tutorials/labs/ascend-hami-dra.md b/tutorials/labs/ascend-hami-dra.md index e1dd47192..a429b61f9 100644 --- a/tutorials/labs/ascend-hami-dra.md +++ b/tutorials/labs/ascend-hami-dra.md @@ -1,7 +1,7 @@ --- -title: "Lab 19: Ascend NPU Sharing with HAMi DRA" +title: "Lab 20: Ascend NPU Sharing with HAMi DRA" description: "Install HAMi DRA 0.2.3 and the Ascend DRA driver on an Ascend 310P3 node, watch the webhook turn huawei.com/Ascend310P requests into ResourceClaims, and verify two-Pod NPU sharing, memory quota enforcement, and scheduler capacity accounting." -sidebar_label: "Lab 19: Ascend + HAMi DRA" +sidebar_label: "Lab 20: Ascend + HAMi DRA" lab: level: Advanced duration: about 120 minutes @@ -9,7 +9,7 @@ lab: cost: requires dedicated Ascend 310P3 aarch64 hardware; HAMivNPUCore soft slicing is ARM-only authors: - rootsongjc - verified: "2026-09-17" + verified: "2026-10-09" tags: - dra - hami @@ -56,6 +56,7 @@ Three components, three responsibilities. Keep them apart throughout the lab: - Ascend driver 25.5 or newer on an ARM (aarch64) host, with device-share mode enabled on each NPU used for soft slicing. - The `ascend` RuntimeClass present (Pods need `runtimeClassName: ascend`). - cert-manager (HAMi-DRA uses it to issue webhook certificates). +- Manifests from [`tutorials/labs/examples/20-ascend-hami-dra/`](https://github.com/Project-HAMi/website/tree/master/tutorials/labs/examples/20-ascend-hami-dra) (`ascend-values.yaml` and `pod.yaml`); both are also inlined in the steps below, so nothing has to be cloned. ### Verification environment @@ -205,7 +206,7 @@ cd ascend-dra-driver git checkout 91d82a28 # the commit this lab was verified against helm upgrade --install ascend-dra-driver \ deployments/helm/ascend-dra-driver \ - --set image.tag=uuid-fix-20260909 \ + --set image.tag=latest \ -n ascend-dra-driver --create-namespace ``` @@ -213,7 +214,7 @@ The chart defaults to HAMivNPUCore mode. Do not set `kubeletPlugin.fullCardAndTr :::note About the image version used in this lab -The verification run used chart `ascend-dra-driver-0.1.1` with the image overridden to the development build `projecthami/ascend-dra-driver:uuid-fix-20260909`, which fixes a ResourceSlice uuid generation bug in earlier builds. Behavior may change before an official release; pin what you tested against. +The verification run used chart `ascend-dra-driver-0.1.1` with the development build `projecthami/ascend-dra-driver:uuid-fix-20260909`, which fixes a ResourceSlice uuid generation bug in earlier builds. That tag was never published to a public registry. The uuid fix is merged on the driver's master branch, and the `latest` image on Docker Hub was pushed on 2026-09-14, after that merge, so the command above installs `image.tag=latest` instead. Do not drop the override: the chart default resolves to an unpublished `0.1.0` tag. `latest` tracks master and moves; pin the exact tag or digest you test against. ::: diff --git a/tutorials/labs/examples/20-ascend-hami-dra/ascend-values.yaml b/tutorials/labs/examples/20-ascend-hami-dra/ascend-values.yaml new file mode 100644 index 000000000..9c5c6b8ea --- /dev/null +++ b/tutorials/labs/examples/20-ascend-hami-dra/ascend-values.yaml @@ -0,0 +1,14 @@ +deviceVendors: + - ascend # Ascend conversion only (910A/B2/B3/B4/B4-1/310P/910C) + +drivers: + nvidia: + enabled: false # no NVIDIA kubelet driver + fake: + enabled: false + +monitor: + enabled: false # optional Prometheus metrics component, not used here + +certs: + certManager: diff --git a/tutorials/labs/examples/20-ascend-hami-dra/pod.yaml b/tutorials/labs/examples/20-ascend-hami-dra/pod.yaml new file mode 100644 index 000000000..9ba3b93d6 --- /dev/null +++ b/tutorials/labs/examples/20-ascend-hami-dra/pod.yaml @@ -0,0 +1,20 @@ +apiVersion: v1 +kind: Pod +metadata: + name: ascend-share-a + namespace: dra-ascend-e2e + labels: + hami.io/webhook: ignore + annotations: + hami.io/use-Ascend310P-uuid: "68496E64-20E05477-92C31323-6E78030A-BD003019" # pins to npu-0-0 +spec: + runtimeClassName: ascend + containers: + - name: app + image: quay.io/ascend/vllm-ascend:v0.23.0-310p + command: [ "sh", "-c", "sleep 3600" ] + resources: + limits: + huawei.com/Ascend310P: 1 + huawei.com/Ascend310P-memory: "8192" + huawei.com/Ascend310P-core: "50" diff --git a/tutorials/overview.md b/tutorials/overview.md index 1485e04e1..5f032234a 100644 --- a/tutorials/overview.md +++ b/tutorials/overview.md @@ -33,4 +33,4 @@ Each lab lists its own prerequisites. - **Lab 16** installs HAMi v2.10.0 on a seven-GPU RTX PRO 6000 server and verifies the complete Dynamic MIG lifecycle: per-Pod placement, mixed profiles, selective reclamation, device-plugin restart recovery, and spillover to a second GPU. - **Lab 17** packages a model as a KitOps ModelKit, pulls it from Jozu Hub through an initContainer, and serves it with SGLang and optionally vLLM on HAMi GPU shares. - **Lab 18** stages a noisy neighbor next to a quiet tenant on two shared GPUs, proves what HAMi isolation covers and what it does not, and routes around the contention with Linkerd. -- **Lab 19** installs HAMi DRA 0.2.3 and the Ascend DRA driver on an Ascend 310P3 node and verifies the request-to-ResourceClaim conversion, two-Pod NPU sharing, memory quota enforcement, and scheduler capacity accounting. +- **Lab 20** installs HAMi DRA 0.2.3 and the Ascend DRA driver on an Ascend 310P3 node and verifies the request-to-ResourceClaim conversion, two-Pod NPU sharing, memory quota enforcement, and scheduler capacity accounting. From 0637280181bbbe512032c3791f1e9140bb7e6905 Mon Sep 17 00:00:00 2001 From: Jimmy Song Date: Fri, 9 Oct 2026 20:04:00 +0800 Subject: [PATCH 06/13] Fix indentation in pod.yaml for container definition Signed-off-by: Jimmy Song --- .../labs/examples/20-ascend-hami-dra/pod.yaml | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/tutorials/labs/examples/20-ascend-hami-dra/pod.yaml b/tutorials/labs/examples/20-ascend-hami-dra/pod.yaml index 9ba3b93d6..dca9c8103 100644 --- a/tutorials/labs/examples/20-ascend-hami-dra/pod.yaml +++ b/tutorials/labs/examples/20-ascend-hami-dra/pod.yaml @@ -10,11 +10,11 @@ metadata: spec: runtimeClassName: ascend containers: - - name: app - image: quay.io/ascend/vllm-ascend:v0.23.0-310p - command: [ "sh", "-c", "sleep 3600" ] - resources: - limits: - huawei.com/Ascend310P: 1 - huawei.com/Ascend310P-memory: "8192" - huawei.com/Ascend310P-core: "50" + - name: app + image: quay.io/ascend/vllm-ascend:v0.23.0-310p + command: ["sh", "-c", "sleep 3600"] + resources: + limits: + huawei.com/Ascend310P: 1 + huawei.com/Ascend310P-memory: "8192" + huawei.com/Ascend310P-core: "50" From 569617985cdfbf989d321b739bda556ec1b2a52b Mon Sep 17 00:00:00 2001 From: Jimmy Song Date: Sat, 10 Oct 2026 17:32:06 +0800 Subject: [PATCH 07/13] Fix examples manifests: cert-manager null override and hardcoded NPU uuid ascend-values.yaml dropped the final 'enabled: true' line on extraction, leaving certs.certManager as a null override that could mask the chart default; restore it. pod.yaml pinned the verification server's uuid; switch it to a REPLACE_WITH_YOUR_NPU_UUID placeholder and tell readers in both locales where to take their own uuid from before applying. Signed-off-by: Jimmy Song --- .../current/labs/ascend-hami-dra.md | 2 +- tutorials/labs/ascend-hami-dra.md | 2 +- tutorials/labs/examples/20-ascend-hami-dra/ascend-values.yaml | 1 + tutorials/labs/examples/20-ascend-hami-dra/pod.yaml | 2 +- 4 files changed, 4 insertions(+), 3 deletions(-) diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md index a0d74f6c9..e46c60616 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md @@ -354,7 +354,7 @@ spec: huawei.com/Ascend310P-core: "50" ``` -manifest 里的三个选择：`hami.io/webhook: ignore` 标签让共存的 HAMi core webhook 不碰这个 Pod；`runtimeClassName: ascend` 让 containerd 走 Ascend Docker Runtime；`vllm-ascend` 镜像内含 CANN、torch_npu 与 npu-smi，方便稍后从容器内部检查设备。 +manifest 里的三个选择：`hami.io/webhook: ignore` 标签让共存的 HAMi core webhook 不碰这个 Pod；`runtimeClassName: ascend` 让 containerd 走 Ascend Docker Runtime；`vllm-ascend` 镜像内含 CANN、torch_npu 与 npu-smi，方便稍后从容器内部检查设备。这里的 uuid 是验证服务器上的；若直接使用仓库里的 `tutorials/labs/examples/20-ascend-hami-dra/pod.yaml` 而非复制上面的代码块，请先把其中的 `REPLACE_WITH_YOUR_NPU_UUID` 占位符替换为你自己 `kubectl get resourceslice`（步骤 3）输出中的 uuid 再 apply。 ```bash kubectl apply -f pod.yaml diff --git a/tutorials/labs/ascend-hami-dra.md b/tutorials/labs/ascend-hami-dra.md index a429b61f9..8e8ed2be7 100644 --- a/tutorials/labs/ascend-hami-dra.md +++ b/tutorials/labs/ascend-hami-dra.md @@ -354,7 +354,7 @@ spec: huawei.com/Ascend310P-core: "50" ``` -Three choices in this manifest: the `hami.io/webhook: ignore` label keeps the coexisting HAMi core webhook away; `runtimeClassName: ascend` routes containerd through the Ascend Docker Runtime; the `vllm-ascend` image ships CANN, torch_npu, and npu-smi so the container can be inspected from inside later. +Three choices in this manifest: the `hami.io/webhook: ignore` label keeps the coexisting HAMi core webhook away; `runtimeClassName: ascend` routes containerd through the Ascend Docker Runtime; the `vllm-ascend` image ships CANN, torch_npu, and npu-smi so the container can be inspected from inside later. The uuid shown here is the verification server's; if you apply `tutorials/labs/examples/20-ascend-hami-dra/pod.yaml` from the repo instead of copying this block, replace its `REPLACE_WITH_YOUR_NPU_UUID` placeholder with a uuid from your own `kubectl get resourceslice` output (Step 3) before applying. ```bash kubectl apply -f pod.yaml diff --git a/tutorials/labs/examples/20-ascend-hami-dra/ascend-values.yaml b/tutorials/labs/examples/20-ascend-hami-dra/ascend-values.yaml index 9c5c6b8ea..9d512e89b 100644 --- a/tutorials/labs/examples/20-ascend-hami-dra/ascend-values.yaml +++ b/tutorials/labs/examples/20-ascend-hami-dra/ascend-values.yaml @@ -12,3 +12,4 @@ monitor: certs: certManager: + enabled: true diff --git a/tutorials/labs/examples/20-ascend-hami-dra/pod.yaml b/tutorials/labs/examples/20-ascend-hami-dra/pod.yaml index dca9c8103..6d34e85e0 100644 --- a/tutorials/labs/examples/20-ascend-hami-dra/pod.yaml +++ b/tutorials/labs/examples/20-ascend-hami-dra/pod.yaml @@ -6,7 +6,7 @@ metadata: labels: hami.io/webhook: ignore annotations: - hami.io/use-Ascend310P-uuid: "68496E64-20E05477-92C31323-6E78030A-BD003019" # pins to npu-0-0 + hami.io/use-Ascend310P-uuid: "REPLACE_WITH_YOUR_NPU_UUID" # take a uuid from `kubectl get resourceslice` output; pins the Pod to that NPU spec: runtimeClassName: ascend containers: From 0d73cee793945da8a25050c01d20d1c54a2daa59 Mon Sep 17 00:00:00 2001 From: Jimmy Song Date: Sat, 10 Oct 2026 17:52:17 +0800 Subject: [PATCH 08/13] Replace runtimeclass AGE placeholder with the real captured output Signed-off-by: Jimmy Song --- .../current/labs/ascend-hami-dra.md | 4 ++-- tutorials/labs/ascend-hami-dra.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md index e46c60616..ae00bad26 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md @@ -127,8 +127,8 @@ kubectl get runtimeclass ascend ``` ```text -NAME HANDLER AGE -ascend ascend ... +NAME HANDLER AGE +ascend ascend 23d ``` ### 与存量 HAMi core 部署共存 diff --git a/tutorials/labs/ascend-hami-dra.md b/tutorials/labs/ascend-hami-dra.md index 8e8ed2be7..fcad0575e 100644 --- a/tutorials/labs/ascend-hami-dra.md +++ b/tutorials/labs/ascend-hami-dra.md @@ -127,8 +127,8 @@ kubectl get runtimeclass ascend ``` ```text -NAME HANDLER AGE -ascend ascend ... +NAME HANDLER AGE +ascend ascend 23d ``` ### Coexistence with an existing HAMi core installation From 2388850e34c01d6bc8f94f876bfe9261e370ad03 Mon Sep 17 00:00:00 2001 From: Jimmy Song Date: Sat, 10 Oct 2026 17:55:51 +0800 Subject: [PATCH 09/13] Add captured outputs for the Step 2 driver and DRA object checks Replaces the 'Expected:' prose with verbatim kubectl get outputs for the driver Pod, DeviceClass, and ResourceSlice (en + zh). Signed-off-by: Jimmy Song --- .../current/labs/ascend-hami-dra.md | 17 ++++++++++++++++- tutorials/labs/ascend-hami-dra.md | 17 ++++++++++++++++- 2 files changed, 32 insertions(+), 2 deletions(-) diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md index ae00bad26..8b1452278 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md @@ -226,7 +226,22 @@ kubectl get deviceclass kubectl get resourceslice ``` -预期：driver Pod 全部 Running，一个名为 `hami-vnpu-core.project-hami.io` 的 DeviceClass，每个节点一条 driver 为 `ascend.project-hami.io` 的 ResourceSlice。 +```text +NAME READY STATUS RESTARTS AGE +ascend-dra-driver-kubeletplugin-h95tk 1/1 Running 0 22d +``` + +```text +NAME AGE +hami-vnpu-core.project-hami.io 22d +``` + +```text +NAME NODE DRIVER POOL AGE +aio-node74-arm-ascend.project-hami.io-gktzs aio-node74-arm ascend.project-hami.io aio-node74-arm 23d +``` + +driver Pod 全部 Running，一个名为 `hami-vnpu-core.project-hami.io` 的 DeviceClass，每个节点一条 driver 为 `ascend.project-hami.io` 的 ResourceSlice。 ## 步骤 3：查看 DeviceClass 与 ResourceSlice diff --git a/tutorials/labs/ascend-hami-dra.md b/tutorials/labs/ascend-hami-dra.md index fcad0575e..ffb2d3ed3 100644 --- a/tutorials/labs/ascend-hami-dra.md +++ b/tutorials/labs/ascend-hami-dra.md @@ -226,7 +226,22 @@ kubectl get deviceclass kubectl get resourceslice ``` -Expected: all driver Pods Running, one DeviceClass named `hami-vnpu-core.project-hami.io`, and one ResourceSlice per node with driver `ascend.project-hami.io`. +```text +NAME READY STATUS RESTARTS AGE +ascend-dra-driver-kubeletplugin-h95tk 1/1 Running 0 22d +``` + +```text +NAME AGE +hami-vnpu-core.project-hami.io 22d +``` + +```text +NAME NODE DRIVER POOL AGE +aio-node74-arm-ascend.project-hami.io-gktzs aio-node74-arm ascend.project-hami.io aio-node74-arm 23d +``` + +All driver Pods Running, one DeviceClass named `hami-vnpu-core.project-hami.io`, and one ResourceSlice per node with driver `ascend.project-hami.io`. ## Step 3: Inspect the DeviceClass and the ResourceSlice From c9e69456a0bcf9ec60054726369a2700e4b2cdb0 Mon Sep 17 00:00:00 2001 From: Jimmy Song Date: Sat, 17 Oct 2026 11:29:02 +0800 Subject: [PATCH 10/13] Fix verification date and update feature gates in ascend-hami-dra lab documentation Signed-off-by: Jimmy Song --- .../current/labs/ascend-hami-dra.md | 22 ++++++++++++++----- tutorials/labs/ascend-hami-dra.md | 22 ++++++++++++++----- 2 files changed, 34 insertions(+), 10 deletions(-) diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md index 8b1452278..1dc9b368b 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md @@ -9,7 +9,7 @@ lab: cost: 需要专属的昇腾 310P3 aarch64 硬件；HAMivNPUCore 软切分仅支持 ARM authors: - rootsongjc - verified: "2026-10-09" + verified: "2026-09-17" tags: - DRA - hami @@ -66,7 +66,7 @@ flowchart TD | 操作系统 | 麒麟 Kylin Linux Advanced Server V10（Lance），aarch64，内核 4.19.90-52.48.v2207.ky10 | | Kubernetes | v1.35.7，单节点 | | 容器运行时 | containerd 1.7.29，已开启 CDI | -| Feature gates | kube-apiserver / kube-scheduler / kubelet 均开启 `DRAConsumableCapacity=true` | +| Feature gates | kube-apiserver / kube-controller-manager / kube-scheduler / kubelet 均开启 `DRAConsumableCapacity=true` | | 昇腾驱动 / 固件 | 25.5.1 / 7.8.0.6.201 | | HAMi-DRA | 0.2.3（chart 与镜像 tag 一致） | | Ascend DRA driver | chart `ascend-dra-driver-0.1.1`（app 0.1.0），镜像 `projecthami/ascend-dra-driver:uuid-fix-20260909`（开发构建，见步骤 2） | @@ -271,8 +271,8 @@ kubectl get resourceslice ``` ```text -NAME DRIVER NODE AGE -aio-node74-arm-ascend.project-hami.io-gktzs ascend.project-hami.io aio-node74-arm 3m +NAME NODE DRIVER POOL AGE +aio-node74-arm-ascend.project-hami.io-gktzs aio-node74-arm ascend.project-hami.io aio-node74-arm 3m ``` 展开查看（节选；完整对象用 `kubectl get resourceslice -o yaml`）： @@ -596,7 +596,17 @@ pod b: /dev/davinci0 NPU_MEM_QUOTA=8192 NPU_PRIORITY=50 ASCEND_VISIBLE_DEVICE ## 步骤 8：验证隔离与配额强制 -在 Pod A 的容器内运行 PyTorch/torch_npu 张量分配测试（镜像内含完整 CANN 运行时）。先分配到接近配额，再尝试超限： +在 Pod A 的容器内运行 PyTorch/torch_npu 张量分配测试（镜像内含完整 CANN 运行时）。先分配到接近配额，再尝试超限。每次分配约 2 GiB 的块，并保留引用让分配器无法回收： + +```bash +kubectl exec -it ascend-share-a -n dra-ascend-e2e -- python3 -c " +import torch, torch_npu +blocks = [] +for i in range(8): # 第 4 块约 2050 MB 的申请越过 8192 MB 配额 + blocks.append(torch.ones(512 * 1024 * 1024, dtype=torch.float32, device='npu')) + print(f'block {i + 1} allocated') +" +``` ```text # limiter（HAMivNPUCore 容器内管理器）注册进程 @@ -624,6 +634,8 @@ npu-smi info # 进程列表（节选） | 4 0 | 49806 | python3 | 2156 | ``` +6150 MB 与 2156 MB 度量的是不同的东西，这个差距是预期行为：limiter 按进程的虚拟显存申请在 8192 MB 的 `NPU_MEM_QUOTA` 上记账，torch 分配器会先保留地址空间、再逐步落到物理显存；而 `npu-smi` 只统计每个进程实际占用的物理显存。因此配额记账先于物理占用到达上限，这正是基于配额的隔离层应有的行为。 + `NPU_PRIORITY=50` 同样被注入，通过时间片调度约束算力。本实验验证了注入，但没有运行长时间的满载对照 workload，因此不对 310P 上时间片隔离的实际强度下结论；算力维度在调度层的记账由下一步完整覆盖。 ## 步骤 9：耗尽容量、释放与再分配 diff --git a/tutorials/labs/ascend-hami-dra.md b/tutorials/labs/ascend-hami-dra.md index ffb2d3ed3..b540ee00d 100644 --- a/tutorials/labs/ascend-hami-dra.md +++ b/tutorials/labs/ascend-hami-dra.md @@ -9,7 +9,7 @@ lab: cost: requires dedicated Ascend 310P3 aarch64 hardware; HAMivNPUCore soft slicing is ARM-only authors: - rootsongjc - verified: "2026-10-09" + verified: "2026-09-17" tags: - dra - hami @@ -66,7 +66,7 @@ Three components, three responsibilities. Keep them apart throughout the lab: | OS | Kylin Linux Advanced Server V10 (Lance), aarch64, kernel 4.19.90-52.48.v2207.ky10 | | Kubernetes | v1.35.7, single node | | Container runtime | containerd 1.7.29, CDI enabled | -| Feature gates | `DRAConsumableCapacity=true` on kube-apiserver, kube-scheduler, kubelet | +| Feature gates | `DRAConsumableCapacity=true` on kube-apiserver, kube-controller-manager, kube-scheduler, kubelet | | Ascend driver / firmware | 25.5.1 / 7.8.0.6.201 | | HAMi-DRA | 0.2.3 (chart and image tag) | | Ascend DRA driver | chart `ascend-dra-driver-0.1.1` (app 0.1.0), image `projecthami/ascend-dra-driver:uuid-fix-20260909` (development build, see Step 2) | @@ -271,8 +271,8 @@ kubectl get resourceslice ``` ```text -NAME DRIVER NODE AGE -aio-node74-arm-ascend.project-hami.io-gktzs ascend.project-hami.io aio-node74-arm 3m +NAME NODE DRIVER POOL AGE +aio-node74-arm-ascend.project-hami.io-gktzs aio-node74-arm ascend.project-hami.io aio-node74-arm 3m ``` Expand it (abridged; `kubectl get resourceslice -o yaml` for the full object): @@ -596,7 +596,17 @@ In soft-slicing mode both containers see the same `/dev/davinci0`, not two vNPU ## Step 8: Verify Isolation and Quota Enforcement -Run a PyTorch/torch_npu tensor allocation test inside Pod A (the image carries the full CANN runtime). Allocate toward the quota, then past it: +Run a PyTorch/torch_npu tensor allocation test inside Pod A (the image carries the full CANN runtime). Allocate toward the quota, then past it. One block of ~2 GiB at a time, keeping references so the allocator cannot reclaim them: + +```bash +kubectl exec -it ascend-share-a -n dra-ascend-e2e -- python3 -c " +import torch, torch_npu +blocks = [] +for i in range(8): # the 4th block's ~2050 MB request crosses the 8192 MB quota + blocks.append(torch.ones(512 * 1024 * 1024, dtype=torch.float32, device='npu')) + print(f'block {i + 1} allocated') +" +``` ```text # the limiter (HAMivNPUCore's in-container manager) registers the process @@ -624,6 +634,8 @@ npu-smi info # process list, abridged | 4 0 | 49806 | python3 | 2156 | ``` +The 6150 MB and the 2156 MB measure different things, and the gap is expected: the limiter debits each process's virtual allocations against the 8192 MB `NPU_MEM_QUOTA`, and the torch allocator reserves address space before physical memory is touched, while `npu-smi` shows only the physical memory each process has pinned on the device. Quota accounting therefore reaches the limit while physical usage is still well below it, which is the intended behavior of a quota-based isolation layer. + `NPU_PRIORITY=50` was injected the same way and constrains compute through time-slice scheduling. This lab verified the injection but did not run a long-running saturated comparison workload, so it draws no conclusion about the effective strength of time-slice isolation on the 310P; the scheduling-side accounting of the compute dimension is fully covered in the next step. ## Step 9: Exhaust Capacity, Release, and Reallocate From 2964950556cdb409165c4de648a528274a104b8c Mon Sep 17 00:00:00 2001 From: Jimmy Song Date: Sat, 17 Oct 2026 12:15:26 +0800 Subject: [PATCH 11/13] Add KitOps and ModelKit tags to the documentation Signed-off-by: Jimmy Song --- .../current/tags.yml | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/tags.yml b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/tags.yml index 86b765c4d..50de359f4 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/tags.yml +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/tags.yml @@ -43,6 +43,12 @@ "kserve": label: "KServe" permalink: "/kserve" +"kitops": + label: "KitOps" + permalink: "/kitops" +"modelkit": + label: "ModelKit" + permalink: "/modelkit" "nvidia": label: "nvidia" permalink: "/nvidia" @@ -100,12 +106,6 @@ "隔离": label: "隔离" permalink: "/isolation" -"kitops": - label: "KitOps" - permalink: "/kitops" -"modelkit": - label: "ModelKit" - permalink: "/modelkit" "service-mesh": label: "服务网格" permalink: "/service-mesh" From ee5c376c01f51e36332d7c77291c6b6d7f3b8e84 Mon Sep 17 00:00:00 2001 From: Jimmy Song Date: Mon, 19 Oct 2026 09:28:14 +0800 Subject: [PATCH 12/13] Update references from Lab 20 to Lab 21 in documentation and examples for HAMi DRA Signed-off-by: Jimmy Song --- blog/hami-dra-npu-sharing/index.md | 10 +++++----- .../hami-dra-npu-sharing/index.md | 10 +++++----- .../current/labs/ascend-hami-dra.md | 8 ++++---- .../current/overview.md | 2 +- tutorials/labs/ascend-hami-dra.md | 8 ++++---- .../ascend-values.yaml | 0 .../pod.yaml | 0 tutorials/overview.md | 2 +- 8 files changed, 20 insertions(+), 20 deletions(-) rename tutorials/labs/examples/{20-ascend-hami-dra => 21-ascend-hami-dra}/ascend-values.yaml (100%) rename tutorials/labs/examples/{20-ascend-hami-dra => 21-ascend-hami-dra}/pod.yaml (100%) diff --git a/blog/hami-dra-npu-sharing/index.md b/blog/hami-dra-npu-sharing/index.md index dc2bbbf3b..d1e0cd930 100644 --- a/blog/hami-dra-npu-sharing/index.md +++ b/blog/hami-dra-npu-sharing/index.md @@ -10,7 +10,7 @@ On September 15, 2026, the HAMi community released [HAMi DRA](https://github.com The discussion around it has not stopped either. Ever since Kubernetes 1.34 took DRA (Dynamic Resource Allocation) to GA, "does DRA make HAMi obsolete?" has been a frequent question in the community channels; the answer given in [Does Kubernetes DRA Replace HAMi?](/blog/does-kubernetes-dra-replace-hami) is that DRA absorbs the request-and-scheduling half, while the enforce-inside-the-container half was never something DRA set out to do, and that is exactly where HAMi stays. -HAMi DRA is the community's engineering answer to that debate: a migration layer that automatically converts the HAMi-style resource requests in existing workloads into native ResourceClaims, hands scheduling and accounting back to kube-scheduler, keeps runtime isolation with HAMi-core, and leaves the business side without a single line to change. The current 0.2.3 release covers NVIDIA GPUs; with the Ascend DRA driver, the chain has been verified end to end on a real 310P3 cluster, and the community shipped the companion [Lab 20: Ascend NPU Sharing with HAMi DRA](/tutorials/labs/ascend-hami-dra) with commands and real captured output. This post is a systematic introduction to HAMi DRA: its motivation, design, usage, and current boundaries. +HAMi DRA is the community's engineering answer to that debate: a migration layer that automatically converts the HAMi-style resource requests in existing workloads into native ResourceClaims, hands scheduling and accounting back to kube-scheduler, keeps runtime isolation with HAMi-core, and leaves the business side without a single line to change. The current 0.2.3 release covers NVIDIA GPUs; with the Ascend DRA driver, the chain has been verified end to end on a real 310P3 cluster, and the community shipped the companion [Lab 21: Ascend NPU Sharing with HAMi DRA](/tutorials/labs/ascend-hami-dra) with commands and real captured output. This post is a systematic introduction to HAMi DRA: its motivation, design, usage, and current boundaries. @@ -151,7 +151,7 @@ For a Pod submitted to a real Ascend 310P cluster, the three `huawei.com/*` entr | `capacity.requests.cores: "50"` | `-core: 50` | percent passed through | | CEL selectors | `use-Ascend310P-uuid` annotation | `uuid in ["..."]`, the value taken from the ResourceSlice | -When two Pods each request 8192 MiB plus 50 cores against the same card, both claims land on the same device with independent shareIDs, and the scheduler debits the first one's consumption before allocating the second. Once capacity is fully debited, the next request is refused by the scheduler itself (the claim stays pending); when a Pod is deleted, its claim is reclaimed automatically and the accounting recovers. In-container enforcement is still HAMi-core: `libvnpu.so` intercepts over-quota memory requests and turns them into an in-container OOM, and the device as seen from the container is virtualized to the requested 8 GiB rather than the card's 21 GiB. The full walkthrough is [Lab 20](/tutorials/labs/ascend-hami-dra). +When two Pods each request 8192 MiB plus 50 cores against the same card, both claims land on the same device with independent shareIDs, and the scheduler debits the first one's consumption before allocating the second. Once capacity is fully debited, the next request is refused by the scheduler itself (the claim stays pending); when a Pod is deleted, its claim is reclaimed automatically and the accounting recovers. In-container enforcement is still HAMi-core: `libvnpu.so` intercepts over-quota memory requests and turns them into an in-container OOM, and the device as seen from the container is virtualized to the requested 8 GiB rather than the card's 21 GiB. The full walkthrough is [Lab 21](/tutorials/labs/ascend-hami-dra). ## The device support timeline, and "why now" @@ -161,7 +161,7 @@ What HAMi DRA can do is bounded by each vendor's DRA driver progress: - 2026.03: Hygon DCU DRA driver integrated with HAMi DRA - 2026.04: Enflame DRA driver, integration under way -The Ascend path extends HAMi DRA's reach to NPUs: [Lab 20](/tutorials/labs/ascend-hami-dra) has already run the complete chain from HAMi request to NPU sharing on a 310P3 server with the official [Project-HAMi/ascend-dra-driver](https://github.com/Project-HAMi/ascend-dra-driver) at chart 0.1.1 (the lab used the development build fixing uuid generation; behavior may change before an official release). HAMi 2.10.0 was released in August 2026 and its release notes now point the community to the Ascend DRA driver repository; the driver itself is still pre-release, so the window for trying it and giving feedback remains open. +The Ascend path extends HAMi DRA's reach to NPUs: [Lab 21](/tutorials/labs/ascend-hami-dra) has already run the complete chain from HAMi request to NPU sharing on a 310P3 server with the official [Project-HAMi/ascend-dra-driver](https://github.com/Project-HAMi/ascend-dra-driver) at chart 0.1.1 (the lab used the development build fixing uuid generation; behavior may change before an official release). HAMi 2.10.0 was released in August 2026 and its release notes now point the community to the Ascend DRA driver repository; the driver itself is still pre-release, so the window for trying it and giving feedback remains open. ## The practical constraints before adoption @@ -172,7 +172,7 @@ The adoption friction groups into four: - **Missing scoring**: DRA lacks scoring at the device-scheduling layer, limiting support for specialized scheduling requirements. - **Learning cost**: DRA's new concepts carry a threshold for the people deploying and maintaining clusters; the HAMi community's countermeasure is precisely the webhook-compatible mode keeping that cost out of the user's sight. -On choosing between them, the community's guidance: for standardized clusters chasing the best scheduling outcome (topology-aware scheduling, for example), plain HAMi is recommended; for highly customized clusters (already running their own scheduler) or plug-and-play device reuse, HAMi DRA is recommended. Two operational facts are also worth remembering: DRA mode and the traditional mode are two different paths for the same request, and one request must not go through both; when HAMi core is still installed, the webhook exemptions used in Lab 20 let the two coexist in one cluster. And DRA mode still goes through HAMi-core. +On choosing between them, the community's guidance: for standardized clusters chasing the best scheduling outcome (topology-aware scheduling, for example), plain HAMi is recommended; for highly customized clusters (already running their own scheduler) or plug-and-play device reuse, HAMi DRA is recommended. Two operational facts are also worth remembering: DRA mode and the traditional mode are two different paths for the same request, and one request must not go through both; when HAMi core is still installed, the webhook exemptions used in Lab 21 let the two coexist in one cluster. And DRA mode still goes through HAMi-core. ## Outlook @@ -188,5 +188,5 @@ Zooming out: Kubernetes is evolving into the control plane of AI infrastructure, - HAMi 2.9 release webinar recap by Li Mengxuan (in Chinese): [HAMi 2.9 Ascend soft slicing and DRA in depth](https://dynamia.ai/zh/blog/hami-2.9-webinar-recap) - Community getting-started guide (in Chinese): [HAMi meets Kubernetes DRA: a practical guide](https://dynamia.ai/zh/blog/hami-dra-quickstart) - Mesut Oezdil: [Does Kubernetes DRA Replace HAMi?](/blog/does-kubernetes-dra-replace-hami) ([CNCF blog original](https://www.cncf.io/blog/2026/08/07/does-kubernetes-dra-replace-hami/)) -- User guide: [How to use HAMi DRA](/docs/installation/how-to-use-hami-dra); hands-on: [Lab 20: Ascend NPU Sharing with HAMi DRA](/tutorials/labs/ascend-hami-dra) and [Lab 4: GPU Slicing with Dynamic Resource Allocation](/tutorials/labs/hami-dra) +- User guide: [How to use HAMi DRA](/docs/installation/how-to-use-hami-dra); hands-on: [Lab 21: Ascend NPU Sharing with HAMi DRA](/tutorials/labs/ascend-hami-dra) and [Lab 4: GPU Slicing with Dynamic Resource Allocation](/tutorials/labs/hami-dra) - Components: [Project-HAMi/HAMi-DRA](https://github.com/Project-HAMi/HAMi-DRA) · [Project-HAMi/ascend-dra-driver](https://github.com/Project-HAMi/ascend-dra-driver) · [Project-HAMi/hami-vnpu-core](https://github.com/Project-HAMi/hami-vnpu-core) diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-dra-npu-sharing/index.md b/i18n/zh/docusaurus-plugin-content-blog/hami-dra-npu-sharing/index.md index aaa5911f3..605f0e84c 100644 --- a/i18n/zh/docusaurus-plugin-content-blog/hami-dra-npu-sharing/index.md +++ b/i18n/zh/docusaurus-plugin-content-blog/hami-dra-npu-sharing/index.md @@ -10,7 +10,7 @@ tags: ["HAMi", "DRA", "Ascend", "NPU 共享", "Kubernetes"] 围绕它的讨论也一直没有停。自从 Kubernetes 1.34 将 DRA（Dynamic Resource Allocation）推进至 GA，“DRA 会不会让 HAMi 过时”就成了社区频道里的高频问题；[《Kubernetes DRA 会取代 HAMi 吗？》](/zh/blog/does-kubernetes-dra-replace-hami)给出的回答是：DRA 吸收的是“请求与调度”的那一半，而“容器内强制执行”的那一半，DRA 从未设计去做，这正是 HAMi 保留的位置。 -HAMi DRA 就是社区对这个争论的工程回应：一个迁移层，把存量 workload 里的 HAMi 风格资源请求自动转换成原生 ResourceClaim，调度与记账交还 kube-scheduler，运行时隔离继续由 HAMi-core 完成，业务侧一行代码不用改。当前 0.2.3 版本覆盖 NVIDIA GPU；在昇腾 DRA Driver 的配合下，这条链路已在真实的 310P3 集群上完成端到端验证，社区同步发布了配套的 [实验 20：用 HAMi DRA 共享昇腾 NPU](/zh/tutorials/labs/ascend-hami-dra)，提供命令与真实采集输出。本文系统介绍 HAMi DRA 的动机、设计、使用方式与当前边界。 +HAMi DRA 就是社区对这个争论的工程回应：一个迁移层，把存量 workload 里的 HAMi 风格资源请求自动转换成原生 ResourceClaim，调度与记账交还 kube-scheduler，运行时隔离继续由 HAMi-core 完成，业务侧一行代码不用改。当前 0.2.3 版本覆盖 NVIDIA GPU；在昇腾 DRA Driver 的配合下，这条链路已在真实的 310P3 集群上完成端到端验证，社区同步发布了配套的 [实验 21：用 HAMi DRA 共享昇腾 NPU](/zh/tutorials/labs/ascend-hami-dra)，提供命令与真实采集输出。本文系统介绍 HAMi DRA 的动机、设计、使用方式与当前边界。 @@ -151,7 +151,7 @@ Webhook 在准入阶段把三个 HAMi 资源条目换成 ResourceClaim 的 `coun | `capacity.requests.cores: "50"` | `-core: 50` | 百分比直传 | | CEL 选择器 | `use-Ascend310P-uuid` 注解 | `uuid in ["..."]`，值来自 ResourceSlice | -两个 Pod 各请求 8192 MiB 显存加 50 算力、指向同一块卡时，两个 claim 落到同一个 device 上，各自带独立 shareID，调度器在分配第二个之前已把第一个的消费记账。容量记满后新请求被调度器直接拒绝（claim 停在 pending），Pod 删除后 claim 自动回收、记账恢复。容器内的强制执行依旧是 HAMi-core：`libvnpu.so` 拦截超限的显存申请并转为容器内 OOM，容器视角的设备被虚拟化为申请的 8 GiB 而非整卡 21 GiB。完整过程见[实验 20](/zh/tutorials/labs/ascend-hami-dra)。 +两个 Pod 各请求 8192 MiB 显存加 50 算力、指向同一块卡时，两个 claim 落到同一个 device 上，各自带独立 shareID，调度器在分配第二个之前已把第一个的消费记账。容量记满后新请求被调度器直接拒绝（claim 停在 pending），Pod 删除后 claim 自动回收、记账恢复。容器内的强制执行依旧是 HAMi-core：`libvnpu.so` 拦截超限的显存申请并转为容器内 OOM，容器视角的设备被虚拟化为申请的 8 GiB 而非整卡 21 GiB。完整过程见[实验 21](/zh/tutorials/labs/ascend-hami-dra)。 ## 设备支持的时间线，与“为什么是现在” @@ -161,7 +161,7 @@ HAMi DRA 的能力边界很大程度上取决于各厂商 DRA driver 的适配 - 2026.03：海光 DCU DRA Driver 完成与 HAMi DRA 的集成 - 2026.04：Enflame（燧原）DRA Driver 集成推进中 -昇腾链路的打通，意味着 HAMi DRA 的能力边界扩展到了 NPU：[实验 20](/zh/tutorials/labs/ascend-hami-dra) 已在一台 310P3 服务器上，用官方 [Project-HAMi/ascend-dra-driver](https://github.com/Project-HAMi/ascend-dra-driver)（chart 0.1.1）把 HAMi 请求到 NPU 共享的完整链路跑通（实验所用镜像为修复 uuid 生成问题的开发构建，正式发布前行为可能变化）。HAMi 2.10.0 已于 2026 年 8 月发布，release notes 中已把昇腾 DRA 指向该 driver 仓库；driver 本身仍是预发布状态，现在仍是尝鲜与反馈的窗口。 +昇腾链路的打通，意味着 HAMi DRA 的能力边界扩展到了 NPU：[实验 21](/zh/tutorials/labs/ascend-hami-dra) 已在一台 310P3 服务器上，用官方 [Project-HAMi/ascend-dra-driver](https://github.com/Project-HAMi/ascend-dra-driver)（chart 0.1.1）把 HAMi 请求到 NPU 共享的完整链路跑通（实验所用镜像为修复 uuid 生成问题的开发构建，正式发布前行为可能变化）。HAMi 2.10.0 已于 2026 年 8 月发布，release notes 中已把昇腾 DRA 指向该 driver 仓库；driver 本身仍是预发布状态，现在仍是尝鲜与反馈的窗口。 ## 落地前的现实约束 @@ -172,7 +172,7 @@ HAMi DRA 的能力边界很大程度上取决于各厂商 DRA driver 的适配 - **Scoring 缺失**：设备调度层面 DRA 的打分能力尚缺，对特性化调度要求的支持有限。 - **学习成本**：DRA 引入的新概念对集群部署与维护同学有门槛，HAMi 社区的对策正是用 Webhook 兼容模式把这份成本挡在用户视线之外。 -选型上，社区给出的建议：标准化集群、追求最优调度效果（如拓扑感知），推荐原生 HAMi；高度定制化（已有自己的调度器）或需要即插即用设备复用的集群，推荐 HAMi DRA。此外两条运维事实值得记住：DRA 模式与传统模式是同一请求的两条不同路径，一个请求不能同时走两条路；集群里仍装有 HAMi core 时，用实验 20 中的 webhook 豁免可以让两者共存于同一集群。使用 DRA 模式仍然会经过 HAMi-core。 +选型上，社区给出的建议：标准化集群、追求最优调度效果（如拓扑感知），推荐原生 HAMi；高度定制化（已有自己的调度器）或需要即插即用设备复用的集群，推荐 HAMi DRA。此外两条运维事实值得记住：DRA 模式与传统模式是同一请求的两条不同路径，一个请求不能同时走两条路；集群里仍装有 HAMi core 时，用实验 21 中的 webhook 豁免可以让两者共存于同一集群。使用 DRA 模式仍然会经过 HAMi-core。 ## 展望 @@ -188,5 +188,5 @@ HAMi-DRA 的 roadmap 上排着：更多异构设备（MetaX、天数智芯 Iluva - HAMi 2.9 版本直播回顾（李孟轩）：[HAMi 2.9 昇腾软切分与 DRA 实战详解](https://dynamia.ai/zh/blog/hami-2.9-webinar-recap) - 社区上手指南：[HAMi 正式接入 Kubernetes DRA：下一代 GPU 资源模型实践指南](https://dynamia.ai/zh/blog/hami-dra-quickstart) - Mesut Oezdil：[Kubernetes DRA 会取代 HAMi 吗？](/zh/blog/does-kubernetes-dra-replace-hami)（[英文原文](https://www.cncf.io/blog/2026/08/07/does-kubernetes-dra-replace-hami/)，CNCF 博客） -- 用户指南：[如何使用 HAMi DRA](/zh/docs/installation/how-to-use-hami-dra)；动手实验：[实验 20：用 HAMi DRA 共享昇腾 NPU](/zh/tutorials/labs/ascend-hami-dra)、[实验 4：用 DRA 切分 GPU](/zh/tutorials/labs/hami-dra) +- 用户指南：[如何使用 HAMi DRA](/zh/docs/installation/how-to-use-hami-dra)；动手实验：[实验 21：用 HAMi DRA 共享昇腾 NPU](/zh/tutorials/labs/ascend-hami-dra)、[实验 4：用 DRA 切分 GPU](/zh/tutorials/labs/hami-dra) - 相关组件：[Project-HAMi/HAMi-DRA](https://github.com/Project-HAMi/HAMi-DRA) · [Project-HAMi/ascend-dra-driver](https://github.com/Project-HAMi/ascend-dra-driver) · [Project-HAMi/hami-vnpu-core](https://github.com/Project-HAMi/hami-vnpu-core) diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md index 1dc9b368b..27948bffd 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md @@ -1,7 +1,7 @@ --- -title: "实验 20：用 HAMi DRA 共享昇腾 NPU" +title: "实验 21：用 HAMi DRA 共享昇腾 NPU" description: "在昇腾 310P3 节点上安装 HAMi DRA 0.2.3 与 Ascend DRA Driver，观察 webhook 如何把 huawei.com/Ascend310P 请求转换成 ResourceClaim，并验证双 Pod NPU 共享、显存配额隔离与调度器容量记账。" -sidebar_label: "实验 20：昇腾 + HAMi DRA" +sidebar_label: "实验 21：昇腾 + HAMi DRA" lab: level: Advanced duration: 约 120 分钟 @@ -56,7 +56,7 @@ flowchart TD - ARM（aarch64）宿主机，昇腾驱动 25.5 或更高，用于软切分的每块 NPU 都开启 device-share 模式。 - `ascend` RuntimeClass 存在（Pod 需要 `runtimeClassName: ascend`）。 - cert-manager（HAMi-DRA 用它签发 webhook 证书）。 -- 清单文件来自 [`tutorials/labs/examples/20-ascend-hami-dra/`](https://github.com/Project-HAMi/website/tree/master/tutorials/labs/examples/20-ascend-hami-dra)（`ascend-values.yaml` 与 `pod.yaml`），下文步骤中均已内联，无需克隆仓库。 +- 清单文件来自 [`tutorials/labs/examples/21-ascend-hami-dra/`](https://github.com/Project-HAMi/website/tree/master/tutorials/labs/examples/21-ascend-hami-dra)（`ascend-values.yaml` 与 `pod.yaml`），下文步骤中均已内联，无需克隆仓库。 ### 验证环境 @@ -369,7 +369,7 @@ spec: huawei.com/Ascend310P-core: "50" ``` -manifest 里的三个选择：`hami.io/webhook: ignore` 标签让共存的 HAMi core webhook 不碰这个 Pod；`runtimeClassName: ascend` 让 containerd 走 Ascend Docker Runtime；`vllm-ascend` 镜像内含 CANN、torch_npu 与 npu-smi，方便稍后从容器内部检查设备。这里的 uuid 是验证服务器上的；若直接使用仓库里的 `tutorials/labs/examples/20-ascend-hami-dra/pod.yaml` 而非复制上面的代码块，请先把其中的 `REPLACE_WITH_YOUR_NPU_UUID` 占位符替换为你自己 `kubectl get resourceslice`（步骤 3）输出中的 uuid 再 apply。 +manifest 里的三个选择：`hami.io/webhook: ignore` 标签让共存的 HAMi core webhook 不碰这个 Pod；`runtimeClassName: ascend` 让 containerd 走 Ascend Docker Runtime；`vllm-ascend` 镜像内含 CANN、torch_npu 与 npu-smi，方便稍后从容器内部检查设备。这里的 uuid 是验证服务器上的；若直接使用仓库里的 `tutorials/labs/examples/21-ascend-hami-dra/pod.yaml` 而非复制上面的代码块，请先把其中的 `REPLACE_WITH_YOUR_NPU_UUID` 占位符替换为你自己 `kubectl get resourceslice`（步骤 3）输出中的 uuid 再 apply。 ```bash kubectl apply -f pod.yaml diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md index c787ba7c8..a0bc7735a 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md @@ -34,4 +34,4 @@ import LabCardGridAuto from '@site/src/components/labs/LabCardGridAuto'; - **实验 17** 将模型打包为 KitOps ModelKit，通过 initContainer 从 Jozu Hub 拉取，并使用 SGLang（可选 vLLM）在 HAMi GPU 共享资源上提供服务。 - **实验 18** 在两张共享 GPU 上同时运行吵闹邻居与安静租户负载，验证 HAMi 隔离能覆盖与不能覆盖的边界，并使用 Linkerd 让敏感流量绕开竞争。 - **实验 19** 在 310P3 节点上安装 HAMi 2.10.0 与昇腾设备插件，验证模板硬切（`vir01`、`vir02`、`vir04`）、多 Pod 共卡、容量耗尽、整卡互斥和分配指标。 -- **实验 20** 在昇腾 310P3 节点上安装 HAMi DRA 0.2.3 与 Ascend DRA Driver，验证请求到 ResourceClaim 的转换、双 Pod NPU 共享、显存配额隔离与调度器容量记账。 +- **实验 21** 在昇腾 310P3 节点上安装 HAMi DRA 0.2.3 与 Ascend DRA Driver，验证请求到 ResourceClaim 的转换、双 Pod NPU 共享、显存配额隔离与调度器容量记账。 diff --git a/tutorials/labs/ascend-hami-dra.md b/tutorials/labs/ascend-hami-dra.md index b540ee00d..862bce704 100644 --- a/tutorials/labs/ascend-hami-dra.md +++ b/tutorials/labs/ascend-hami-dra.md @@ -1,7 +1,7 @@ --- -title: "Lab 20: Ascend NPU Sharing with HAMi DRA" +title: "Lab 21: Ascend NPU Sharing with HAMi DRA" description: "Install HAMi DRA 0.2.3 and the Ascend DRA driver on an Ascend 310P3 node, watch the webhook turn huawei.com/Ascend310P requests into ResourceClaims, and verify two-Pod NPU sharing, memory quota enforcement, and scheduler capacity accounting." -sidebar_label: "Lab 20: Ascend + HAMi DRA" +sidebar_label: "Lab 21: Ascend + HAMi DRA" lab: level: Advanced duration: about 120 minutes @@ -56,7 +56,7 @@ Three components, three responsibilities. Keep them apart throughout the lab: - Ascend driver 25.5 or newer on an ARM (aarch64) host, with device-share mode enabled on each NPU used for soft slicing. - The `ascend` RuntimeClass present (Pods need `runtimeClassName: ascend`). - cert-manager (HAMi-DRA uses it to issue webhook certificates). -- Manifests from [`tutorials/labs/examples/20-ascend-hami-dra/`](https://github.com/Project-HAMi/website/tree/master/tutorials/labs/examples/20-ascend-hami-dra) (`ascend-values.yaml` and `pod.yaml`); both are also inlined in the steps below, so nothing has to be cloned. +- Manifests from [`tutorials/labs/examples/21-ascend-hami-dra/`](https://github.com/Project-HAMi/website/tree/master/tutorials/labs/examples/21-ascend-hami-dra) (`ascend-values.yaml` and `pod.yaml`); both are also inlined in the steps below, so nothing has to be cloned. ### Verification environment @@ -369,7 +369,7 @@ spec: huawei.com/Ascend310P-core: "50" ``` -Three choices in this manifest: the `hami.io/webhook: ignore` label keeps the coexisting HAMi core webhook away; `runtimeClassName: ascend` routes containerd through the Ascend Docker Runtime; the `vllm-ascend` image ships CANN, torch_npu, and npu-smi so the container can be inspected from inside later. The uuid shown here is the verification server's; if you apply `tutorials/labs/examples/20-ascend-hami-dra/pod.yaml` from the repo instead of copying this block, replace its `REPLACE_WITH_YOUR_NPU_UUID` placeholder with a uuid from your own `kubectl get resourceslice` output (Step 3) before applying. +Three choices in this manifest: the `hami.io/webhook: ignore` label keeps the coexisting HAMi core webhook away; `runtimeClassName: ascend` routes containerd through the Ascend Docker Runtime; the `vllm-ascend` image ships CANN, torch_npu, and npu-smi so the container can be inspected from inside later. The uuid shown here is the verification server's; if you apply `tutorials/labs/examples/21-ascend-hami-dra/pod.yaml` from the repo instead of copying this block, replace its `REPLACE_WITH_YOUR_NPU_UUID` placeholder with a uuid from your own `kubectl get resourceslice` output (Step 3) before applying. ```bash kubectl apply -f pod.yaml diff --git a/tutorials/labs/examples/20-ascend-hami-dra/ascend-values.yaml b/tutorials/labs/examples/21-ascend-hami-dra/ascend-values.yaml similarity index 100% rename from tutorials/labs/examples/20-ascend-hami-dra/ascend-values.yaml rename to tutorials/labs/examples/21-ascend-hami-dra/ascend-values.yaml diff --git a/tutorials/labs/examples/20-ascend-hami-dra/pod.yaml b/tutorials/labs/examples/21-ascend-hami-dra/pod.yaml similarity index 100% rename from tutorials/labs/examples/20-ascend-hami-dra/pod.yaml rename to tutorials/labs/examples/21-ascend-hami-dra/pod.yaml diff --git a/tutorials/overview.md b/tutorials/overview.md index 5f69bdaea..483f8b334 100644 --- a/tutorials/overview.md +++ b/tutorials/overview.md @@ -34,4 +34,4 @@ Each lab lists its own prerequisites. - **Lab 17** packages a model as a KitOps ModelKit, pulls it from Jozu Hub through an initContainer, and serves it with SGLang and optionally vLLM on HAMi GPU shares. - **Lab 18** stages a noisy neighbor next to a quiet tenant on two shared GPUs, proves what HAMi isolation covers and what it does not, and routes around the contention with Linkerd. - **Lab 19** installs HAMi 2.10.0 and the Ascend device plugin on a 310P3 node, then verifies template-based hard slicing (`vir01`, `vir02`, and `vir04`), multi-Pod sharing, capacity exhaustion, whole-card exclusion, and allocation metrics. -- **Lab 20** installs HAMi DRA 0.2.3 and the Ascend DRA driver on an Ascend 310P3 node and verifies the request-to-ResourceClaim conversion, two-Pod NPU sharing, memory quota enforcement, and scheduler capacity accounting. +- **Lab 21** installs HAMi DRA 0.2.3 and the Ascend DRA driver on an Ascend 310P3 node and verifies the request-to-ResourceClaim conversion, two-Pod NPU sharing, memory quota enforcement, and scheduler capacity accounting. From b105ee0ad7c9955d9c44beb0427487e609d8fede Mon Sep 17 00:00:00 2001 From: Jimmy Song Date: Mon, 19 Oct 2026 17:27:09 +0800 Subject: [PATCH 13/13] docs: clarify resource allocation behavior and update Helm installation instructions for HAMi DRA Signed-off-by: Jimmy Song --- blog/hami-dra-npu-sharing/index.md | 2 +- .../hami-dra-npu-sharing/index.md | 2 +- .../current/labs/ascend-hami-dra.md | 30 ++++++++++--------- tutorials/labs/ascend-hami-dra.md | 30 ++++++++++--------- 4 files changed, 34 insertions(+), 30 deletions(-) diff --git a/blog/hami-dra-npu-sharing/index.md b/blog/hami-dra-npu-sharing/index.md index d1e0cd930..c0173a3b7 100644 --- a/blog/hami-dra-npu-sharing/index.md +++ b/blog/hami-dra-npu-sharing/index.md @@ -151,7 +151,7 @@ For a Pod submitted to a real Ascend 310P cluster, the three `huawei.com/*` entr | `capacity.requests.cores: "50"` | `-core: 50` | percent passed through | | CEL selectors | `use-Ascend310P-uuid` annotation | `uuid in ["..."]`, the value taken from the ResourceSlice | -When two Pods each request 8192 MiB plus 50 cores against the same card, both claims land on the same device with independent shareIDs, and the scheduler debits the first one's consumption before allocating the second. Once capacity is fully debited, the next request is refused by the scheduler itself (the claim stays pending); when a Pod is deleted, its claim is reclaimed automatically and the accounting recovers. In-container enforcement is still HAMi-core: `libvnpu.so` intercepts over-quota memory requests and turns them into an in-container OOM, and the device as seen from the container is virtualized to the requested 8 GiB rather than the card's 21 GiB. The full walkthrough is [Lab 21](/tutorials/labs/ascend-hami-dra). +When two Pods each request 8192 MiB plus 50 cores against the same card, both claims land on the same device with independent shareIDs, and the scheduler debits the first one's consumption before allocating the second. Once core capacity is exhausted (the two claims hold 100 of 100 cores; memory still has headroom), the next identical request is refused by the scheduler itself (the claim stays pending); when a Pod is deleted, its claim is reclaimed automatically and the accounting recovers. In-container enforcement is still HAMi-core: `libvnpu.so` intercepts over-quota memory requests and turns them into an in-container OOM, and the device as seen from the container is virtualized to the requested 8 GiB rather than the card's 21 GiB. The full walkthrough is [Lab 21](/tutorials/labs/ascend-hami-dra). ## The device support timeline, and "why now" diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-dra-npu-sharing/index.md b/i18n/zh/docusaurus-plugin-content-blog/hami-dra-npu-sharing/index.md index 605f0e84c..ac86394d7 100644 --- a/i18n/zh/docusaurus-plugin-content-blog/hami-dra-npu-sharing/index.md +++ b/i18n/zh/docusaurus-plugin-content-blog/hami-dra-npu-sharing/index.md @@ -151,7 +151,7 @@ Webhook 在准入阶段把三个 HAMi 资源条目换成 ResourceClaim 的 `coun | `capacity.requests.cores: "50"` | `-core: 50` | 百分比直传 | | CEL 选择器 | `use-Ascend310P-uuid` 注解 | `uuid in ["..."]`，值来自 ResourceSlice | -两个 Pod 各请求 8192 MiB 显存加 50 算力、指向同一块卡时，两个 claim 落到同一个 device 上，各自带独立 shareID，调度器在分配第二个之前已把第一个的消费记账。容量记满后新请求被调度器直接拒绝（claim 停在 pending），Pod 删除后 claim 自动回收、记账恢复。容器内的强制执行依旧是 HAMi-core：`libvnpu.so` 拦截超限的显存申请并转为容器内 OOM，容器视角的设备被虚拟化为申请的 8 GiB 而非整卡 21 GiB。完整过程见[实验 21](/zh/tutorials/labs/ascend-hami-dra)。 +两个 Pod 各请求 8192 MiB 显存加 50 算力、指向同一块卡时，两个 claim 落到同一个 device 上，各自带独立 shareID，调度器在分配第二个之前已把第一个的消费记账。算力记满后（两个 claim 占满 100/100 cores，显存尚有余量），下一个相同请求被调度器直接拒绝（claim 停在 pending），Pod 删除后 claim 自动回收、记账恢复。容器内的强制执行依旧是 HAMi-core：`libvnpu.so` 拦截超限的显存申请并转为容器内 OOM，容器视角的设备被虚拟化为申请的 8 GiB 而非整卡 21 GiB。完整过程见[实验 21](/zh/tutorials/labs/ascend-hami-dra)。 ## 设备支持的时间线，与“为什么是现在” diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md index 27948bffd..fa508e67e 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/ascend-hami-dra.md @@ -146,17 +146,7 @@ kubectl label ns dra-ascend-e2e hami.io/webhook=ignore cert-manager 是前置依赖；验证集群已有 v1.21.1。干净集群先安装 cert-manager。 -官方 Helm 仓库方式： - -```bash -helm repo add hami-dra https://project-hami.github.io/HAMi-DRA -helm repo update -helm search repo hami-dra # 确认 0.2.3 -helm install hami-dra hami-dra/hami-dra -n hami-system \ - --version 0.2.3 -f ascend-values.yaml -``` - -`ascend-values.yaml` 只开启昇腾转换： +先把 `ascend-values.yaml` 保存下来；它只开启昇腾转换： ```yaml deviceVendors: @@ -176,6 +166,16 @@ certs: enabled: true ``` +再从官方 Helm 仓库安装。干净集群上 `--create-namespace` 会创建 `hami-system`；步骤 2 会切换进 driver 仓库目录，请留意这个文件保存在哪里： + +```bash +helm repo add hami-dra https://project-hami.github.io/HAMi-DRA +helm repo update +helm search repo hami-dra # 确认 0.2.3 +helm install hami-dra hami-dra/hami-dra -n hami-system --create-namespace \ + --version 0.2.3 -f ascend-values.yaml +``` + 确认 webhook 正常： ```bash @@ -346,6 +346,8 @@ ResourceSlice device 与物理卡的对应关系（共享实验期间用 `npu-sm - 选卡：`hami.io/use-Ascend310P-uuid`，值从 ResourceSlice 的 `uuid` 抄写 - 选型：`hami.io/use-nputype`，与 `productName` 匹配（本实验未使用） +把下面的清单保存为 `pod.yaml`；后续步骤用 `sed` 从它派生 Pod B 和 Pod C： + ```yaml apiVersion: v1 kind: Pod @@ -501,7 +503,7 @@ status: ```bash kubectl exec -it ascend-share-a -n dra-ascend-e2e -- env | grep -E "NPU_|ASCEND_" -kubectl exec -it ascend-share-a -n dra-ascend-e2e -- ls /dev/davinci* /dev/davinci_manager /dev/devmm_svm +kubectl exec -it ascend-share-a -n dra-ascend-e2e -- sh -c 'ls /dev/davinci* /dev/davinci_manager /dev/devmm_svm' ``` ```text @@ -786,8 +788,8 @@ claim 的删除由 HAMi-DRA validating webhook 负责。如果 Pod 删除时 web ```bash kubectl delete pod ascend-share-a ascend-share-b ascend-share-c \ ascend-ctrl-a ascend-ctrl-b -n dra-ascend-e2e --ignore-not-found -kubectl get resourceclaim -A | grep ascend310p | awk '{print $1, $2}' | \ - xargs -n2 kubectl delete resourceclaim -n 2>/dev/null || true +kubectl get resourceclaim -n dra-ascend-e2e | grep ascend310p | awk '{print $1}' | \ + xargs -r kubectl delete resourceclaim -n dra-ascend-e2e 2>/dev/null || true kubectl delete ns dra-ascend-e2e --ignore-not-found helm uninstall ascend-dra-driver -n ascend-dra-driver diff --git a/tutorials/labs/ascend-hami-dra.md b/tutorials/labs/ascend-hami-dra.md index 862bce704..eb37224f2 100644 --- a/tutorials/labs/ascend-hami-dra.md +++ b/tutorials/labs/ascend-hami-dra.md @@ -146,17 +146,7 @@ The Pods in this lab additionally carry the label `hami.io/webhook: ignore`. On cert-manager is a dependency; the verification cluster already had v1.21.1. On a clean cluster install cert-manager first. -From the official Helm repository: - -```bash -helm repo add hami-dra https://project-hami.github.io/HAMi-DRA -helm repo update -helm search repo hami-dra # confirm 0.2.3 -helm install hami-dra hami-dra/hami-dra -n hami-system \ - --version 0.2.3 -f ascend-values.yaml -``` - -`ascend-values.yaml` enables only the Ascend conversion: +Save `ascend-values.yaml` first; it enables only the Ascend conversion: ```yaml deviceVendors: @@ -176,6 +166,16 @@ certs: enabled: true ``` +Then install from the official Helm repository. On a clean cluster `--create-namespace` creates `hami-system`; Step 2 changes into the driver repository directory, so keep this file's location in mind: + +```bash +helm repo add hami-dra https://project-hami.github.io/HAMi-DRA +helm repo update +helm search repo hami-dra # confirm 0.2.3 +helm install hami-dra hami-dra/hami-dra -n hami-system --create-namespace \ + --version 0.2.3 -f ascend-values.yaml +``` + Verify the webhook came up: ```bash @@ -346,6 +346,8 @@ The request syntax is unchanged HAMi syntax: - Card selection: `hami.io/use-Ascend310P-uuid`, the value copied from the ResourceSlice `uuid` - Model selection: `hami.io/use-nputype`, matched against `productName` (not used in this lab) +Save the following manifest as `pod.yaml`; later steps derive Pods B and C from it with `sed`: + ```yaml apiVersion: v1 kind: Pod @@ -501,7 +503,7 @@ Three layers, from the cluster down to the node. ```bash kubectl exec -it ascend-share-a -n dra-ascend-e2e -- env | grep -E "NPU_|ASCEND_" -kubectl exec -it ascend-share-a -n dra-ascend-e2e -- ls /dev/davinci* /dev/davinci_manager /dev/devmm_svm +kubectl exec -it ascend-share-a -n dra-ascend-e2e -- sh -c 'ls /dev/davinci* /dev/davinci_manager /dev/devmm_svm' ``` ```text @@ -786,8 +788,8 @@ Claim deletion is the HAMi-DRA validating webhook's job. If the webhook is down ```bash kubectl delete pod ascend-share-a ascend-share-b ascend-share-c \ ascend-ctrl-a ascend-ctrl-b -n dra-ascend-e2e --ignore-not-found -kubectl get resourceclaim -A | grep ascend310p | awk '{print $1, $2}' | \ - xargs -n2 kubectl delete resourceclaim -n 2>/dev/null || true +kubectl get resourceclaim -n dra-ascend-e2e | grep ascend310p | awk '{print $1}' | \ + xargs -r kubectl delete resourceclaim -n dra-ascend-e2e 2>/dev/null || true kubectl delete ns dra-ascend-e2e --ignore-not-found helm uninstall ascend-dra-driver -n ascend-dra-driver