From 64084321f8b3afbfc773ed88aa136d5244e3f78e Mon Sep 17 00:00:00 2001 From: mesutoezdil Date: Fri, 9 Oct 2026 11:11:19 +0200 Subject: [PATCH 1/4] docs: add the pages moved out of the HAMi repo docs folder Add the scheduler metrics and general technical review pages, and bring the roadmap device table in line with the supported devices list. Signed-off-by: mesutoezdil --- docs/contributor/general-technical-review.md | 229 ++++++++++++++++++ docs/contributor/roadmap.md | 29 +-- docs/developers/scheduler-metrics.md | 28 +++ .../current/contributor/roadmap.md | 29 +-- .../current/developers/scheduler-metrics.md | 29 +++ sidebars.js | 2 + 6 files changed, 318 insertions(+), 28 deletions(-) create mode 100644 docs/contributor/general-technical-review.md create mode 100644 docs/developers/scheduler-metrics.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/developers/scheduler-metrics.md diff --git a/docs/contributor/general-technical-review.md b/docs/contributor/general-technical-review.md new file mode 100644 index 000000000..8082b34ac --- /dev/null +++ b/docs/contributor/general-technical-review.md @@ -0,0 +1,229 @@ +--- +id: general-technical-review +title: General Technical Review +sidebar_label: General Technical Review +--- + +- **Project:** HAMi +- **Project Version:** v2.8.0 +- **Website:** https://project-hami.io +- **Date Updated:** 2026-04-13 +- **Template Version:** v1.0 +- **Description:** HAMi is a Kubernetes middleware for heterogeneous AI device sharing, isolation, and scheduling across GPUs/NPUs and other accelerators. + +--- + +## Day 0 - Planning Phase + +### Scope + +- **Roadmap and scope process:** Scope is defined through issues/PRs and maintainer review, with governance maintained in the HAMi community repo. The roadmap is maintained in GitHub issues (for example, per-version tracking such as https://github.com/Project-HAMi/HAMi/issues/1615). After each version is released, the next weekly community meeting discusses roadmap goals for upcoming releases in detail, and each goal is assigned an assignee on the issue. Community users can reply on those issues to add or request features. + - Design docs: https://project-hami.io/docs/developers/hami-core-design + - Governance: https://github.com/Project-HAMi/community/blob/main/governance.md +- **Target personas:** + - **Platform engineers:** Deploy HAMi as middleware on Kubernetes 1.23+ clusters with NVIDIA driver 440+ support for straightforward integration into existing platform products. GPU management and isolation are a common product differentiator that HAMi supports end to end. + - **MLOps teams:** Use GPU virtualization to reduce job wait times; rely on topology-aware scheduling and device-type selection to place training and inference workloads efficiently. + - **Individual AI workload owners:** Often operate a small GPU footprint; use sharing and multiplexing to improve utilization without changing application code. + - **IT operations teams:** Raise cluster utilization through GPU sharing; consolidate heterogeneous accelerators with integrated monitoring in a single cluster to lower maintenance cost. + +- **Primary use case:** Share and isolate accelerator resources (especially GPU memory/core) for multiple workloads on Kubernetes without application code changes. +- **Additional supported use cases:** Topology-aware scheduling, multiple scheduling policies (binpack/spread), dynamic MIG support for NVIDIA, and heterogeneous device support (e.g., NVIDIA, Cambricon, Hygon, Iluvatar, MetaX, Enflame, Ascend integration references). +- **Unsupported/non-goal use cases:** HAMi is not a model serving framework, training framework, or general-purpose cluster autoscaler. It focuses on accelerator virtualization/scheduling. +- **Organizations that benefit:** Public cloud, private cloud, enterprise AI platforms, telecom, finance, education, manufacturing, and internet companies operating shared accelerator clusters. +- **End-user research:** Formal, survey-style user research is still limited. The clearest public evidence of production adoption is **CNCF end-user case studies**, including multiple community-submitted stories that feature HAMi; the [HAMi-filtered case study list](https://www.cncf.io/case-studies/?_sft_lf-project=hami) is the updating catalog on cncf.io. Representative named end users include: + - [SF Technology](https://www.cncf.io/case-studies/sf-technology/) (technology arm of SF Express) + - [Ke Holdings Inc.](https://www.cncf.io/case-studies/ke-holdings-inc/) + - [NIO](https://www.cncf.io/case-studies/nio/) + +### Usability + +- **Interaction model:** Operators and workload owners interact with HAMi through Helm, Kubernetes Pod specs, component Services, and chart-driven scheduler/webhook settings. + - **Lifecycle:** Install, upgrade, and uninstall with Helm (see **Installation** below and the repository README quick start). + - **Workloads:** Request shared accelerators with Pod `requests`/`limits` plus HAMi annotations for slices, device selection, and scheduling policy. + - **Observability:** Inspect cluster-wide device and scheduling overview via the HAMi scheduler metrics HTTP endpoint; query the device-plugin vGPU monitor interface for live NVIDIA per-pod usage (listen ports are Helm-configurable; defaults are documented in the README and the [configure page](https://project-hami.io/docs/userguide/configure)). + +- **UX/UI:** The default experience is Kubernetes-native: `kubectl`, the Pod/Node APIs, and YAML/Helm values. For graphical administration and monitoring, deploy the optional **HAMi WebUI**; Grafana-oriented examples are documented separately. + - HAMi WebUI user guide: https://project-hami.io/docs/userguide/hami-webui-user-guide + - Dashboard examples: https://project-hami.io/docs/userguide/monitoring/device-allocation + +- **Production integration:** HAMi is in-cluster middleware: teams must already run a supported Kubernetes cluster. In production it extends the control plane with a mutating admission webhook, scheduler extender integration (default `kube-scheduler` or the chart-managed scheduler), device plugins, and CDI-capable runtimes where applicable. Additional batch/AI scheduler ecosystems are documented or tracked as follows: + - **Volcano:** Volcano-managed NVIDIA sharing via the Volcano vGPU device plugin (HAMi-core-based isolation); see https://project-hami.io/docs/installation/how-to-use-volcano-vgpu + - **Koordinator:** End-to-end GPU sharing with HAMi; see https://koordinator.sh/docs/user-manuals/device-scheduling-gpu-share-with-hami + - **KAI Scheduler:** GPU sharing isolation design and implementation are still evolving; see https://github.com/kai-scheduler/KAI-Scheduler/pull/60 + +### Design + +#### Design principles + +The following principles guide HAMi's architecture and feature trade-offs (see also the [design overview](https://project-hami.io/docs/developers/hami-core-design)): + +1. **Kubernetes-native control paths** + - Accelerators are requested, scheduled, and allocated through standard Kubernetes surfaces: Pod specs, the scheduler extender protocol, the device plugin gRPC API, and admission webhooks. + - HAMi avoids introducing proprietary control planes; cluster state remains in the Kubernetes API and node/device plugin reporting paths operators already trust. +2. **No application code changes** + - Workloads express sharing and isolation with resource requests, limits, and annotations. No rewrites of training, inference, or HPC code to "speak HAMi." + - This keeps adoption friction low for platform teams onboarding existing AI/ML jobs. +3. **Pluggable heterogeneous devices** + - Device-specific logic lives in device plugins and in-container control libraries (for example HAMi-core for NVIDIA, analogous components for other vendors). + - On the scheduling side, HAMi exposes a device API; each device implementation can extend this API to add device-specific scheduling optimizations. + - New hardware families integrate by extending these layers rather than forking the whole stack. +4. **Policy-driven scheduling** + - Placement uses filter/score semantics on the scheduler extender so administrators can steer binpack, spread, topology awareness, and vendor-specific constraints consistently. + - Decisions are recorded on Pods (via annotations), which enables scenario-specific scheduling behavior for different workload classes. +5. **Hard isolation on shared accelerators** + - Where the stack supports it, memory and compute caps are enforced inside containers, not only at admission time, so tenants cannot exceed declared slices on shared devices. +6. **Clear separation of responsibilities** + - Mutating admission validates and normalizes requests; the extender selects nodes; device plugins allocate and mount; in-container libraries enforce limits. + - Each stage has a narrow contract, and intermediate state is written to Pod/Node annotations, which helps operators localize issues quickly. + +#### Best practices + +Engineering and community practices that keep HAMi maintainable and safe to run in production: + +1. **Stay aligned with Kubernetes semantics** + - Follow upstream scheduler extender, device plugin, and admission webhook contracts; prefer deprecating non-standard behavior with clear release notes rather than silent divergence. +2. **Automate verification** + - Gate changes with CI (lint, unit tests, chart packaging, and workflows that exercise realistic scheduling and device paths) to catch regressions before merge. +3. **Test in realistic environments** + - Validate features against representative accelerator hardware, drivers, and runtime combinations; document supported matrices and known limitations per vendor. +4. **Community-driven prioritization** + - Use public issues, roadmap discussions, and community meetings so roadmap items reflect operator pain and contributor capacity (see **Roadmap and scope process** under Scope). +5. **Documentation and operability first** + - Ship user-facing docs, examples, and Helm values commentary alongside features so platforms can install, tune, and observe HAMi without reading the implementation. + +#### Architecture requirements + +Core components include MutatingWebhook, Scheduler Extender, Device Plugins, and in-container control libraries. + +- Architecture and flow: https://project-hami.io/docs/developers/hami-core-design + +#### Environment differences + +- PoC/dev/test/prod: the core deployment model is the same (GPU-enabled Kubernetes with HAMi); production environments typically apply stricter node labeling, webhook TLS hardening, scheduler/device policy tuning, and monitoring integration. + +#### Service dependencies in cluster + +Kubernetes API server, kube-scheduler/Volcano scheduling integration, kubelet device plugin APIs, and (optionally) cert-manager for webhook certificates. + +#### IAM model + +Uses Kubernetes RBAC for component service accounts and least-required API operations for webhook patch jobs and scheduler operations. + +#### Sovereignty/compliance + +Data plane is cluster-local; no mandatory external telemetry service. Regional/organizational compliance posture depends on deployment choices. + +#### HA requirements + +Scheduler supports leader election and configurable replicas. + +- Workload side: since HAMi v2.5, already-running tasks are designed to remain stable and are not expected to fail solely due to cluster-side events such as HAMi upgrades/uninstallations or transient Kubernetes/HAMi control-plane faults. +- Scheduling side: since HAMi v2.8, multi-replica scheduler deployment with leader election is supported to provide high availability for scheduling decisions. + +#### Resource requirements (CPU/memory/network) + +Configurable per component via Helm values. The chart leaves `resources` unset by default, so production clusters should set explicit requests/limits. The following estimates are practical planning baselines for HAMi v2.8.0 on Kubernetes 1.20+, with NVIDIA sharing enabled and normal scheduling churn. + +- **Assumptions for estimates:** 1 scheduler replica (`kube-scheduler` + HAMi extender), 1 device-plugin DaemonSet pod per GPU node (`device-plugin` + `vgpu-monitor`), Prometheus scraping every 15-30s, and no unusual pod-creation spikes. + +| Component / scope | CPU (recommended request to typical peak) | Memory (recommended request to typical peak) | Network (control/observability plane) | +| --- | --- | --- | --- | +| Scheduler pod (per replica) | 300m-800m | 512Mi-2Gi | 0.2-1 Mbps typical, 2-5 Mbps burst during heavy pod churn | +| Device-plugin pod (per GPU node) | 150m-600m | 256Mi-1Gi | 0.05-0.2 Mbps per node (telemetry and control signaling) | +| Metrics scraping overhead (per target, 15s interval) | N/A | N/A | 50-300 KiB/scrape, about 0.03-0.16 Mbps | +| Cluster total (N GPU nodes) | (0.3 to 0.8) + N * (0.15 to 0.6) cores | (0.5 to 2) + N * (0.25 to 1) GiB | Approximately linear with N based on per-node telemetry plus scheduler control traffic | +| Example footprint (N=20 GPU nodes) | 3.3-12.8 cores | 5.5-22 GiB | 1.2-5 Mbps typical aggregate control/metrics traffic, with higher short bursts during large scheduling events | + +#### Storage requirements + +HAMi has no mandatory external database. Storage use is mainly image layers, logs/metrics buffers, and hostPath-mounted runtime paths used by scheduler/device-plugin components. The following are practical sizing estimates for production planning. + +- **Assumptions for estimates:** default image retention policy, standard Kubernetes/container runtime logging, and mixed training/inference workloads with normal pod churn. + +| Component / scope | Estimated storage footprint | Notes | +| --- | --- | --- | +| Scheduler pod (per replica) | 0.2-0.8 GiB node-local ephemeral storage | Mostly image layers, logs, and temporary runtime files | +| Device-plugin + monitor pod (per GPU node) | 0.3-1.2 GiB node-local ephemeral storage | Includes plugin/monitor image layers, logs, and runtime temp/cache paths | +| Host runtime paths used by HAMi on GPU nodes | 0.2-1 GiB recommended free space on top of pod ephemeral usage | Covers `/var/lib/kubelet/device-plugins`, `/usr/local/vgpu`, `/var/run/cdi`, and transient files under `/tmp` | +| Cluster total (N GPU nodes, 1 scheduler replica) | approximately `(0.2 to 0.8) + N * (0.5 to 2.2)` GiB | Combines scheduler ephemeral + per-GPU-node pod + host runtime reserve | +| Example footprint (N=20 GPU nodes) | approximately 10.2-44.8 GiB cluster-wide reserved capacity | Planning envelope for stable operations and upgrade headroom | + +#### API design + +- Uses Kubernetes native APIs (Pods, Nodes, annotations, admission webhooks, device plugin gRPC). +- Defaults and optional configurations are exposed in Helm chart values and config docs. +- No custom CRD is required for core workflows. +- Device-slice resource names are exposed as Kubernetes resources, so users can request and limit slices directly in `resources.limits` / `resources.requests`. +- GPU scheduling policies and advanced behaviors are expressed through Pod annotations. +- HAMi runtime behavior can be configured through scheduler and device-plugin ConfigMaps, with Helm values as the primary entry point. + +#### Release process + +Semantic versioning, tagged releases, release branches, automated image/chart/release-note workflows, with documented manual verification steps. + +- Release process doc: https://github.com/Project-HAMi/HAMi/blob/master/.github/release-process.md + +### Installation + +- **Installation and validation:** Helm-based install with node labeling and chart value customization. + - Quick start: https://project-hami.io/docs/installation/online-installation + - Example deployment commands: + +```bash +# 1) Label GPU nodes for HAMi management +kubectl label nodes gpu=on + +# 2) Add HAMi Helm repository +helm repo add hami-charts https://project-hami.github.io/HAMi/ +helm repo update + +# 3) Install HAMi into kube-system +helm install hami hami-charts/hami -n kube-system + +# 4) (Optional) Customize values during install +# helm install hami hami-charts/hami -n kube-system -f values.yaml + +# 5) Verify core components are running +kubectl get pods -n kube-system | grep -E "hami-scheduler|hami-device-plugin" +``` + +### Security + +See separate [document](https://github.com/cncf/toc/blob/main/projects/hami/security-assesment/self-assessment.md) + +--- + +## Day 1 - Installation and Deployment Phase + +### Project Installation and Configuration + +- Installation uses the Helm chart, including configuration for scheduler policy, resource naming, webhook selectors, TLS strategy, and device-plugin behavior. + - Installation guide: https://project-hami.io/docs/installation/online-installation +- Operational behavior is configurable through Helm values and ConfigMaps (for example scheduler and device policy config maps). + - Configuration guide: https://project-hami.io/docs/userguide/configure + +### Project Enablement and Rollback + +- **Enable/disable in live cluster:** Deploy/upgrade/uninstall via Helm. Enabling does not require control-plane downtime; workloads may experience scheduling behavior changes when webhook/scheduler integration changes. +- **Behavior changes when enabled:** + - Pod mutation for recognized device requests. + - Scheduling for device-sharing workloads routed through HAMi logic and policy. + - Device plugin allocation/mount/env handling for runtime enforcement. +- **Testing enable/disable:** Deployment and uninstall paths are exercised through e2e workflows; PRs are expected to pass required e2e checks before merge. +- **Resource cleanup:** Helm uninstall removes chart-managed resources; no core CRDs are introduced by default path. + +### Rollout, Upgrade and Rollback Planning + +- **Compatibility cadence:** Kubernetes and related dependencies are tracked through CI and automated dependency updates (including Dependabot). Version evolution is governed by the documented release process and release workflows. +- **Rollback procedures:** Use `helm rollback` for in-place rollback, or uninstall and redeploy with a previous chart version when a full re-provisioning path is preferred. +- **How rollout/rollback can fail:** + - Rolling back images without rolling back the matching chart/configuration revision. + - Failing to roll back customized values and policy ConfigMaps together with workloads/images. + - Performing multi-version jump rollbacks (for example two or more minor versions) without staged validation. +- **Impact to running workloads:** Since HAMi v2.5, already-running pods are generally expected to remain unaffected by control-plane rollout actions; new scheduling/allocation decisions may still change during upgrades/rollbacks. +- **Rollback indicators:** Primary indicators include admission/scheduling errors, abnormal pending-pod growth, device plugin registration issues, and policy mismatch symptoms. Operationally, the project recommends staying on recent supported releases whenever possible. +- **Upgrade path testing:** Unit/e2e coverage exists; explicit long-chain upgrade->downgrade->upgrade matrices are still evolving and should be expanded for incubation. +- **Deprecation communication:** Deprecations are communicated in documentation and community meetings, with explicit in-doc annotations. A typical transition window keeps old and new APIs available for one subsequent release, followed by old API removal after the next release cycle. +- **Alpha/beta capabilities:** Exposed by configuration flags and values; users opt in through chart values and documented settings. + +--- diff --git a/docs/contributor/roadmap.md b/docs/contributor/roadmap.md index 3e0025a1b..8ab135753 100644 --- a/docs/contributor/roadmap.md +++ b/docs/contributor/roadmap.md @@ -6,20 +6,21 @@ sidebar_label: Roadmap ## Device Support Matrix -| Device Type | Manufacturer | Models | Memory Isolation | Core Isolation | Multi-Card | -| ----------- | ------------- | ----------------- | :--------------: | :------------: | :--------: | -| GPU | NVIDIA | All | Yes | Yes | Yes | -| MLU | Cambricon | 370, 590 | Yes | Yes | No | -| GCU | Enflame | S60 | Yes | Yes | No | -| HCU | Hygon | All | Yes | Yes | No | -| NPU | Huawei Ascend | 310P, 910B, 910B3 | Yes | Yes | No | -| GPU | Iluvatar | All | Yes | Yes | No | -| DPU | Teco | Checking | In progress | In progress | No | -| GPU | Moore Threads | MTT S4000 | Yes | Yes | No | -| GPU | Birentech | Model 110 | In progress | In progress | No | -| GPU | MetaX | MXC500 | Yes | Yes | No | -| XPU | Kunlunxin | P800 | Yes | Yes | No | -| GPU | Vastai | VA16 | Yes | Yes | No | +| Device Type | Manufacturer | Models | Memory Isolation | Core Isolation | Multi-Card | +| --- | --- | --- | :-: | :-: | :-: | +| GPU | NVIDIA | All | Yes | Yes | Yes | +| MLU | Cambricon | 370, 590 | Yes | Yes | No | +| GCU | Enflame | S60 | Yes | Yes | No | +| HCU | Hygon | All | Yes | Yes | No | +| NPU | Huawei Ascend | 310P, 910B, 910B3 | Yes | Yes | No | +| GPU | Iluvatar | All | Yes | Yes | No | +| DPU | Teco | Checking | In progress | In progress | No | +| GPU | Moore Threads | MTT S4000 | Yes | Yes | No | +| GPU | Birentech | Biren166M | Yes | Yes | No | +| GPU | MetaX | MXC500 | Yes | Yes | No | +| XPU | Kunlunxin | P800 | Yes | Yes | No | +| GPU | Vastai | VA16 | Yes | Yes | No | +| Neuron | AWS | Inferentia, Trainium | No | Yes | Yes | ## Planned Features diff --git a/docs/developers/scheduler-metrics.md b/docs/developers/scheduler-metrics.md new file mode 100644 index 000000000..ff76d7abd --- /dev/null +++ b/docs/developers/scheduler-metrics.md @@ -0,0 +1,28 @@ +--- +id: scheduler-metrics +title: Scheduler Metrics +sidebar_label: Scheduler Metrics +--- + +HAMi exposes scheduler metrics on the configured scheduler metrics endpoint. The allocation outcome metrics use only bounded label values: + +| Metric | Meaning | +| --- | --- | +| `hami_scheduler_allocations_total` | Successful device reservations recorded by the filter phase. | +| `hami_scheduler_allocation_failures_total` | Filter and bind allocation failures. | +| `hami_scheduler_bind_rollbacks_total` | Bind failures that release a reservation. | + +The labels are `phase`, `device_type`, and `failure_reason`. They never contain pod, namespace, node, UUID, or request identifiers. `failure_reason` uses this complete, fixed set: + +| Value | Meaning | +| ------------------ | ------------------------------------------------------------------ | +| `none` | The operation succeeded. | +| `no_fit` | No candidate node could satisfy the device request. | +| `lookup` | A required pod or node could not be read from the scheduler cache. | +| `identity` | The bind request did not match the current pod identity or target. | +| `lock` | Device reservation locking failed. | +| `annotation_patch` | A required pod annotation update failed. | +| `bind` | Another bind operation failed. | +| `internal` | Internal usage, scoring, discovery, or cleanup failed. | + +A successful filter reservation uses `failure_reason="none"`; bind failures are reported by the failure and rollback counters. diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor/roadmap.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor/roadmap.md index da9fb380e..7b3a445d2 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor/roadmap.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor/roadmap.md @@ -7,20 +7,21 @@ sidebar_label: 路线图 ## 设备支持矩阵 -| 设备类型 | 制造商 | 型号 | 显存隔离 | 核心隔离 | 多卡支持 | -| -------- | ------------- | ----------------- | :------: | :------: | :------: | -| GPU | NVIDIA | 全系列 | 是 | 是 | 是 | -| MLU | Cambricon | 370、590 | 是 | 是 | 否 | -| GCU | Enflame | S60 | 是 | 是 | 否 | -| HCU | Hygon | 全系列 | 是 | 是 | 否 | -| NPU | Huawei Ascend | 310P、910B、910B3 | 是 | 是 | 否 | -| GPU | Iluvatar | 全系列 | 是 | 是 | 否 | -| DPU | Teco | 检查中 | 进行中 | 进行中 | 否 | -| GPU | Moore Threads | MTT S4000 | 是 | 是 | 否 | -| GPU | Birentech | Model 110 | 进行中 | 进行中 | 否 | -| GPU | MetaX | MXC500 | 是 | 是 | 否 | -| XPU | Kunlunxin | P800 | 是 | 是 | 否 | -| GPU | Vastai | VA16 | 是 | 是 | 否 | +| 设备类型 | 制造商 | 型号 | 显存隔离 | 核心隔离 | 多卡支持 | +| -------- | ------------- | -------------------- | :------: | :------: | :------: | +| GPU | NVIDIA | 全系列 | 是 | 是 | 是 | +| MLU | Cambricon | 370、590 | 是 | 是 | 否 | +| GCU | Enflame | S60 | 是 | 是 | 否 | +| HCU | Hygon | 全系列 | 是 | 是 | 否 | +| NPU | Huawei Ascend | 310P、910B、910B3 | 是 | 是 | 否 | +| GPU | Iluvatar | 全系列 | 是 | 是 | 否 | +| DPU | Teco | 检查中 | 进行中 | 进行中 | 否 | +| GPU | Moore Threads | MTT S4000 | 是 | 是 | 否 | +| GPU | Birentech | Biren166M | 是 | 是 | 否 | +| GPU | MetaX | MXC500 | 是 | 是 | 否 | +| XPU | Kunlunxin | P800 | 是 | 是 | 否 | +| GPU | Vastai | VA16 | 是 | 是 | 否 | +| Neuron | AWS | Inferentia, Trainium | 否 | 是 | 是 | ## 计划功能 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/developers/scheduler-metrics.md b/i18n/zh/docusaurus-plugin-content-docs/current/developers/scheduler-metrics.md new file mode 100644 index 000000000..a5af2783e --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/developers/scheduler-metrics.md @@ -0,0 +1,29 @@ +--- +id: scheduler-metrics +title: 调度器指标 +translated: true +sidebar_label: 调度器指标 +--- + +HAMi 在已配置的调度器指标端点上暴露调度器指标。分配结果类指标只使用有限范围的标签值： + +| 指标 | 含义 | +| ------------------------------------------ | -------------------------------- | +| `hami_scheduler_allocations_total` | 过滤阶段记录的成功设备预留次数。 | +| `hami_scheduler_allocation_failures_total` | 过滤和绑定阶段的分配失败次数。 | +| `hami_scheduler_bind_rollbacks_total` | 释放预留的绑定失败次数。 | + +标签为 `phase`、`device_type` 和 `failure_reason`，不会包含 Pod、命名空间、节点、UUID 或请求标识。`failure_reason` 使用以下完整且固定的取值： + +| 取值 | 含义 | +| ------------------ | ----------------------------------------- | +| `none` | 操作成功。 | +| `no_fit` | 没有候选节点能满足设备请求。 | +| `lookup` | 无法从调度器缓存中读取所需的 Pod 或节点。 | +| `identity` | 绑定请求与当前 Pod 身份或目标不匹配。 | +| `lock` | 设备预留加锁失败。 | +| `annotation_patch` | 必需的 Pod 注解更新失败。 | +| `bind` | 其他绑定操作失败。 | +| `internal` | 内部用量、打分、发现或清理失败。 | + +成功的过滤预留使用 `failure_reason="none"`；绑定失败由失败计数器和回滚计数器报告。 diff --git a/sidebars.js b/sidebars.js index 75ada7ff6..323d6e6a6 100644 --- a/sidebars.js +++ b/sidebars.js @@ -173,6 +173,7 @@ module.exports = { "developers/kunlunxin-topology", "developers/profiling-scheduler", "developers/scheduler-event-log", + "developers/scheduler-metrics", "developers/gpu-utilization-metrics", "developers/hami-vnpu-core-integration", "developers/hami-webui-development-guide", @@ -199,6 +200,7 @@ module.exports = { "contributor/governance", "contributor/ladder", "contributor/roadmap", + "contributor/general-technical-review", "contributor/e2e-testing", ], }, From 10c5e6612376336a4b5672b8b0e866ce20c4ac37 Mon Sep 17 00:00:00 2001 From: mesutoezdil Date: Fri, 9 Oct 2026 11:20:01 +0200 Subject: [PATCH 2/4] docs: align the roadmap with the supported devices list and narrow the upgrade claims Signed-off-by: mesutoezdil --- docs/contributor/general-technical-review.md | 6 +++--- docs/contributor/roadmap.md | 4 ++-- .../current/contributor/roadmap.md | 2 +- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/contributor/general-technical-review.md b/docs/contributor/general-technical-review.md index 8082b34ac..5b7695c3f 100644 --- a/docs/contributor/general-technical-review.md +++ b/docs/contributor/general-technical-review.md @@ -30,7 +30,7 @@ sidebar_label: General Technical Review - **Additional supported use cases:** Topology-aware scheduling, multiple scheduling policies (binpack/spread), dynamic MIG support for NVIDIA, and heterogeneous device support (e.g., NVIDIA, Cambricon, Hygon, Iluvatar, MetaX, Enflame, Ascend integration references). - **Unsupported/non-goal use cases:** HAMi is not a model serving framework, training framework, or general-purpose cluster autoscaler. It focuses on accelerator virtualization/scheduling. - **Organizations that benefit:** Public cloud, private cloud, enterprise AI platforms, telecom, finance, education, manufacturing, and internet companies operating shared accelerator clusters. -- **End-user research:** Formal, survey-style user research is still limited. The clearest public evidence of production adoption is **CNCF end-user case studies**, including multiple community-submitted stories that feature HAMi; the [HAMi-filtered case study list](https://www.cncf.io/case-studies/?_sft_lf-project=hami) is the updating catalog on cncf.io. Representative named end users include: +- **End-user research:** Formal, survey-style user research is still limited. The clearest public evidence of production adoption is **CNCF end-user case studies**, including multiple community-submitted stories that feature HAMi; the [HAMi-filtered case study list](https://www.cncf.io/case-studies/?_sft_lf-project=hami) is the up-to-date catalog on cncf.io. Representative named end users include: - [SF Technology](https://www.cncf.io/case-studies/sf-technology/) (technology arm of SF Express) - [Ke Holdings Inc.](https://www.cncf.io/case-studies/ke-holdings-inc/) - [NIO](https://www.cncf.io/case-studies/nio/) @@ -117,7 +117,7 @@ Data plane is cluster-local; no mandatory external telemetry service. Regional/o Scheduler supports leader election and configurable replicas. -- Workload side: since HAMi v2.5, already-running tasks are designed to remain stable and are not expected to fail solely due to cluster-side events such as HAMi upgrades/uninstallations or transient Kubernetes/HAMi control-plane faults. +- Workload side: HAMi v2.5 addressed running-task crashes during reinstall. This is not a guarantee for upgrades or transient control-plane faults; stop or reschedule GPU workloads before upgrading. - Scheduling side: since HAMi v2.8, multi-replica scheduler deployment with leader election is supported to provide high availability for scheduling decisions. #### Resource requirements (CPU/memory/network) @@ -220,7 +220,7 @@ See separate [document](https://github.com/cncf/toc/blob/main/projects/hami/secu - Rolling back images without rolling back the matching chart/configuration revision. - Failing to roll back customized values and policy ConfigMaps together with workloads/images. - Performing multi-version jump rollbacks (for example two or more minor versions) without staged validation. -- **Impact to running workloads:** Since HAMi v2.5, already-running pods are generally expected to remain unaffected by control-plane rollout actions; new scheduling/allocation decisions may still change during upgrades/rollbacks. +- **Impact to running workloads:** HAMi v2.5 addressed running-task crashes during reinstall, but upgrades and rollbacks can still affect active workloads; stop or reschedule GPU workloads before upgrading. New scheduling/allocation decisions may also change. - **Rollback indicators:** Primary indicators include admission/scheduling errors, abnormal pending-pod growth, device plugin registration issues, and policy mismatch symptoms. Operationally, the project recommends staying on recent supported releases whenever possible. - **Upgrade path testing:** Unit/e2e coverage exists; explicit long-chain upgrade->downgrade->upgrade matrices are still evolving and should be expanded for incubation. - **Deprecation communication:** Deprecations are communicated in documentation and community meetings, with explicit in-doc annotations. A typical transition window keeps old and new APIs available for one subsequent release, followed by old API removal after the next release cycle. diff --git a/docs/contributor/roadmap.md b/docs/contributor/roadmap.md index 8ab135753..bc01753bb 100644 --- a/docs/contributor/roadmap.md +++ b/docs/contributor/roadmap.md @@ -16,11 +16,11 @@ sidebar_label: Roadmap | GPU | Iluvatar | All | Yes | Yes | No | | DPU | Teco | Checking | In progress | In progress | No | | GPU | Moore Threads | MTT S4000 | Yes | Yes | No | -| GPU | Birentech | Biren166M | Yes | Yes | No | +| GPU | Birentech | Biren166M | No | No | No | | GPU | MetaX | MXC500 | Yes | Yes | No | | XPU | Kunlunxin | P800 | Yes | Yes | No | | GPU | Vastai | VA16 | Yes | Yes | No | -| Neuron | AWS | Inferentia, Trainium | No | Yes | Yes | +| Neuron | AWS | Inferentia, Trainium | No | Yes | No | ## Planned Features diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor/roadmap.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor/roadmap.md index 7b3a445d2..75db88cce 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor/roadmap.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor/roadmap.md @@ -21,7 +21,7 @@ sidebar_label: 路线图 | GPU | MetaX | MXC500 | 是 | 是 | 否 | | XPU | Kunlunxin | P800 | 是 | 是 | 否 | | GPU | Vastai | VA16 | 是 | 是 | 否 | -| Neuron | AWS | Inferentia, Trainium | 否 | 是 | 是 | +| Neuron | AWS | Inferentia, Trainium | 否 | 是 | 否 | ## 计划功能 From 706f87f54f04203eff22a0e9f68f325b38e07b21 Mon Sep 17 00:00:00 2001 From: mesutoezdil Date: Sat, 10 Oct 2026 09:20:33 +0200 Subject: [PATCH 3/4] docs: drop horizontal rules and add zh translation of general technical review Signed-off-by: mesutoezdil --- docs/contributor/general-technical-review.md | 6 - .../contributor/general-technical-review.md | 224 ++++++++++++++++++ 2 files changed, 224 insertions(+), 6 deletions(-) create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/contributor/general-technical-review.md diff --git a/docs/contributor/general-technical-review.md b/docs/contributor/general-technical-review.md index 5b7695c3f..52860cb40 100644 --- a/docs/contributor/general-technical-review.md +++ b/docs/contributor/general-technical-review.md @@ -11,8 +11,6 @@ sidebar_label: General Technical Review - **Template Version:** v1.0 - **Description:** HAMi is a Kubernetes middleware for heterogeneous AI device sharing, isolation, and scheduling across GPUs/NPUs and other accelerators. ---- - ## Day 0 - Planning Phase ### Scope @@ -191,8 +189,6 @@ kubectl get pods -n kube-system | grep -E "hami-scheduler|hami-device-plugin" See separate [document](https://github.com/cncf/toc/blob/main/projects/hami/security-assesment/self-assessment.md) ---- - ## Day 1 - Installation and Deployment Phase ### Project Installation and Configuration @@ -225,5 +221,3 @@ See separate [document](https://github.com/cncf/toc/blob/main/projects/hami/secu - **Upgrade path testing:** Unit/e2e coverage exists; explicit long-chain upgrade->downgrade->upgrade matrices are still evolving and should be expanded for incubation. - **Deprecation communication:** Deprecations are communicated in documentation and community meetings, with explicit in-doc annotations. A typical transition window keeps old and new APIs available for one subsequent release, followed by old API removal after the next release cycle. - **Alpha/beta capabilities:** Exposed by configuration flags and values; users opt in through chart values and documented settings. - ---- diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor/general-technical-review.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor/general-technical-review.md new file mode 100644 index 000000000..bb7c85fcf --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor/general-technical-review.md @@ -0,0 +1,224 @@ +--- +id: general-technical-review +title: 通用技术评审 +translated: true +sidebar_label: 通用技术评审 +--- + +- **项目：** HAMi +- **项目版本：** v2.8.0 +- **网站：** https://project-hami.io +- **更新日期：** 2026-04-13 +- **模板版本：** v1.0 +- **描述：** HAMi 是一个 Kubernetes 中间件，用于在 GPU/NPU 及其他加速器之间实现异构 AI 设备的共享、隔离和调度。 + +## 第 0 天 - 规划阶段 + +### 范围 + +- **路线图与范围流程：** 范围通过 issue/PR 和维护者评审来确定，治理文档维护在 HAMi 社区仓库中。路线图维护在 GitHub issue 中（例如按版本跟踪的 https://github.com/Project-HAMi/HAMi/issues/1615）。每个版本发布后，下一次社区周会会详细讨论后续版本的路线图目标，并在 issue 上为每个目标指定负责人。社区用户可以在这些 issue 下回复，补充或提出功能需求。 + - 设计文档：https://project-hami.io/docs/developers/hami-core-design + - 治理：https://github.com/Project-HAMi/community/blob/main/governance.md +- **目标用户画像：** + - **平台工程师：** 在 Kubernetes 1.23+ 集群上将 HAMi 作为中间件部署，支持 NVIDIA 驱动 440+，便于集成到现有平台产品中。GPU 管理与隔离是常见的产品差异化能力，HAMi 提供端到端支持。 + - **MLOps 团队：** 使用 GPU 虚拟化缩短作业等待时间；借助拓扑感知调度和设备类型选择，高效地放置训练与推理工作负载。 + - **个人 AI 工作负载所有者：** 通常只运行少量 GPU；利用共享与复用提升利用率，无需修改应用代码。 + - **IT 运维团队：** 通过 GPU 共享提升集群利用率；在单个集群中整合异构加速器并集成监控，降低维护成本。 + +- **主要使用场景：** 在 Kubernetes 上为多个工作负载共享和隔离加速器资源（尤其是 GPU 显存/算力），无需修改应用代码。 +- **其他支持的使用场景：** 拓扑感知调度、多种调度策略（binpack/spread）、NVIDIA 动态 MIG 支持，以及异构设备支持（例如 NVIDIA、寒武纪、海光、天数智芯、沐曦、燧原，以及昇腾集成参考）。 +- **不支持/非目标场景：** HAMi 不是模型服务框架、训练框架或通用集群自动扩缩容器，它专注于加速器的虚拟化与调度。 +- **受益的组织：** 运营共享加速器集群的公有云、私有云、企业 AI 平台、电信、金融、教育、制造和互联网公司。 +- **终端用户调研：** 正式的问卷式用户调研仍然有限。生产环境采用情况最清晰的公开证据是 **CNCF 终端用户案例**，其中包括多个由社区提交、介绍 HAMi 的案例；[按 HAMi 筛选的案例列表](https://www.cncf.io/case-studies/?_sft_lf-project=hami)是 cncf.io 上持续更新的目录。具有代表性的终端用户包括： + - [SF Technology](https://www.cncf.io/case-studies/sf-technology/)（顺丰科技） + - [Ke Holdings Inc.](https://www.cncf.io/case-studies/ke-holdings-inc/)（贝壳） + - [NIO](https://www.cncf.io/case-studies/nio/)（蔚来） + +### 可用性 + +- **交互模型：** 运维人员和工作负载所有者通过 Helm、Kubernetes Pod 规格、组件 Service 以及由 chart 驱动的调度器/webhook 设置与 HAMi 交互。 + - **生命周期：** 使用 Helm 安装、升级和卸载（见下文**安装**和仓库 README 快速入门）。 + - **工作负载：** 通过 Pod 的 `requests`/`limits` 加上 HAMi 注解来申请共享加速器，并设置切片、设备选择和调度策略。 + - **可观测性：** 通过 HAMi 调度器的指标 HTTP 端点查看集群范围的设备与调度概览；通过 device-plugin 的 vGPU monitor 接口查询 NVIDIA 各 Pod 的实时使用情况（监听端口可通过 Helm 配置，默认值见 README 和[配置页面](https://project-hami.io/docs/userguide/configure)）。 + +- **用户体验/界面：** 默认体验是 Kubernetes 原生的：`kubectl`、Pod/Node API 以及 YAML/Helm values。如需图形化管理和监控，可部署可选的 **HAMi WebUI**；面向 Grafana 的示例另有文档说明。 + - HAMi WebUI 用户指南：https://project-hami.io/docs/userguide/hami-webui-user-guide + - 仪表盘示例：https://project-hami.io/docs/userguide/monitoring/device-allocation + +- **生产环境集成：** HAMi 是集群内中间件：团队必须已经运行受支持的 Kubernetes 集群。在生产环境中，它通过 mutating admission webhook、scheduler extender 集成（默认的 `kube-scheduler` 或由 chart 管理的调度器）、device plugin 以及适用场景下支持 CDI 的运行时来扩展控制平面。其他批处理/AI 调度生态的文档或跟踪情况如下： + - **Volcano：** 通过 Volcano vGPU device plugin（基于 HAMi-core 的隔离）实现由 Volcano 管理的 NVIDIA 共享；参见 https://project-hami.io/docs/installation/how-to-use-volcano-vgpu + - **Koordinator：** 与 HAMi 结合的端到端 GPU 共享；参见 https://koordinator.sh/docs/user-manuals/device-scheduling-gpu-share-with-hami + - **KAI Scheduler：** GPU 共享隔离的设计与实现仍在演进中；参见 https://github.com/kai-scheduler/KAI-Scheduler/pull/60 + +### 设计 + +#### 设计原则 + +以下原则指导 HAMi 的架构和功能取舍（另见[设计概览](https://project-hami.io/docs/developers/hami-core-design)）： + +1. **Kubernetes 原生的控制路径** + - 加速器通过标准的 Kubernetes 接口来申请、调度和分配：Pod 规格、scheduler extender 协议、device plugin gRPC API 以及准入 webhook。 + - HAMi 避免引入私有控制平面；集群状态保留在 Kubernetes API 以及运维人员已经信任的节点/device plugin 上报路径中。 +2. **无需修改应用代码** + - 工作负载通过资源请求、限制和注解来表达共享与隔离需求，无需为了“适配 HAMi”而重写训练、推理或 HPC 代码。 + - 这降低了平台团队接入现有 AI/ML 作业的阻力。 +3. **可插拔的异构设备** + - 设备相关逻辑位于 device plugin 和容器内控制库中（例如 NVIDIA 使用 HAMi-core，其他厂商使用类似组件）。 + - 在调度侧，HAMi 暴露设备 API；每个设备实现都可以扩展该 API，加入设备特定的调度优化。 + - 新硬件通过扩展这些层来接入，而不是复制整个技术栈。 +4. **策略驱动的调度** + - 放置决策在 scheduler extender 上采用 filter/score 语义，管理员可以一致地引导 binpack、spread、拓扑感知和厂商特定的约束。 + - 决策会通过注解记录在 Pod 上，从而支持针对不同工作负载类别的场景化调度行为。 +5. **共享加速器上的硬隔离** + - 在技术栈支持的情况下，显存和算力上限在容器内强制执行，而不仅仅在准入阶段检查，因此租户无法超出在共享设备上声明的切片。 +6. **清晰的职责分离** + - Mutating admission 负责校验并规范化请求；extender 负责选择节点；device plugin 负责分配和挂载；容器内库负责强制执行限制。 + - 每个阶段的约定都很窄，中间状态写入 Pod/Node 注解，帮助运维人员快速定位问题。 + +#### 最佳实践 + +让 HAMi 易于维护、可安全用于生产环境的工程与社区实践： + +1. **与 Kubernetes 语义保持一致** + - 遵循上游 scheduler extender、device plugin 和准入 webhook 的约定；对非标准行为，优先通过明确的发布说明进行弃用，而不是悄悄偏离。 +2. **自动化验证** + - 通过 CI 把关变更（lint、单元测试、chart 打包，以及覆盖真实调度和设备路径的工作流），在合并前发现回归。 +3. **在真实环境中测试** + - 针对有代表性的加速器硬件、驱动和运行时组合验证功能；按厂商记录受支持的矩阵和已知限制。 +4. **社区驱动的优先级** + - 通过公开的 issue、路线图讨论和社区会议，让路线图条目反映运维人员的痛点和贡献者的投入能力（参见“范围”下的**路线图与范围流程**）。 +5. **文档与可运维性优先** + - 随功能一起提供面向用户的文档、示例和 Helm values 注释，让平台团队无需阅读实现就能安装、调优和观测 HAMi。 + +#### 架构要求 + +核心组件包括 MutatingWebhook、Scheduler Extender、Device Plugin 和容器内控制库。 + +- 架构与流程：https://project-hami.io/docs/developers/hami-core-design + +#### 环境差异 + +- PoC/开发/测试/生产：核心部署模型相同（启用 GPU 的 Kubernetes 加 HAMi）；生产环境通常会采用更严格的节点标签、webhook TLS 加固、调度器/设备策略调优以及监控集成。 + +#### 集群内服务依赖 + +Kubernetes API server、kube-scheduler/Volcano 调度集成、kubelet device plugin API，以及（可选的）用于 webhook 证书的 cert-manager。 + +#### IAM 模型 + +组件的 service account 使用 Kubernetes RBAC，并且 webhook patch 任务和调度器操作只使用必需的最小 API 权限。 + +#### 主权/合规 + +数据平面限于集群本地，没有强制使用的外部遥测服务。地区/组织层面的合规情况取决于部署选择。 + +#### 高可用要求 + +调度器支持 leader election 和可配置的副本数。 + +- 工作负载侧：HAMi v2.5 解决了重新安装期间运行中任务崩溃的问题。这不能保证升级或控制平面的瞬时故障不受影响；升级前请停止或重新调度 GPU 工作负载。 +- 调度侧：自 HAMi v2.8 起，支持带 leader election 的多副本调度器部署，为调度决策提供高可用。 + +#### 资源需求（CPU/内存/网络） + +可通过 Helm values 按组件配置。chart 默认不设置 `resources`，因此生产集群应显式设置 requests/limits。以下估算是在 Kubernetes 1.20+ 上、启用 NVIDIA 共享且调度变动正常的前提下，针对 HAMi v2.8.0 的实用规划基线。 + +- **估算假设：** 1 个调度器副本（`kube-scheduler` + HAMi extender），每个 GPU 节点 1 个 device-plugin DaemonSet Pod（`device-plugin` + `vgpu-monitor`），Prometheus 每 15-30 秒抓取一次，且没有异常的 Pod 创建高峰。 + +| 组件 / 范围 | CPU（建议 request 到典型峰值） | 内存（建议 request 到典型峰值） | 网络（控制/可观测平面） | +| --- | --- | --- | --- | +| 调度器 Pod（每个副本） | 300m-800m | 512Mi-2Gi | 典型 0.2-1 Mbps，Pod 大量变动时突发 2-5 Mbps | +| Device-plugin Pod（每个 GPU 节点） | 150m-600m | 256Mi-1Gi | 每节点 0.05-0.2 Mbps（遥测与控制信令） | +| 指标抓取开销（每个目标，15 秒间隔） | N/A | N/A | 50-300 KiB/次抓取，约 0.03-0.16 Mbps | +| 集群总计（N 个 GPU 节点） | (0.3 到 0.8) + N * (0.15 到 0.6) 核 | (0.5 到 2) + N * (0.25 到 1) GiB | 大致与 N 线性相关，取决于每节点遥测加调度器控制流量 | +| 示例规模（N=20 个 GPU 节点） | 3.3-12.8 核 | 5.5-22 GiB | 典型聚合控制/指标流量 1.2-5 Mbps，大规模调度事件期间有更高的短时突发 | + +#### 存储需求 + +HAMi 没有强制使用的外部数据库。存储主要用于镜像层、日志/指标缓冲，以及调度器/device plugin 组件使用的 hostPath 挂载的运行时路径。以下是用于生产规划的实用容量估算。 + +- **估算假设：** 默认镜像保留策略、标准的 Kubernetes/容器运行时日志，以及 Pod 变动正常的训练/推理混合工作负载。 + +| 组件 / 范围 | 估算的存储占用 | 说明 | +| --- | --- | --- | +| 调度器 Pod（每个副本） | 0.2-0.8 GiB 节点本地临时存储 | 主要是镜像层、日志和临时运行时文件 | +| Device-plugin + monitor Pod（每个 GPU 节点） | 0.3-1.2 GiB 节点本地临时存储 | 包括插件/monitor 镜像层、日志以及运行时临时/缓存路径 | +| GPU 节点上 HAMi 使用的宿主机运行时路径 | 在 Pod 临时存储用量之外，建议预留 0.2-1 GiB 空闲空间 | 涵盖 `/var/lib/kubelet/device-plugins`、`/usr/local/vgpu`、`/var/run/cdi` 以及 `/tmp` 下的临时文件 | +| 集群总计（N 个 GPU 节点，1 个调度器副本） | 约 `(0.2 到 0.8) + N * (0.5 到 2.2)` GiB | 合计调度器临时存储、每个 GPU 节点的 Pod 以及宿主机运行时预留 | +| 示例规模（N=20 个 GPU 节点） | 集群范围内约预留 10.2-44.8 GiB | 为稳定运维和升级余量准备的规划区间 | + +#### API 设计 + +- 使用 Kubernetes 原生 API（Pod、Node、注解、准入 webhook、device plugin gRPC）。 +- 默认值和可选配置通过 Helm chart values 和配置文档提供。 +- 核心工作流不需要自定义 CRD。 +- 设备切片的资源名称以 Kubernetes 资源的形式暴露，用户可以直接在 `resources.limits` / `resources.requests` 中申请和限制切片。 +- GPU 调度策略和高级行为通过 Pod 注解来表达。 +- HAMi 的运行时行为可通过调度器和 device plugin 的 ConfigMap 配置，其中 Helm values 是主要入口。 + +#### 发布流程 + +采用语义化版本、带标签的发布、发布分支、自动化的镜像/chart/发布说明工作流，并有文档化的人工验证步骤。 + +- 发布流程文档：https://github.com/Project-HAMi/HAMi/blob/master/.github/release-process.md + +### 安装 + +- **安装与验证：** 基于 Helm 安装，包含节点标签和 chart values 自定义。 + - 快速入门：https://project-hami.io/docs/installation/online-installation + - 部署命令示例： + +```bash +# 1) Label GPU nodes for HAMi management +kubectl label nodes gpu=on + +# 2) Add HAMi Helm repository +helm repo add hami-charts https://project-hami.github.io/HAMi/ +helm repo update + +# 3) Install HAMi into kube-system +helm install hami hami-charts/hami -n kube-system + +# 4) (Optional) Customize values during install +# helm install hami hami-charts/hami -n kube-system -f values.yaml + +# 5) Verify core components are running +kubectl get pods -n kube-system | grep -E "hami-scheduler|hami-device-plugin" +``` + +### 安全 + +参见单独的[文档](https://github.com/cncf/toc/blob/main/projects/hami/security-assesment/self-assessment.md) + +## 第 1 天 - 安装与部署阶段 + +### 项目安装与配置 + +- 安装使用 Helm chart，包括调度器策略、资源命名、webhook 选择器、TLS 策略和 device plugin 行为的配置。 + - 安装指南：https://project-hami.io/docs/installation/online-installation +- 运行行为可通过 Helm values 和 ConfigMap 配置（例如调度器和设备策略的 ConfigMap）。 + - 配置指南：https://project-hami.io/docs/userguide/configure + +### 项目启用与回滚 + +- **在运行中的集群里启用/停用：** 通过 Helm 部署/升级/卸载。启用不需要控制平面停机；当 webhook/调度器集成发生变化时，工作负载可能会遇到调度行为的变化。 +- **启用后的行为变化：** + - 对可识别的设备请求进行 Pod 变更。 + - 设备共享工作负载的调度会经由 HAMi 的逻辑和策略。 + - 由 device plugin 负责分配/挂载/环境变量处理，以便在运行时强制执行限制。 +- **启用/停用的测试：** 部署和卸载路径通过 e2e 工作流验证；PR 合并前须通过必需的 e2e 检查。 +- **资源清理：** Helm 卸载会移除由 chart 管理的资源；默认路径不会引入核心 CRD。 + +### 发布、升级与回滚规划 + +- **兼容性节奏：** Kubernetes 及相关依赖通过 CI 和自动化依赖更新（包括 Dependabot）进行跟踪。版本演进遵循文档化的发布流程和发布工作流。 +- **回滚流程：** 使用 `helm rollback` 原地回滚；若希望完整重新部署，也可以卸载后用旧版本 chart 重新部署。 +- **发布/回滚可能失败的情形：** + - 回滚了镜像，却没有回滚与之匹配的 chart/配置版本。 + - 没有将自定义 values 和策略 ConfigMap 与工作负载/镜像一起回滚。 + - 未经分阶段验证就进行跨多个版本的回滚（例如两个或更多次版本）。 +- **对运行中工作负载的影响：** HAMi v2.5 解决了重新安装期间运行中任务崩溃的问题，但升级和回滚仍可能影响活动的工作负载；升级前请停止或重新调度 GPU 工作负载。新的调度/分配决策也可能随之改变。 +- **回滚指标：** 主要指标包括准入/调度错误、Pending Pod 异常增多、device plugin 注册问题以及策略不匹配的症状。在运维上，项目建议尽可能保持在较新的受支持版本上。 +- **升级路径测试：** 已有单元/e2e 覆盖；明确的长链路“升级->降级->升级”矩阵仍在演进中，孵化阶段应进一步扩展。 +- **弃用沟通：** 弃用信息通过文档和社区会议传达，并在文档中显式标注。典型的过渡期是新旧 API 在随后的一个版本中同时保留，下一个发布周期之后再移除旧 API。 +- **Alpha/Beta 能力：** 通过配置开关和 values 暴露；用户通过 chart values 和文档化的设置选择启用。 From b99d7ec0b72c25b5a046f874f8c25507763263fa Mon Sep 17 00:00:00 2001 From: mesutoezdil Date: Sat, 10 Oct 2026 09:29:48 +0200 Subject: [PATCH 4/4] docs: align general technical review Kubernetes baseline with prerequisites Signed-off-by: mesutoezdil --- docs/contributor/general-technical-review.md | 2 +- .../current/contributor/general-technical-review.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/contributor/general-technical-review.md b/docs/contributor/general-technical-review.md index 52860cb40..aed584062 100644 --- a/docs/contributor/general-technical-review.md +++ b/docs/contributor/general-technical-review.md @@ -120,7 +120,7 @@ Scheduler supports leader election and configurable replicas. #### Resource requirements (CPU/memory/network) -Configurable per component via Helm values. The chart leaves `resources` unset by default, so production clusters should set explicit requests/limits. The following estimates are practical planning baselines for HAMi v2.8.0 on Kubernetes 1.20+, with NVIDIA sharing enabled and normal scheduling churn. +Configurable per component via Helm values. The chart leaves `resources` unset by default, so production clusters should set explicit requests/limits. The following estimates are practical planning baselines for HAMi v2.8.0 on Kubernetes 1.23+, with NVIDIA sharing enabled and normal scheduling churn. - **Assumptions for estimates:** 1 scheduler replica (`kube-scheduler` + HAMi extender), 1 device-plugin DaemonSet pod per GPU node (`device-plugin` + `vgpu-monitor`), Prometheus scraping every 15-30s, and no unusual pod-creation spikes. diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor/general-technical-review.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor/general-technical-review.md index bb7c85fcf..c96a8c056 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor/general-technical-review.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor/general-technical-review.md @@ -121,7 +121,7 @@ Kubernetes API server、kube-scheduler/Volcano 调度集成、kubelet device plu #### 资源需求（CPU/内存/网络） -可通过 Helm values 按组件配置。chart 默认不设置 `resources`，因此生产集群应显式设置 requests/limits。以下估算是在 Kubernetes 1.20+ 上、启用 NVIDIA 共享且调度变动正常的前提下，针对 HAMi v2.8.0 的实用规划基线。 +可通过 Helm values 按组件配置。chart 默认不设置 `resources`，因此生产集群应显式设置 requests/limits。以下估算是在 Kubernetes 1.23+ 上、启用 NVIDIA 共享且调度变动正常的前提下，针对 HAMi v2.8.0 的实用规划基线。 - **估算假设：** 1 个调度器副本（`kube-scheduler` + HAMi extender），每个 GPU 节点 1 个 device-plugin DaemonSet Pod（`device-plugin` + `vgpu-monitor`），Prometheus 每 15-30 秒抓取一次，且没有异常的 Pod 创建高峰。