diff --git a/docs/installation/online-installation.md b/docs/installation/online-installation.md index a5375b948..fbea592f5 100644 --- a/docs/installation/online-installation.md +++ b/docs/installation/online-installation.md @@ -30,7 +30,7 @@ Ensure the `scheduler.kubeScheduler.image.tag` matches your Kubernetes server ve helm install hami hami-charts/hami --set scheduler.kubeScheduler.image.tag=v1.29.0 -n kube-system ``` -Customize your installation by editing the [configurations](../userguide/configure.md). +Customize your installation by editing the [configurations](../userguide/configure.md). If you are running on managed cloud platforms such as AWS or Microsoft Azure (AKS), refer to the [Platform Guides](./platforms/aks.md) for platform-tailored installation instructions. ## Verify your installation diff --git a/docs/installation/platforms/aks.md b/docs/installation/platforms/aks.md new file mode 100644 index 000000000..271a91d7f --- /dev/null +++ b/docs/installation/platforms/aks.md @@ -0,0 +1,205 @@ +--- +title: HAMi on Microsoft Azure (AKS) +sidebar_label: Microsoft Azure (AKS) +--- + +This guide provides step-by-step instructions for deploying and running HAMi on **Azure Kubernetes Service (AKS)** to enable GPU sharing and resource virtualization across NVIDIA GPU node pools. + +## Overview + +Azure Kubernetes Service offers several GPU-enabled VM series (such as the `NCv3`, `NCasT4_v3`, `NVadsA10_v5`, and `NDv4` families). By default, Kubernetes assigns whole physical GPUs to single containers. HAMi allows multiple pods to share the same physical GPU with fine-grained memory and compute core isolation on AKS. + +## Prerequisites + +Before deploying HAMi on AKS, ensure you have: + +- **Azure CLI (`az`)**: Installed and authenticated (`az login`). +- **`kubectl`** and **`helm` (v3.0+)**: Installed locally. +- An existing AKS cluster with a GPU-enabled node pool (or follow the steps below to create one). +- Kubernetes server version `>= 1.23`. + +## Step 1: Create a GPU Node Pool in AKS + +If your cluster does not yet have GPU nodes, add a GPU node pool using the Azure CLI. + +For example, to create a node pool with NVIDIA V100 GPUs (`Standard_NC6s_v3`): + +```bash +az aks nodepool add \ + --resource-group \ + --cluster-name \ + --name gpunodes \ + --node-count 2 \ + --node-vm-size Standard_NC6s_v3 \ + --node-taints sku=gpu:NoSchedule \ + --labels gpu=on +``` + +:::note Common Azure GPU VM sizes include: + +- `Standard_NC6s_v3` (1x NVIDIA Tesla V100 16GB) +- `Standard_NC4as_T4_v3` (1x NVIDIA Tesla T4 16GB) +- `Standard_NV6ads_A10_v5` (1x NVIDIA A10 24GB) +- `Standard_ND96amsr_A100_v4` (8x NVIDIA A100 80GB) + +::: + +### Label Your Nodes + +HAMi monitors and schedules workloads only on nodes with the label `gpu=on`. If your node pool was created without this label, add it manually: + +```bash +kubectl label nodes gpu=on +``` + +### Install NVIDIA Drivers + +Ensure NVIDIA drivers are installed on the GPU nodes. You can either: + +- Use AKS automated GPU driver provisioning (`--enable-gpu-driver-daemonset` on supported Azure Linux / Ubuntu images). +- Or install the [NVIDIA GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html) with driver installation enabled. + +## Step 2: Prevent Device Plugin Conflicts + +AKS clusters may automatically deploy the default NVIDIA Kubernetes Device Plugin (`nvidia-device-plugin-daemonset`). + +If both the default NVIDIA device plugin and HAMi's device plugin run simultaneously, both will attempt to register `nvidia.com/gpu` with the kubelet, causing double-registration conflicts. + +1. Check if the default NVIDIA device plugin DaemonSet is running: + + ```bash + kubectl get ds -n kube-system -l app=nvidia-device-plugin-daemonset + ``` + +1. If present, disable or remove the default DaemonSet so that HAMi can act as the sole GPU resource registrar: + + ```bash + kubectl delete ds -n kube-system + ``` + +## Step 3: Install HAMi via Helm + +### Add the HAMi Helm Repository + +```bash +helm repo add hami-charts https://project-hami.github.io/HAMi/ +helm repo update +``` + +### Identify Your Kubernetes Version + +Get your AKS cluster Kubernetes version: + +```bash +kubectl version --short +``` + +### Create AKS Custom Values + +Create a file named `custom-aks-values.yaml`. Make sure to configure the `tolerations` matching your AKS GPU node taints (`sku=gpu:NoSchedule`), set `scheduler.kubeScheduler.image.tag` to match your cluster version, and optionally enable scheduler High Availability: + +```yaml +scheduler: + leaderElect: true + replicaCount: 2 + kubeScheduler: + image: + # Set tag to match your AKS Kubernetes server version (e.g. v1.29.0) + tag: "v1.29.0" + +devicePlugin: + tolerations: + - key: "sku" + operator: "Equal" + value: "gpu" + effect: "NoSchedule" + - key: "nvidia.com/gpu" + operator: "Exists" + effect: "NoSchedule" +``` + +### Deploy the Chart + +Deploy HAMi to the `kube-system` namespace: + +```bash +helm install hami hami-charts/hami \ + -f custom-aks-values.yaml \ + -n kube-system +``` + +## Step 4: Verify Your Installation + +### 1. Verify Pod Status + +Check that the HAMi scheduler and device plugin pods are in `Running` state: + +```bash +kubectl get pods -n kube-system -l app.kubernetes.io/name=hami +``` + +Expected output: + +```text +NAME READY STATUS RESTARTS AGE +hami-device-plugin-xxxxx 1/1 Running 0 2m +hami-device-plugin-yyyyy 1/1 Running 0 2m +hami-scheduler-6d8b97bc49-abcde 1/1 Running 0 2m +hami-scheduler-6d8b97bc49-fghij 1/1 Running 0 2m +``` + +### 2. Verify Node Extended Resources + +Check that your GPU node advertises HAMi virtual GPU resources (`nvidia.com/gpumem` and `nvidia.com/gpucores`): + +```bash +kubectl describe node | grep -E "(nvidia.com/gpu|nvidia.com/gpumem|nvidia.com/gpucores):" +``` + +Expected output shows `nvidia.com/gpu`, `nvidia.com/gpumem` (in MiB), and `nvidia.com/gpucores` (percentage): + +```text + nvidia.com/gpu: 1 + nvidia.com/gpumem: 16280 + nvidia.com/gpucores: 100 +``` + +## Step 5: Run a Smoke Test Workload + +Submit a test Pod that requests a fraction of the GPU memory (e.g., 4000 MiB) and 40% of GPU compute cores: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: aks-gpu-test +spec: + restartPolicy: OnFailure + tolerations: + - key: "sku" + operator: "Equal" + value: "gpu" + effect: "NoSchedule" + containers: + - name: cuda-test + image: nvidia/cuda:12.2.0-base-ubuntu22.04 + command: ["nvidia-smi"] + resources: + limits: + nvidia.com/gpu: 1 + nvidia.com/gpumem: 4000 + nvidia.com/gpucores: 40 +``` + +Apply the pod manifest and inspect logs: + +```bash +kubectl apply -f aks-gpu-test.yaml +kubectl logs aks-gpu-test +``` + +## AKS-Specific Gotchas & Troubleshooting + +- **Node Taint Mismatches**: If `hami-device-plugin` pods remain in `Pending` state, check your GPU node taints (`kubectl describe node `) and ensure all taints are present in `devicePlugin.tolerations` within `values.yaml`. +- **Node Auto-scaling**: If you enable the AKS Cluster Autoscaler on GPU node pools, new nodes will automatically receive the node labels and taints defined during node pool creation. Ensure `gpu=on` is set on the nodepool configuration. +- **Image Pull Issues**: If your AKS cluster is in a restricted or private virtual network, consider mirroring HAMi container images to an Azure Container Registry (ACR) and configuring `image.repository` in `values.yaml`. diff --git a/docs/installation/aws-installation.md b/docs/installation/platforms/aws.md similarity index 96% rename from docs/installation/aws-installation.md rename to docs/installation/platforms/aws.md index fc595b3b8..5a25a6e52 100644 --- a/docs/installation/aws-installation.md +++ b/docs/installation/platforms/aws.md @@ -1,5 +1,6 @@ --- title: HAMi on AWS +sidebar_label: Amazon Web Services (AWS) translated: true --- @@ -26,7 +27,7 @@ tar xf $(pwd)/* && find $(pwd) -maxdepth 1 -type f -delete helm install --generate-name --namespace ./* ``` -You can customize the installation by adjusting the [configuration](../userguide/configure.md). +You can customize the installation by adjusting the [configuration](../../userguide/configure.md). ## Install with AWS Add-on diff --git a/i18n/zh/docusaurus-plugin-content-docs/current.json b/i18n/zh/docusaurus-plugin-content-docs/current.json index 4f107183e..d3799e07f 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current.json +++ b/i18n/zh/docusaurus-plugin-content-docs/current.json @@ -15,6 +15,10 @@ "message": "安装", "description": "The label for category 'Installation' in sidebar 'docs'" }, + "sidebar.docs.category.Platform Guides": { + "message": "平台指南", + "description": "The label for category 'Platform Guides' in sidebar 'docs'" + }, "sidebar.docs.category.User Guide": { "message": "用户指南", "description": "The label for category 'User Guide' in sidebar 'docs'" diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/installation/online-installation.md b/i18n/zh/docusaurus-plugin-content-docs/current/installation/online-installation.md index 290de6f06..ab13b0a23 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/installation/online-installation.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/installation/online-installation.md @@ -31,7 +31,7 @@ kubectl version helm install hami hami-charts/hami --set scheduler.kubeScheduler.image.tag=v1.29.0 -n kube-system ``` -你可以通过编辑[配置](../userguide/configure.md)来自定义安装。 +你可以通过编辑[配置](../userguide/configure.md)来自定义安装。如果你在 AWS 或 Microsoft Azure (AKS) 等托管云平台上运行，请参阅[平台指南](./platforms/aks.md)获取针对特定平台的安装说明。 ## 验证你的安装 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/installation/platforms/aks.md b/i18n/zh/docusaurus-plugin-content-docs/current/installation/platforms/aks.md new file mode 100644 index 000000000..19851cd3a --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/installation/platforms/aks.md @@ -0,0 +1,205 @@ +--- +title: 在 Microsoft Azure (AKS) 上运行 HAMi +sidebar_label: Microsoft Azure (AKS) +--- + +本指南提供了在 **Azure Kubernetes Service (AKS)** 上部署和运行 HAMi 的逐步操作说明，以实现跨 NVIDIA GPU 节点池的 GPU 共享和资源虚拟化。 + +## 概述 + +Azure Kubernetes Service 提供了多种支持 GPU 的虚拟机系列（例如 `NCv3`、`NCasT4_v3`、`NVadsA10_v5` 和 `NDv4` 系列）。默认情况下，Kubernetes 会将整张物理 GPU 分配给单个容器。HAMi 允许在 AKS 上让多个 Pod 共享同一张物理 GPU，并提供细粒度的显存与算力核心隔离。 + +## 前提条件 + +在 AKS 上部署 HAMi 之前，请确保具备以下条件： + +- **Azure CLI (`az`)**：已安装并完成身份验证（`az login`）。 +- **`kubectl`** 和 **`helm` (v3.0+)**：已在本地安装。 +- 一个已创建的 AKS 集群，并配备支持 GPU 的节点池（或按照以下步骤创建）。 +- Kubernetes 服务器版本 `>= 1.23`。 + +## 步骤 1：在 AKS 中创建 GPU 节点池 + +如果您的集群尚未拥有 GPU 节点，可以使用 Azure CLI 添加 GPU 节点池。 + +例如，创建一个带有 NVIDIA V100 GPU (`Standard_NC6s_v3`) 的节点池： + +```bash +az aks nodepool add \ + --resource-group \ + --cluster-name \ + --name gpunodes \ + --node-count 2 \ + --node-vm-size Standard_NC6s_v3 \ + --node-taints sku=gpu:NoSchedule \ + --labels gpu=on +``` + +:::note 常用的 Azure GPU 虚拟机规格包括： + +- `Standard_NC6s_v3`（1x NVIDIA Tesla V100 16GB） +- `Standard_NC4as_T4_v3`（1x NVIDIA Tesla T4 16GB） +- `Standard_NV6ads_A10_v5`（1x NVIDIA A10 24GB） +- `Standard_ND96amsr_A100_v4`（8x NVIDIA A100 80GB） + +::: + +### 标记您的节点 + +HAMi 仅在带有 `gpu=on` 标签的节点上监控和调度工作负载。如果您的节点池创建时未添加此标签，请手动添加： + +```bash +kubectl label nodes gpu=on +``` + +### 安装 NVIDIA 驱动程序 + +确保 GPU 节点上已安装 NVIDIA 驱动程序。您可以选择： + +- 使用 AKS 自动化 GPU 驱动程序配置（在受支持的 Azure Linux / Ubuntu 镜像上启用 `--enable-gpu-driver-daemonset`）。 +- 或者安装启用了驱动程序安装的 [NVIDIA GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html)。 + +## 步骤 2：避免设备插件冲突 + +AKS 集群可能会自动部署默认的 NVIDIA Kubernetes 设备插件 (`nvidia-device-plugin-daemonset`)。 + +如果默认的 NVIDIA 设备插件与 HAMi 设备插件同时运行，二者都会尝试向 kubelet 注册 `nvidia.com/gpu`，从而导致重复注册冲突。 + +1. 检查默认的 NVIDIA 设备插件 DaemonSet 是否正在运行： + + ```bash + kubectl get ds -n kube-system -l app=nvidia-device-plugin-daemonset + ``` + +1. 如果存在，请禁用或删除默认的 DaemonSet，以便 HAMi 能够作为唯一的 GPU 资源注册器运行： + + ```bash + kubectl delete ds -n kube-system + ``` + +## 步骤 3：使用 Helm 安装 HAMi + +### 添加 HAMi Helm 仓库 + +```bash +helm repo add hami-charts https://project-hami.github.io/HAMi/ +helm repo update +``` + +### 确认您的 Kubernetes 版本 + +获取您的 AKS 集群 Kubernetes 版本： + +```bash +kubectl version --short +``` + +### 创建 AKS 自定义配置文件 + +创建名为 `custom-aks-values.yaml` 的文件。请确保配置匹配 AKS GPU 节点污点 (`sku=gpu:NoSchedule`) 的容忍度 (`tolerations`)，将 `scheduler.kubeScheduler.image.tag` 设置为匹配您的集群版本，并可选择启用调度器的高可用配置： + +```yaml +scheduler: + leaderElect: true + replicaCount: 2 + kubeScheduler: + image: + # 设置与您的 AKS Kubernetes 服务器版本匹配的标签（例如 v1.29.0） + tag: "v1.29.0" + +devicePlugin: + tolerations: + - key: "sku" + operator: "Equal" + value: "gpu" + effect: "NoSchedule" + - key: "nvidia.com/gpu" + operator: "Exists" + effect: "NoSchedule" +``` + +### 部署 Chart + +将 HAMi 部署到 `kube-system` 命名空间： + +```bash +helm install hami hami-charts/hami \ + -f custom-aks-values.yaml \ + -n kube-system +``` + +## 步骤 4：验证安装 + +### 1. 验证 Pod 状态 + +检查 HAMi 调度器和设备插件 Pod 是否处于 `Running` 状态： + +```bash +kubectl get pods -n kube-system -l app.kubernetes.io/name=hami +``` + +预期输出： + +```text +NAME READY STATUS RESTARTS AGE +hami-device-plugin-xxxxx 1/1 Running 0 2m +hami-device-plugin-yyyyy 1/1 Running 0 2m +hami-scheduler-6d8b97bc49-abcde 1/1 Running 0 2m +hami-scheduler-6d8b97bc49-fghij 1/1 Running 0 2m +``` + +### 2. 验证节点扩展资源 + +检查您的 GPU 节点是否已上报 HAMi 虚拟 GPU 资源 (`nvidia.com/gpumem` 和 `nvidia.com/gpucores`)： + +```bash +kubectl describe node | grep -E "(nvidia.com/gpu|nvidia.com/gpumem|nvidia.com/gpucores):" +``` + +预期输出会显示 `nvidia.com/gpu`、`nvidia.com/gpumem`（单位为 MiB）和 `nvidia.com/gpucores`（百分比）： + +```text + nvidia.com/gpu: 1 + nvidia.com/gpumem: 16280 + nvidia.com/gpucores: 100 +``` + +## 步骤 5：运行冒烟测试工作负载 + +提交一个申请部分显存（例如 4000 MiB）和 40% GPU 算力核心的测试 Pod： + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: aks-gpu-test +spec: + restartPolicy: OnFailure + tolerations: + - key: "sku" + operator: "Equal" + value: "gpu" + effect: "NoSchedule" + containers: + - name: cuda-test + image: nvidia/cuda:12.2.0-base-ubuntu22.04 + command: ["nvidia-smi"] + resources: + limits: + nvidia.com/gpu: 1 + nvidia.com/gpumem: 4000 + nvidia.com/gpucores: 40 +``` + +应用 Pod 配置并查看日志： + +```bash +kubectl apply -f aks-gpu-test.yaml +kubectl logs aks-gpu-test +``` + +## AKS 专属注意事项与故障排查 + +- **节点污点不匹配**：如果 `hami-device-plugin` Pod 处于 `Pending` 状态，请检查 GPU 节点污点（`kubectl describe node `），并确保 `values.yaml` 中的 `devicePlugin.tolerations` 包含所有对应的污点。 +- **节点自动扩缩容**：如果在 GPU 节点池上启用了 AKS Cluster Autoscaler，新扩容的节点将自动继承创建节点池时定义的标签和污点。请确保在节点池配置中设置了 `gpu=on`。 +- **镜像拉取问题**：如果您的 AKS 集群位于受限或专用虚拟网络中，请考虑将 HAMi 容器镜像镜像到 Azure Container Registry (ACR)，并在 `values.yaml` 中配置 `image.repository`。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/installation/aws-installation.md b/i18n/zh/docusaurus-plugin-content-docs/current/installation/platforms/aws.md similarity index 94% rename from i18n/zh/docusaurus-plugin-content-docs/current/installation/aws-installation.md rename to i18n/zh/docusaurus-plugin-content-docs/current/installation/platforms/aws.md index 16e076f08..c87274124 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/installation/aws-installation.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/installation/platforms/aws.md @@ -1,6 +1,6 @@ --- title: 在 AWS 上安装与使用 HAMi -sidebar_label: AWS 上的 HAMi +sidebar_label: Amazon Web Services (AWS) translated: true --- @@ -27,7 +27,7 @@ tar xf $(pwd)/* && find $(pwd) -maxdepth 1 -type f -delete helm install --generate-name --namespace ./* ``` -你可以通过调整[配置](../userguide/configure.md)来自定义安装。 +你可以通过调整[配置](../../userguide/configure.md)来自定义安装。 ## 使用 AWS add-on 安装 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0.json b/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0.json index 8b36feb20..d6ef5186a 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0.json +++ b/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0.json @@ -15,6 +15,10 @@ "message": "安装", "description": "The label for category 'Installation' in sidebar 'docs'" }, + "sidebar.docs.category.Platform Guides": { + "message": "平台指南", + "description": "The label for category 'Platform Guides' in sidebar 'docs'" + }, "sidebar.docs.category.User Guide": { "message": "用户指南", "description": "The label for category 'User Guide' in sidebar 'docs'" diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/installation/online-installation.md b/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/installation/online-installation.md index 53dfbcfb4..ab13b0a23 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/installation/online-installation.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/installation/online-installation.md @@ -12,6 +12,7 @@ translated: true ```bash helm repo add hami-charts https://project-hami.github.io/HAMi/ +helm repo update ``` ## 获取你的 Kubernetes 版本 @@ -19,18 +20,18 @@ helm repo add hami-charts https://project-hami.github.io/HAMi/ 安装时需要 Kubernetes 版本。你可以使用以下命令获取此信息： ```bash -kubectl version --short +kubectl version ``` ## 安装 -确保 `scheduler.kubeScheduler.imageTag` 与你的 Kubernetes 服务器版本匹配。例如，如果你的集群服务器版本是 v1.16.8，请使用以下命令进行部署： +确保 `scheduler.kubeScheduler.image.tag` 与你的 Kubernetes 服务器版本匹配。例如，如果你的集群服务器版本是 v1.29.0，请使用以下命令进行部署： ```bash -helm install hami hami-charts/hami --set scheduler.kubeScheduler.imageTag=v1.16.8 -n kube-system +helm install hami hami-charts/hami --set scheduler.kubeScheduler.image.tag=v1.29.0 -n kube-system ``` -你可以通过编辑[配置](../userguide/configure.md)来自定义安装。 +你可以通过编辑[配置](../userguide/configure.md)来自定义安装。如果你在 AWS 或 Microsoft Azure (AKS) 等托管云平台上运行，请参阅[平台指南](./platforms/aks.md)获取针对特定平台的安装说明。 ## 验证你的安装 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/installation/platforms/aks.md b/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/installation/platforms/aks.md new file mode 100644 index 000000000..960e6250c --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/installation/platforms/aks.md @@ -0,0 +1,204 @@ +--- +title: 在 Microsoft Azure (AKS) 上运行 HAMi +sidebar_label: Microsoft Azure (AKS) +--- + +本指南提供了在 **Azure Kubernetes Service (AKS)** 上部署和运行 HAMi 的逐步操作说明，以实现跨 NVIDIA GPU 节点池的 GPU 共享和资源虚拟化。 + +## 概述 + +Azure Kubernetes Service 提供了多种支持 GPU 的虚拟机系列（例如 `NCv3`、`NCasT4_v3`、`NVadsA10_v5` 和 `NDv4` 系列）。默认情况下，Kubernetes 会将整张物理 GPU 分配给单个容器。HAMi 允许在 AKS 上让多个 Pod 共享同一张物理 GPU，并提供细粒度的显存与算力核心隔离。 + +## 前提条件 + +在 AKS 上部署 HAMi 之前，请确保具备以下条件： + +- **Azure CLI (`az`)**：已安装并完成身份验证（`az login`）。 +- **`kubectl`** 和 **`helm` (v3.0+)**：已在本地安装。 +- 一个已创建的 AKS 集群，并配备支持 GPU 的节点池（或按照以下步骤创建）。 +- Kubernetes 服务器版本 `>= 1.23`。 + +## 步骤 1：在 AKS 中创建 GPU 节点池 + +如果您的集群尚未拥有 GPU 节点，可以使用 Azure CLI 添加 GPU 节点池。 + +例如，创建一个带有 NVIDIA V100 GPU (`Standard_NC6s_v3`) 的节点池： + +```bash +az aks nodepool add \ + --resource-group \ + --cluster-name \ + --name gpunodes \ + --node-count 2 \ + --node-vm-size Standard_NC6s_v3 \ + --node-taints sku=gpu:NoSchedule \ + --labels gpu=on +``` + +:::note 常用的 Azure GPU 虚拟机规格包括： + +- `Standard_NC6s_v3`（1x NVIDIA Tesla V100 16GB） +- `Standard_NC4as_T4_v3`（1x NVIDIA Tesla T4 16GB） +- `Standard_NV6ads_A10_v5`（1x NVIDIA A10 24GB） +- `Standard_ND96amsr_A100_v4`（8x NVIDIA A100 80GB） + +::: + +### 标记您的节点 + +HAMi 仅在带有 `gpu=on` 标签的节点上监控和调度工作负载。如果您的节点池创建时未添加此标签，请手动添加： + +```bash +kubectl label nodes gpu=on +``` + +### 安装 NVIDIA 驱动程序 + +确保 GPU 节点上已安装 NVIDIA 驱动程序。您可以选择： +- 使用 AKS 自动化 GPU 驱动程序配置（在受支持的 Azure Linux / Ubuntu 镜像上启用 `--enable-gpu-driver-daemonset`）。 +- 或者安装启用了驱动程序安装的 [NVIDIA GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html)。 + +## 步骤 2：避免设备插件冲突 + +AKS 集群可能会自动部署默认的 NVIDIA Kubernetes 设备插件 (`nvidia-device-plugin-daemonset`)。 + +如果默认的 NVIDIA 设备插件与 HAMi 设备插件同时运行，二者都会尝试向 kubelet 注册 `nvidia.com/gpu`，从而导致重复注册冲突。 + +1. 检查默认的 NVIDIA 设备插件 DaemonSet 是否正在运行： + + ```bash + kubectl get ds -n kube-system -l app=nvidia-device-plugin-daemonset + ``` + +1. 如果存在，请禁用或删除默认的 DaemonSet，以便 HAMi 能够作为唯一的 GPU 资源注册器运行： + + ```bash + kubectl delete ds -n kube-system + ``` + +## 步骤 3：使用 Helm 安装 HAMi + +### 添加 HAMi Helm 仓库 + +```bash +helm repo add hami-charts https://project-hami.github.io/HAMi/ +helm repo update +``` + +### 确认您的 Kubernetes 版本 + +获取您的 AKS 集群 Kubernetes 版本： + +```bash +kubectl version --short +``` + +### 创建 AKS 自定义配置文件 + +创建名为 `custom-aks-values.yaml` 的文件。请确保配置匹配 AKS GPU 节点污点 (`sku=gpu:NoSchedule`) 的容忍度 (`tolerations`)，将 `scheduler.kubeScheduler.image.tag` 设置为匹配您的集群版本，并可选择启用调度器的高可用配置： + +```yaml +scheduler: + leaderElect: true + replicaCount: 2 + kubeScheduler: + image: + # 设置与您的 AKS Kubernetes 服务器版本匹配的标签（例如 v1.29.0） + tag: "v1.29.0" + +devicePlugin: + tolerations: + - key: "sku" + operator: "Equal" + value: "gpu" + effect: "NoSchedule" + - key: "nvidia.com/gpu" + operator: "Exists" + effect: "NoSchedule" +``` + +### 部署 Chart + +将 HAMi 部署到 `kube-system` 命名空间： + +```bash +helm install hami hami-charts/hami \ + -f custom-aks-values.yaml \ + -n kube-system +``` + +## 步骤 4：验证安装 + +### 1. 验证 Pod 状态 + +检查 HAMi 调度器和设备插件 Pod 是否处于 `Running` 状态： + +```bash +kubectl get pods -n kube-system -l app.kubernetes.io/name=hami +``` + +预期输出： + +```text +NAME READY STATUS RESTARTS AGE +hami-device-plugin-xxxxx 1/1 Running 0 2m +hami-device-plugin-yyyyy 1/1 Running 0 2m +hami-scheduler-6d8b97bc49-abcde 1/1 Running 0 2m +hami-scheduler-6d8b97bc49-fghij 1/1 Running 0 2m +``` + +### 2. 验证节点扩展资源 + +检查您的 GPU 节点是否已上报 HAMi 虚拟 GPU 资源 (`nvidia.com/gpumem` 和 `nvidia.com/gpucores`)： + +```bash +kubectl describe node | grep -E "(nvidia.com/gpu|nvidia.com/gpumem|nvidia.com/gpucores):" +``` + +预期输出会显示 `nvidia.com/gpu`、`nvidia.com/gpumem`（单位为 MiB）和 `nvidia.com/gpucores`（百分比）： + +```text + nvidia.com/gpu: 1 + nvidia.com/gpumem: 16280 + nvidia.com/gpucores: 100 +``` + +## 步骤 5：运行冒烟测试工作负载 + +提交一个申请部分显存（例如 4000 MiB）和 40% GPU 算力核心的测试 Pod： + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: aks-gpu-test +spec: + restartPolicy: OnFailure + tolerations: + - key: "sku" + operator: "Equal" + value: "gpu" + effect: "NoSchedule" + containers: + - name: cuda-test + image: nvidia/cuda:12.2.0-base-ubuntu22.04 + command: ["nvidia-smi"] + resources: + limits: + nvidia.com/gpu: 1 + nvidia.com/gpumem: 4000 + nvidia.com/gpucores: 40 +``` + +应用 Pod 配置并查看日志： + +```bash +kubectl apply -f aks-gpu-test.yaml +kubectl logs aks-gpu-test +``` + +## AKS 专属注意事项与故障排查 + +- **节点污点不匹配**：如果 `hami-device-plugin` Pod 处于 `Pending` 状态，请检查 GPU 节点污点（`kubectl describe node `），并确保 `values.yaml` 中的 `devicePlugin.tolerations` 包含所有对应的污点。 +- **节点自动扩缩容**：如果在 GPU 节点池上启用了 AKS Cluster Autoscaler，新扩容的节点将自动继承创建节点池时定义的标签和污点。请确保在节点池配置中设置了 `gpu=on`。 +- **镜像拉取问题**：如果您的 AKS 集群位于受限或专用虚拟网络中，请考虑将 HAMi 容器镜像镜像到 Azure Container Registry (ACR)，并在 `values.yaml` 中配置 `image.repository`。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/installation/aws-installation.md b/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/installation/platforms/aws.md similarity index 90% rename from i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/installation/aws-installation.md rename to i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/installation/platforms/aws.md index 67ffbafb5..c87274124 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/installation/aws-installation.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/installation/platforms/aws.md @@ -1,6 +1,6 @@ --- title: 在 AWS 上安装与使用 HAMi -sidebar_label: AWS 上的 HAMi +sidebar_label: Amazon Web Services (AWS) translated: true --- @@ -27,7 +27,7 @@ tar xf $(pwd)/* && find $(pwd) -maxdepth 1 -type f -delete helm install --generate-name --namespace ./* ``` -你可以通过调整[配置](../userguide/configure.md)来自定义安装。 +你可以通过调整[配置](../../userguide/configure.md)来自定义安装。 ## 使用 AWS add-on 安装 @@ -52,4 +52,4 @@ kubectl get pods -n kube-system - [使用独占 GPU](/zh/docs/userguide/nvidia-device/examples/use-exclusive-card) - [为容器分配特定设备显存](/zh/docs/userguide/nvidia-device/examples/allocate-device-memory) - [为容器分配设备核心资源](/zh/docs/userguide/nvidia-device/examples/allocate-device-core) -- [将任务分配给 mig 实例](/zh/docs/userguide/nvidia-device/examples/dynamic-mig-example) +- [将任务分配给 MIG 实例](/zh/docs/userguide/nvidia-device/examples/dynamic-mig-example) diff --git a/sidebars.js b/sidebars.js index 828ff4e7c..04b13a2e4 100644 --- a/sidebars.js +++ b/sidebars.js @@ -55,7 +55,16 @@ module.exports = { "installation/upgrade", "installation/uninstall", "installation/webui-installation", - "installation/aws-installation", + { + type: "category", + label: "Platform Guides", + link: { + type: "generated-index", + title: "Platform Guides", + description: "Deploy HAMi on various managed Kubernetes and cloud platforms.", + }, + items: ["installation/platforms/aws", "installation/platforms/aks"], + }, "installation/how-to-use-hami-dra", "installation/how-to-use-volcano-vgpu", "installation/how-to-use-volcano-ascend", diff --git a/versioned_docs/version-v2.9.0/installation/online-installation.md b/versioned_docs/version-v2.9.0/installation/online-installation.md index 3b2f5eded..fbea592f5 100644 --- a/versioned_docs/version-v2.9.0/installation/online-installation.md +++ b/versioned_docs/version-v2.9.0/installation/online-installation.md @@ -24,13 +24,13 @@ kubectl version ## Installation -Ensure the `scheduler.kubeScheduler.imageTag` matches your Kubernetes server version. For instance, if your cluster server is v1.29.0, use the following command to deploy: +Ensure the `scheduler.kubeScheduler.image.tag` matches your Kubernetes server version. For instance, if your cluster server is v1.29.0, use the following command to deploy: ```bash -helm install hami hami-charts/hami --set scheduler.kubeScheduler.imageTag=v1.29.0 -n kube-system +helm install hami hami-charts/hami --set scheduler.kubeScheduler.image.tag=v1.29.0 -n kube-system ``` -Customize your installation by editing the [configurations](../userguide/configure.md). +Customize your installation by editing the [configurations](../userguide/configure.md). If you are running on managed cloud platforms such as AWS or Microsoft Azure (AKS), refer to the [Platform Guides](./platforms/aks.md) for platform-tailored installation instructions. ## Verify your installation diff --git a/versioned_docs/version-v2.9.0/installation/platforms/aks.md b/versioned_docs/version-v2.9.0/installation/platforms/aks.md new file mode 100644 index 000000000..ec97744ea --- /dev/null +++ b/versioned_docs/version-v2.9.0/installation/platforms/aks.md @@ -0,0 +1,204 @@ +--- +title: HAMi on Microsoft Azure (AKS) +sidebar_label: Microsoft Azure (AKS) +--- + +This guide provides step-by-step instructions for deploying and running HAMi on **Azure Kubernetes Service (AKS)** to enable GPU sharing and resource virtualization across NVIDIA GPU node pools. + +## Overview + +Azure Kubernetes Service offers several GPU-enabled VM series (such as the `NCv3`, `NCasT4_v3`, `NVadsA10_v5`, and `NDv4` families). By default, Kubernetes assigns whole physical GPUs to single containers. HAMi allows multiple pods to share the same physical GPU with fine-grained memory and compute core isolation on AKS. + +## Prerequisites + +Before deploying HAMi on AKS, ensure you have: + +- **Azure CLI (`az`)**: Installed and authenticated (`az login`). +- **`kubectl`** and **`helm` (v3.0+)**: Installed locally. +- An existing AKS cluster with a GPU-enabled node pool (or follow the steps below to create one). +- Kubernetes server version `>= 1.23`. + +## Step 1: Create a GPU Node Pool in AKS + +If your cluster does not yet have GPU nodes, add a GPU node pool using the Azure CLI. + +For example, to create a node pool with NVIDIA V100 GPUs (`Standard_NC6s_v3`): + +```bash +az aks nodepool add \ + --resource-group \ + --cluster-name \ + --name gpunodes \ + --node-count 2 \ + --node-vm-size Standard_NC6s_v3 \ + --node-taints sku=gpu:NoSchedule \ + --labels gpu=on +``` + +:::note Common Azure GPU VM sizes include: + +- `Standard_NC6s_v3` (1x NVIDIA Tesla V100 16GB) +- `Standard_NC4as_T4_v3` (1x NVIDIA Tesla T4 16GB) +- `Standard_NV6ads_A10_v5` (1x NVIDIA A10 24GB) +- `Standard_ND96amsr_A100_v4` (8x NVIDIA A100 80GB) + +::: + +### Label Your Nodes + +HAMi monitors and schedules workloads only on nodes with the label `gpu=on`. If your node pool was created without this label, add it manually: + +```bash +kubectl label nodes gpu=on +``` + +### Install NVIDIA Drivers + +Ensure NVIDIA drivers are installed on the GPU nodes. You can either: +- Use AKS automated GPU driver provisioning (`--enable-gpu-driver-daemonset` on supported Azure Linux / Ubuntu images). +- Or install the [NVIDIA GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html) with driver installation enabled. + +## Step 2: Prevent Device Plugin Conflicts + +AKS clusters may automatically deploy the default NVIDIA Kubernetes Device Plugin (`nvidia-device-plugin-daemonset`). + +If both the default NVIDIA device plugin and HAMi's device plugin run simultaneously, both will attempt to register `nvidia.com/gpu` with the kubelet, causing double-registration conflicts. + +1. Check if the default NVIDIA device plugin DaemonSet is running: + + ```bash + kubectl get ds -n kube-system -l app=nvidia-device-plugin-daemonset + ``` + +1. If present, disable or remove the default DaemonSet so that HAMi can act as the sole GPU resource registrar: + + ```bash + kubectl delete ds -n kube-system + ``` + +## Step 3: Install HAMi via Helm + +### Add the HAMi Helm Repository + +```bash +helm repo add hami-charts https://project-hami.github.io/HAMi/ +helm repo update +``` + +### Identify Your Kubernetes Version + +Get your AKS cluster Kubernetes version: + +```bash +kubectl version --short +``` + +### Create AKS Custom Values + +Create a file named `custom-aks-values.yaml`. Make sure to configure the `tolerations` matching your AKS GPU node taints (`sku=gpu:NoSchedule`), set `scheduler.kubeScheduler.image.tag` to match your cluster version, and optionally enable scheduler High Availability: + +```yaml +scheduler: + leaderElect: true + replicaCount: 2 + kubeScheduler: + image: + # Set tag to match your AKS Kubernetes server version (e.g. v1.29.0) + tag: "v1.29.0" + +devicePlugin: + tolerations: + - key: "sku" + operator: "Equal" + value: "gpu" + effect: "NoSchedule" + - key: "nvidia.com/gpu" + operator: "Exists" + effect: "NoSchedule" +``` + +### Deploy the Chart + +Deploy HAMi to the `kube-system` namespace: + +```bash +helm install hami hami-charts/hami \ + -f custom-aks-values.yaml \ + -n kube-system +``` + +## Step 4: Verify Your Installation + +### 1. Verify Pod Status + +Check that the HAMi scheduler and device plugin pods are in `Running` state: + +```bash +kubectl get pods -n kube-system -l app.kubernetes.io/name=hami +``` + +Expected output: + +```text +NAME READY STATUS RESTARTS AGE +hami-device-plugin-xxxxx 1/1 Running 0 2m +hami-device-plugin-yyyyy 1/1 Running 0 2m +hami-scheduler-6d8b97bc49-abcde 1/1 Running 0 2m +hami-scheduler-6d8b97bc49-fghij 1/1 Running 0 2m +``` + +### 2. Verify Node Extended Resources + +Check that your GPU node advertises HAMi virtual GPU resources (`nvidia.com/gpumem` and `nvidia.com/gpucores`): + +```bash +kubectl describe node | grep -E "(nvidia.com/gpu|nvidia.com/gpumem|nvidia.com/gpucores):" +``` + +Expected output shows `nvidia.com/gpu`, `nvidia.com/gpumem` (in MiB), and `nvidia.com/gpucores` (percentage): + +```text + nvidia.com/gpu: 1 + nvidia.com/gpumem: 16280 + nvidia.com/gpucores: 100 +``` + +## Step 5: Run a Smoke Test Workload + +Submit a test Pod that requests a fraction of the GPU memory (e.g., 4000 MiB) and 40% of GPU compute cores: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: aks-gpu-test +spec: + restartPolicy: OnFailure + tolerations: + - key: "sku" + operator: "Equal" + value: "gpu" + effect: "NoSchedule" + containers: + - name: cuda-test + image: nvidia/cuda:12.2.0-base-ubuntu22.04 + command: ["nvidia-smi"] + resources: + limits: + nvidia.com/gpu: 1 + nvidia.com/gpumem: 4000 + nvidia.com/gpucores: 40 +``` + +Apply the pod manifest and inspect logs: + +```bash +kubectl apply -f aks-gpu-test.yaml +kubectl logs aks-gpu-test +``` + +## AKS-Specific Gotchas & Troubleshooting + +- **Node Taint Mismatches**: If `hami-device-plugin` pods remain in `Pending` state, check your GPU node taints (`kubectl describe node `) and ensure all taints are present in `devicePlugin.tolerations` within `values.yaml`. +- **Node Auto-scaling**: If you enable the AKS Cluster Autoscaler on GPU node pools, new nodes will automatically receive the node labels and taints defined during node pool creation. Ensure `gpu=on` is set on the nodepool configuration. +- **Image Pull Issues**: If your AKS cluster is in a restricted or private virtual network, consider mirroring HAMi container images to an Azure Container Registry (ACR) and configuring `image.repository` in `values.yaml`. diff --git a/versioned_docs/version-v2.9.0/installation/aws-installation.md b/versioned_docs/version-v2.9.0/installation/platforms/aws.md similarity index 96% rename from versioned_docs/version-v2.9.0/installation/aws-installation.md rename to versioned_docs/version-v2.9.0/installation/platforms/aws.md index fc595b3b8..5a25a6e52 100644 --- a/versioned_docs/version-v2.9.0/installation/aws-installation.md +++ b/versioned_docs/version-v2.9.0/installation/platforms/aws.md @@ -1,5 +1,6 @@ --- title: HAMi on AWS +sidebar_label: Amazon Web Services (AWS) translated: true --- @@ -26,7 +27,7 @@ tar xf $(pwd)/* && find $(pwd) -maxdepth 1 -type f -delete helm install --generate-name --namespace ./* ``` -You can customize the installation by adjusting the [configuration](../userguide/configure.md). +You can customize the installation by adjusting the [configuration](../../userguide/configure.md). ## Install with AWS Add-on diff --git a/versioned_sidebars/version-v2.9.0-sidebars.json b/versioned_sidebars/version-v2.9.0-sidebars.json index 6d0cd8678..eed223885 100644 --- a/versioned_sidebars/version-v2.9.0-sidebars.json +++ b/versioned_sidebars/version-v2.9.0-sidebars.json @@ -60,7 +60,19 @@ "installation/upgrade", "installation/uninstall", "installation/webui-installation", - "installation/aws-installation", + { + "type": "category", + "label": "Platform Guides", + "link": { + "type": "generated-index", + "title": "Platform Guides", + "description": "Deploy HAMi on various managed Kubernetes and cloud platforms." + }, + "items": [ + "installation/platforms/aws", + "installation/platforms/aks" + ] + }, "installation/how-to-use-hami-dra", "installation/how-to-use-volcano-vgpu", "installation/how-to-use-volcano-ascend"