docs(installation): add Microsoft Azure (AKS) guide and organize platform directory #809
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -30,7 +30,7 @@ Ensure the `scheduler.kubeScheduler.image.tag` matches your Kubernetes server ve | |
| helm install hami hami-charts/hami --set scheduler.kubeScheduler.image.tag=v1.29.0 -n kube-system | ||
| ``` | ||
|
|
||
| Customize your installation by editing the [configurations](../userguide/configure.md). If you are running on managed cloud platforms such as AWS or Microsoft Azure (AKS), refer to the [Platform Guides](./platforms/aks.md) for platform-tailored installation instructions. | ||
|
Contributor
There was a problem hiding this comment. #812 cuts a v2.10.0 snapshot that becomes the default version. if it lands first, these new pages and sidebar entries need v2.10.0 copies too or the default docs will not have them.
Contributor
Author
There was a problem hiding this comment. If #812 merges first, I will rebase and add the v2.10.0 copies for AKS and the sidebar too |
||
|
|
||
| ## Verify your installation | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,205 @@ | ||
| --- | ||
| title: HAMi on Microsoft Azure (AKS) | ||
| sidebar_label: Microsoft Azure (AKS) | ||
| --- | ||
|
|
||
| This guide provides step-by-step instructions for deploying and running HAMi on **Azure Kubernetes Service (AKS)** to enable GPU sharing and resource virtualization across NVIDIA GPU node pools. | ||
|
|
||
| ## Overview | ||
|
|
||
| Azure Kubernetes Service offers several GPU-enabled VM series (such as the `NCv3`, `NCasT4_v3`, `NVadsA10_v5`, and `NDv4` families). By default, Kubernetes assigns whole physical GPUs to single containers. HAMi allows multiple pods to share the same physical GPU with fine-grained memory and compute core isolation on AKS. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| Before deploying HAMi on AKS, ensure you have: | ||
|
|
||
| - **Azure CLI (`az`)**: Installed and authenticated (`az login`). | ||
| - **`kubectl`** and **`helm` (v3.0+)**: Installed locally. | ||
| - An existing AKS cluster with a GPU-enabled node pool (or follow the steps below to create one). | ||
| - Kubernetes server version `>= 1.23`. | ||
|
|
||
| ## Step 1: Create a GPU Node Pool in AKS | ||
|
|
||
| If your cluster does not yet have GPU nodes, add a GPU node pool using the Azure CLI. | ||
|
|
||
| For example, to create a node pool with NVIDIA V100 GPUs (`Standard_NC6s_v3`): | ||
|
|
||
| ```bash | ||
| az aks nodepool add \ | ||
| --resource-group <MY_RESOURCE_GROUP> \ | ||
| --cluster-name <MY_AKS_CLUSTER> \ | ||
| --name gpunodes \ | ||
| --node-count 2 \ | ||
| --node-vm-size Standard_NC6s_v3 \ | ||
| --node-taints sku=gpu:NoSchedule \ | ||
| --labels gpu=on | ||
| ``` | ||
|
|
||
| :::note Common Azure GPU VM sizes include: | ||
|
|
||
| - `Standard_NC6s_v3` (1x NVIDIA Tesla V100 16GB) | ||
| - `Standard_NC4as_T4_v3` (1x NVIDIA Tesla T4 16GB) | ||
| - `Standard_NV6ads_A10_v5` (1x NVIDIA A10 24GB) | ||
| - `Standard_ND96amsr_A100_v4` (8x NVIDIA A100 80GB) | ||
|
|
||
| ::: | ||
|
|
||
| ### Label Your Nodes | ||
|
|
||
| HAMi monitors and schedules workloads only on nodes with the label `gpu=on`. If your node pool was created without this label, add it manually: | ||
|
|
||
| ```bash | ||
| kubectl label nodes <node-name> gpu=on | ||
| ``` | ||
|
|
||
| ### Install NVIDIA Drivers | ||
|
|
||
| Ensure NVIDIA drivers are installed on the GPU nodes. You can either: | ||
|
|
||
| - Use AKS automated GPU driver provisioning (`--enable-gpu-driver-daemonset` on supported Azure Linux / Ubuntu images). | ||
| - Or install the [NVIDIA GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html) with driver installation enabled. | ||
|
Comment on lines
+59
to
+60
There was a problem hiding this comment. 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -eu
for f in \
docs/installation/platforms/aks.md \
i18n/zh/docusaurus-plugin-content-docs/current/installation/platforms/aks.md \
versioned_docs/version-v2.9.0/installation/platforms/aks.md \
i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/installation/platforms/aks.md
do
echo "===== $f ====="
sed -n '45,70p' "$f"
done
echo "===== repository occurrences ====="
rg -n --glob '*.md' --glob '*.mdx' \
-- '--enable-gpu-driver-daemonset|managedGpu|managed-gpu|gpu-driver-daemonset' \
docs i18n versioned_docs 2>/dev/null || true
echo "===== Microsoft Learn page excerpts ====="
python3 - <<'PY'
from urllib.request import Request, urlopen
from html.parser import HTMLParser
import re
url = "https://learn.microsoft.com/en-us/azure/aks/aks-managed-gpu-nodes"
req = Request(url, headers={"User-Agent": "Mozilla/5.0"})
html = urlopen(req, timeout=20).read().decode("utf-8", "replace")
text = re.sub(r"<script\b[^>]*>.*?</script>", " ", html, flags=re.I | re.S)
text = re.sub(r"<style\b[^>]*>.*?</style>", " ", text, flags=re.I | re.S)
text = re.sub(r"<[^>]+>", " ", text)
text = re.sub(r"\s+", " ", text)
for term in ("--enable-gpu-driver-daemonset", "managedGpu", "gpu-driver", "az aks nodepool add"):
print(f"\nTERM: {term}")
found = list(re.finditer(re.escape(term), text, flags=re.I))
for m in found[:5]:
print(text[max(0, m.start()-350):m.end()+700])
PYRepository: Project-HAMi/website Length of output: 7079 🌐 Web query:
💡 Result: There is no standard Kubernetes flag or widely recognized parameter named --enable-gpu-driver-daemonset. The term appears to be a misconfiguration or a confusion with platform-specific management tools or operator settings [1][2][3][4][5][6]. In managed Kubernetes environments like Azure Kubernetes Service (AKS), GPU management is controlled via specific CLI flags such as --enable-managed-gpu, which allows you to toggle between fully managed GPU stacks, driver-only installation, or custom (bring-your-own) configurations [1]. If you are using the NVIDIA GPU Operator, components such as the driver and vGPU manager are deployed as DaemonSets automatically by the operator [2][3][6]. You do not typically manage these via a command-line flag named --enable-gpu-driver-daemonset. Instead, control is handled through Helm values (e.g., driver.enabled=true) [5] or by configuring the operator's custom resources to enable or disable specific components [2][5]. If you encountered this term in a specific script or third-party tool, it is likely a custom flag defined by that specific environment rather than a native Kubernetes or standard NVIDIA GPU Operator command [2][4][6]. Citations:
🏁 Script executed: #!/bin/bash
set -eu
url='https://learn.microsoft.com/en-us/azure/aks/aks-managed-gpu-nodes'
tmp=$(mktemp)
trap 'rm -f "$tmp"' EXIT
curl -k -L --fail --silent --show-error "$url" -o "$tmp"
python3 - "$tmp" <<'PY'
import re, sys
from html import unescape
html = open(sys.argv[1], encoding="utf-8").read()
# Preserve code/pre contents as text, then strip remaining markup.
html = re.sub(r"<(script|style)\b[^>]*>.*?</\1>", " ", html, flags=re.I | re.S)
html = re.sub(r"</?(?:code|pre)\b[^>]*>", "\n", html, flags=re.I)
text = re.sub(r"<[^>]+>", " ", html)
text = unescape(re.sub(r"\s+", " ", text))
terms = [
"--enable-gpu-driver-daemonset",
"--enable-managed-gpu",
"managed GPU",
"driver-only",
"bring-your-own",
"az aks nodepool add",
]
for term in terms:
print(f"\n===== {term} =====")
matches = list(re.finditer(re.escape(term), text, flags=re.I))
for match in matches[:8]:
print(text[max(0, match.start()-500):match.end()+1000])
PYRepository: Project-HAMi/website Length of output: 36285 🌐 Web query:
💡 Result: The flags --enable-managed-gpu and --gpu-driver are used in Azure Kubernetes Service (AKS) when creating node pools with GPU-enabled virtual machines [1]. They determine how the NVIDIA GPU driver and the supporting Kubernetes GPU software stack (such as device plugins and monitoring tools) are managed by the platform [1]. These flags work together to define the installation profile [1]: Full managed stack: When --enable-managed-gpu is set to true, AKS installs and manages the entire GPU stack, including the NVIDIA driver, device plugin, DCGM metrics exporter, and GPU health monitoring [1]. Driver only: When --enable-managed-gpu is set to false (which is the default if neither flag is specified), AKS only installs the NVIDIA GPU driver [1]. In this configuration, the user is responsible for installing and managing the device plugin, metrics exporter, and health monitoring [1]. None (Bring Your Own): By setting --enable-managed-gpu=false and --gpu-driver None, AKS will not install any GPU components, allowing the user to manage the entire stack independently [1]. It is important to note that these settings are immutable after the node pool is created; to change the management profile, a new node pool must be created [1]. Additionally, if --enable-managed-gpu is set to true, AKS will enforce the installation of the driver regardless of the --gpu-driver setting, as the managed stack requires it [1]. Citations: Use supported AKS GPU profiles in all four documents. 📍 Affects 4 files
🤖 Prompt for AI Agents |
||
|
|
||
| ## Step 2: Prevent Device Plugin Conflicts | ||
|
|
||
| AKS clusters may automatically deploy the default NVIDIA Kubernetes Device Plugin (`nvidia-device-plugin-daemonset`). | ||
|
|
||
| If both the default NVIDIA device plugin and HAMi's device plugin run simultaneously, both will attempt to register `nvidia.com/gpu` with the kubelet, causing double-registration conflicts. | ||
|
|
||
| 1. Check if the default NVIDIA device plugin DaemonSet is running: | ||
|
|
||
| ```bash | ||
| kubectl get ds -n kube-system -l app=nvidia-device-plugin-daemonset | ||
| ``` | ||
|
|
||
| 1. If present, disable or remove the default DaemonSet so that HAMi can act as the sole GPU resource registrar: | ||
|
|
||
| ```bash | ||
| kubectl delete ds <daemonset-name> -n kube-system | ||
| ``` | ||
|
|
||
| ## Step 3: Install HAMi via Helm | ||
|
|
||
| ### Add the HAMi Helm Repository | ||
|
|
||
| ```bash | ||
| helm repo add hami-charts https://project-hami.github.io/HAMi/ | ||
| helm repo update | ||
| ``` | ||
|
|
||
| ### Identify Your Kubernetes Version | ||
|
|
||
| Get your AKS cluster Kubernetes version: | ||
|
|
||
| ```bash | ||
| kubectl version --short | ||
| ``` | ||
|
Comment on lines
+93
to
+95
There was a problem hiding this comment. 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -e
for f in \
docs/installation/platforms/aks.md \
i18n/zh/docusaurus-plugin-content-docs/current/installation/platforms/aks.md \
versioned_docs/version-v2.9.0/installation/platforms/aks.md \
i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/installation/platforms/aks.md
do
echo "=== $f ==="
sed -n '80,100p' "$f"
done
echo "=== kubectl version references ==="
rg -n --glob '*.md' 'kubectl version( --short)?' docs i18n versioned_docs | head -80
echo "=== repository state ==="
git status --shortRepository: Project-HAMi/website Length of output: 8236 🌐 Web query:
💡 Result: The --short flag for the kubectl version command has been removed and is no longer supported in current versions of Kubernetes [1][2]. The --short flag was deprecated starting in Kubernetes v1.26 and was removed entirely in v1.28 [1][2]. In modern versions of kubectl, the default output of the kubectl version command now provides the concise, simplified information that the --short flag previously offered [3][2]. Consequently, no replacement flag is required to achieve that output format [2]. If you are writing scripts or require structured data, you can use the --output flag with either json or yaml to obtain machine-readable version information (e.g., kubectl version --output=json) [3][4][1]. Attempting to use the --short flag in current versions will result in an "unknown flag" error [1][2]. Citations:
🏁 Script executed: #!/bin/bash
set -e
python3 - <<'PY'
from pathlib import Path
from urllib.request import urlopen
files = [
Path("docs/installation/platforms/aks.md"),
Path("i18n/zh/docusaurus-plugin-content-docs/current/installation/platforms/aks.md"),
Path("versioned_docs/version-v2.9.0/installation/platforms/aks.md"),
Path("i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/installation/platforms/aks.md"),
]
for path in files:
text = path.read_text()
matches = text.count("kubectl version --short")
print(f"{path}: {matches} unsupported command occurrence(s)")
source_url = "https://raw.githubusercontent.com/kubernetes/kubectl/master/pkg/cmd/version/version.go"
source = urlopen(source_url, timeout=10).read().decode()
print(f"current kubectl source contains --short flag declaration: {'--short' in source}")
print(f"current kubectl source contains output flag declaration: {'--output' in source or 'output' in source}")
PYRepository: Project-HAMi/website Length of output: 2849 🌐 Web query:
💡 Result: The --short flag for the kubectl version command was removed in Kubernetes v1.28. Starting with that release, the flag is no longer supported, and users are expected to use the standard output or alternative formatting options like --output=yaml or --output=json to retrieve version information [1]. Citations: Replace 📍 Affects 4 files
🤖 Prompt for AI Agents |
||
|
|
||
| ### Create AKS Custom Values | ||
|
|
||
| Create a file named `custom-aks-values.yaml`. Make sure to configure the `tolerations` matching your AKS GPU node taints (`sku=gpu:NoSchedule`), set `scheduler.kubeScheduler.image.tag` to match your cluster version, and optionally enable scheduler High Availability: | ||
|
|
||
| ```yaml | ||
| scheduler: | ||
| leaderElect: true | ||
| replicaCount: 2 | ||
| kubeScheduler: | ||
| image: | ||
| # Set tag to match your AKS Kubernetes server version (e.g. v1.29.0) | ||
| tag: "v1.29.0" | ||
|
Comment on lines
+102
to
+108
There was a problem hiding this comment. 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -eu
printf '%s\n' '--- affected documentation snippets ---'
for file in \
docs/installation/platforms/aks.md \
i18n/zh/docusaurus-plugin-content-docs/current/installation/platforms/aks.md \
versioned_docs/version-v2.9.0/installation/platforms/aks.md \
i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/installation/platforms/aks.md
do
printf '\n--- %s ---\n' "$file"
sed -n '88,112p' "$file"
done
printf '\n--- HAMi v2.9.0 chart values ---\n'
curl -fsSL https://raw.githubusercontent.com/Project-HAMi/HAMi/v2.9.0/charts/hami/values.yaml |
awk '
/^scheduler:/ { in_scheduler=1 }
in_scheduler && /^volcano:/ { exit }
in_scheduler { print }
' | sed -n '1,100p'Repository: Project-HAMi/website Length of output: 6835 🏁 Script executed: #!/bin/bash
set -eu
printf '%s\n' '--- chart files that reference scheduler replica settings ---'
curl -fsSL 'https://api.github.com/repos/Project-HAMi/HAMi/git/trees/v2.9.0?recursive=1' |
python3 -c '
import json, sys
tree = json.load(sys.stdin)["tree"]
for item in tree:
path = item.get("path", "")
if "chart" in path.lower() and path.endswith((".yaml", ".yml", ".tpl")):
print(path)
' |
while IFS= read -r path; do
content=$(curl -fsSL "https://raw.githubusercontent.com/Project-HAMi/HAMi/v2.9.0/$path")
if printf '%s\n' "$content" | grep -Eq 'replicaCount|\.Values\.scheduler\.replicas|scheduler\.replicas|replicas:'; then
printf '\n--- %s ---\n' "$path"
printf '%s\n' "$content" | grep -nE -C 3 'replicaCount|\.Values\.scheduler\.replicas|scheduler\.replicas|replicas:'
fi
done
printf '\n--- all replica settings in the affected docs ---\n'
rg -n -C 2 'replicaCount|replicas' \
docs/installation/platforms/aks.md \
i18n/zh/docusaurus-plugin-content-docs/current/installation/platforms/aks.md \
versioned_docs/version-v2.9.0/installation/platforms/aks.md \
i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/installation/platforms/aks.mdRepository: Project-HAMi/website Length of output: 2546 🏁 Script executed: #!/bin/bash
set -eu
printf '%s\n' '--- AKS installation references ---'
for file in \
docs/installation/platforms/aks.md \
i18n/zh/docusaurus-plugin-content-docs/current/installation/platforms/aks.md \
versioned_docs/version-v2.9.0/installation/platforms/aks.md \
i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/installation/platforms/aks.md
do
printf '\n--- %s ---\n' "$file"
rg -n -C 4 'helm|hami|chart|version|replicaCount|replicas' "$file"
done
printf '\n--- current HAMi chart scheduler replica configuration ---\n'
tree_json=$(curl -fsSL 'https://api.github.com/repos/Project-HAMi/HAMi/git/trees/main?recursive=1')
printf '%s\n' "$tree_json" |
python3 -c '
import json, sys
for item in json.load(sys.stdin)["tree"]:
path = item.get("path", "")
if path in ("charts/hami/values.yaml", "charts/hami/templates/scheduler/deployment.yaml"):
print(path)
' |
while IFS= read -r path; do
printf '\n--- %s ---\n' "$path"
curl -fsSL "https://raw.githubusercontent.com/Project-HAMi/HAMi/main/$path" |
grep -n -E -C 3 'replicaCount|scheduler\.replicas|replicas:'
doneRepository: Project-HAMi/website Length of output: 10350 🏁 Script executed: #!/bin/bash
set -eu
printf '%s\n' '--- HAMi repository metadata ---'
curl -fsSL https://api.github.com/repos/Project-HAMi/HAMi |
python3 -c 'import json,sys; d=json.load(sys.stdin); print("default_branch:", d.get("default_branch"))'
printf '\n%s\n' '--- published Helm chart index ---'
curl -fsSL https://project-hami.github.io/HAMi/index.yaml |
sed -n '1,100p'Repository: Project-HAMi/website Length of output: 3171 Rename 📍 Affects 4 files
🤖 Prompt for AI Agents |
||
|
|
||
| devicePlugin: | ||
| tolerations: | ||
| - key: "sku" | ||
| operator: "Equal" | ||
| value: "gpu" | ||
| effect: "NoSchedule" | ||
| - key: "nvidia.com/gpu" | ||
| operator: "Exists" | ||
| effect: "NoSchedule" | ||
| ``` | ||
|
|
||
| ### Deploy the Chart | ||
|
|
||
| Deploy HAMi to the `kube-system` namespace: | ||
|
|
||
| ```bash | ||
| helm install hami hami-charts/hami \ | ||
| -f custom-aks-values.yaml \ | ||
| -n kube-system | ||
| ``` | ||
|
|
||
| ## Step 4: Verify Your Installation | ||
|
|
||
| ### 1. Verify Pod Status | ||
|
|
||
| Check that the HAMi scheduler and device plugin pods are in `Running` state: | ||
|
|
||
| ```bash | ||
| kubectl get pods -n kube-system -l app.kubernetes.io/name=hami | ||
| ``` | ||
|
|
||
| Expected output: | ||
|
|
||
| ```text | ||
| NAME READY STATUS RESTARTS AGE | ||
| hami-device-plugin-xxxxx 1/1 Running 0 2m | ||
| hami-device-plugin-yyyyy 1/1 Running 0 2m | ||
| hami-scheduler-6d8b97bc49-abcde 1/1 Running 0 2m | ||
| hami-scheduler-6d8b97bc49-fghij 1/1 Running 0 2m | ||
| ``` | ||
|
|
||
| ### 2. Verify Node Extended Resources | ||
|
|
||
| Check that your GPU node advertises HAMi virtual GPU resources (`nvidia.com/gpumem` and `nvidia.com/gpucores`): | ||
|
|
||
| ```bash | ||
| kubectl describe node <gpu-node-name> | grep -E "(nvidia.com/gpu|nvidia.com/gpumem|nvidia.com/gpucores):" | ||
| ``` | ||
|
|
||
| Expected output shows `nvidia.com/gpu`, `nvidia.com/gpumem` (in MiB), and `nvidia.com/gpucores` (percentage): | ||
|
|
||
| ```text | ||
| nvidia.com/gpu: 1 | ||
| nvidia.com/gpumem: 16280 | ||
| nvidia.com/gpucores: 100 | ||
| ``` | ||
|
Comment on lines
+151
to
+165
There was a problem hiding this comment. 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -eu
for f in \
docs/installation/platforms/aks.md \
i18n/zh/docusaurus-plugin-content-docs/current/installation/platforms/aks.md \
versioned_docs/version-v2.9.0/installation/platforms/aks.md \
i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/installation/platforms/aks.md
do
echo "===== $f ====="
sed -n '135,175p' "$f"
done
echo "===== matching AKS verification text ====="
rg -n -C 4 'Verify Node Extended Resources|验证节点扩展资源|gpumem|gpucores|describe node' \
docs/installation/platforms/aks.md \
i18n/zh/docusaurus-plugin-content-docs/current/installation/platforms/aks.md \
versioned_docs/version-v2.9.0/installation/platforms/aks.md \
i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/installation/platforms/aks.mdRepository: Project-HAMi/website Length of output: 20988 🌐 Web query:
💡 Result: Project-HAMi issue Citations:
🏁 Script executed: #!/bin/bash
set -eu
echo "===== HAMi issue 646 metadata and body ====="
curl -fsSL https://api.github.com/repos/Project-HAMi/HAMi/issues/646 |
jq '{number, state, title, html_url, body}'
echo "===== referenced HAMi documentation ====="
for url in \
https://project-hami.io/docs/core-concepts/gpu-virtualization \
https://project-hami.io/tutorials/labs/gpu-partitioning
do
echo "--- $url ---"
curl -fsSL "$url" | tr '\n' ' ' |
sed 's/<[^>]*>/ /g' |
sed 's/[[:space:]]\\+/ /g' |
grep -oE '.{0,220}(node-nvidia-register|status.allocatable|gpumem|gpucores|kubectl get node).{0,320}' |
head -20 || true
doneRepository: Project-HAMi/website Length of output: 26223 🌐 Web query:
💡 Result: The command kubectl describe node gpumem gpucores is not a standard Kubernetes command. It appears to be an attempt to query node-level GPU resources, specifically memory (gpumem) and compute cores (gpucores) [1]. In Kubernetes, standard device plugins—such as the NVIDIA device plugin—are often limited by the Device Plugin API, which typically allows reporting only a single resource type per plugin instance [1]. As a result, detailed metrics like GPU memory or compute cores are frequently not visible directly under the Node's Capacity or Allocatable fields in the output of kubectl describe node [1]. To view this information, systems that manage GPU virtualization (such as HAMi or Volcano) often use the following methods [1]: 1. Node Annotations: Detailed GPU metrics (e.g., memory, model, compute power) are frequently stored as annotations on the node object. You can view these by running kubectl describe node and inspecting the Annotations section [1]. 2. Custom Resource Names: Some plugins create multiple device plugin instances to register individual resource types (e.g., volcano.sh/vgpu-memory or volcano.sh/vgpu-cores) with the kubelet. If your cluster is configured this way, these resources will appear under Capacity and Allocatable [1]. To find where your specific GPU metrics are stored, you can inspect the node object directly: kubectl get node -o yaml This will display the full node configuration, including annotations and status fields, where these values are typically recorded [1]. Citations: 🌐 Web query:
💡 Result: To check custom extended resources, such as those used for GPU management (e.g., nvidia.com/gpu), you use the kubectl describe node command [1][2]. Extended resources are advertised by the node as part of its capacity and allocatable status [2][3]. You can view these resources by running: kubectl describe node In the output, look for the Capacity and Allocatable sections. Custom resources will be listed alongside standard resources like cpu and memory [1][4][5]. Example output structure: Capacity: cpu: 2 memory: 2049008Ki nvidia.com/gpu: 4 Allocatable: cpu: 2 memory: 2049008Ki nvidia.com/gpu: 4 Key points regarding extended resources: - Resource Naming: Extended resources must use a fully-qualified domain name format (vendor-domain/resource-name), such as nvidia.com/gpu [6]. - Integer Requirements: Extended resources are restricted to integer values and cannot be overcommitted [1][6]. - How they are registered: These are typically managed by Device Plugins (like the official NVIDIA device plugin) that run on the node and update the Kubernetes API server with the available capacity [3][7][6]. - GPU Memory and Cores: While Kubernetes tracks resources at the device level (e.g., nvidia.com/gpu), it does not natively track sub-device metrics like specific GPU memory or core counts in the Node Allocatable field unless the device plugin specifically advertises them as separate extended resources (e.g., via time-slicing or MIG configurations) [7][8]. For detailed monitoring of GPU performance metrics like memory or compute utilization, you must use external monitoring tools or the metrics exported by the NVIDIA device plugin, as the standard kubectl describe node command only shows the available count of the advertised resource [4][7]. If you need to verify these values programmatically, you can use jsonpath: kubectl get node -o jsonpath='{.status.allocatable}' Citations:
🏁 Script executed: #!/bin/bash
set -eu
echo "===== repository references for HAMi node verification ====="
rg -n -C 3 \
'hami\.io/node-nvidia-register|status\.allocatable|metadata\.annotations|jsonpath=.*nvidia' \
--glob '*.md' --glob '*.mdx' --glob '*.yaml' --glob '*.yml' . |
head -240
echo "===== exact annotation-key references ====="
rg -n \
'hami\.io/node-nvidia-register|nvidia\.com/gpumem|nvidia\.com/gpucores' \
--glob '*.md' --glob '*.mdx' . |
head -240Repository: Project-HAMi/website Length of output: 50378 🏁 Script executed: #!/bin/bash
set -eu
python3 - <<'PY'
import json
# Model the node shape documented by HAMi: one allocatable GPU count and
# detailed device data in the node annotation.
node = {
"metadata": {
"annotations": {
"hami.io/node-nvidia-register": "GPU-example,10,49140,100,NVIDIA-L40S,0,true"
}
},
"status": {
"allocatable": {
"nvidia.com/gpu": "10"
}
},
}
annotation_key = "hami.io/node-nvidia-register"
allocatable_key = "nvidia.com/gpu"
print("annotation present:", annotation_key in node["metadata"]["annotations"])
print("allocatable GPU count:", node["status"]["allocatable"].get(allocatable_key))
print("allocatable gpumem present:", "nvidia.com/gpumem" in node["status"]["allocatable"])
print("allocatable gpucores present:", "nvidia.com/gpucores" in node["status"]["allocatable"])
PYRepository: Project-HAMi/website Length of output: 279 Correct the HAMi node verification in all four AKS pages. Check 📍 Affects 4 files
🤖 Prompt for AI Agents |
||
|
|
||
| ## Step 5: Run a Smoke Test Workload | ||
|
|
||
| Submit a test Pod that requests a fraction of the GPU memory (e.g., 4000 MiB) and 40% of GPU compute cores: | ||
|
|
||
| ```yaml | ||
| apiVersion: v1 | ||
| kind: Pod | ||
| metadata: | ||
| name: aks-gpu-test | ||
| spec: | ||
| restartPolicy: OnFailure | ||
| tolerations: | ||
| - key: "sku" | ||
| operator: "Equal" | ||
| value: "gpu" | ||
| effect: "NoSchedule" | ||
| containers: | ||
| - name: cuda-test | ||
| image: nvidia/cuda:12.2.0-base-ubuntu22.04 | ||
| command: ["nvidia-smi"] | ||
| resources: | ||
| limits: | ||
| nvidia.com/gpu: 1 | ||
| nvidia.com/gpumem: 4000 | ||
| nvidia.com/gpucores: 40 | ||
| ``` | ||
|
|
||
| Apply the pod manifest and inspect logs: | ||
|
|
||
| ```bash | ||
| kubectl apply -f aks-gpu-test.yaml | ||
| kubectl logs aks-gpu-test | ||
| ``` | ||
|
|
||
| ## AKS-Specific Gotchas & Troubleshooting | ||
|
|
||
| - **Node Taint Mismatches**: If `hami-device-plugin` pods remain in `Pending` state, check your GPU node taints (`kubectl describe node <node-name>`) and ensure all taints are present in `devicePlugin.tolerations` within `values.yaml`. | ||
| - **Node Auto-scaling**: If you enable the AKS Cluster Autoscaler on GPU node pools, new nodes will automatically receive the node labels and taints defined during node pool creation. Ensure `gpu=on` is set on the nodepool configuration. | ||
| - **Image Pull Issues**: If your AKS cluster is in a restricted or private virtual network, consider mirroring HAMi container images to an Azure Container Registry (ACR) and configuring `image.repository` in `values.yaml`. | ||
|
Comment on lines
+203
to
+205
There was a problem hiding this comment. 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -eu
printf '%s\n' '--- candidate files ---'
git ls-files | rg '(^|/)(aks\.md|values\.yaml|custom-aks-values\.yaml)$|installation/platforms/aks\.md'
printf '%s\n' '--- AKS references ---'
rg -n -C 3 'custom-aks-values|values\.yaml|image\.repository|global\.imageRegistry|devicePlugin\.image|helm install|helm upgrade' \
docs/installation/platforms/aks.md \
i18n/zh/docusaurus-plugin-content-docs/current/installation/platforms/aks.md \
versioned_docs/version-v2.9.0/installation/platforms/aks.md \
i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/installation/platforms/aks.md \
charts 2>/dev/null || true
printf '%s\n' '--- chart value declarations ---'
rg -n -C 3 'imageRegistry|repository|devicePlugin:' charts --glob 'values.yaml' --glob '*.yaml' 2>/dev/null | head -240Repository: Project-HAMi/website Length of output: 10127 🏁 Script executed: #!/bin/bash
set -eu
printf '%s\n' '--- current English custom values and troubleshooting ---'
sed -n '90,130p;195,208p' docs/installation/platforms/aks.md
printf '%s\n' '--- versioned English custom values and troubleshooting ---'
sed -n '90,130p;192,206p' versioned_docs/version-v2.9.0/installation/platforms/aks.md
printf '%s\n' '--- v2.9.0 chart values: image-related keys and tolerations ---'
url='https://raw.githubusercontent.com/Project-HAMi/HAMi/v2.9.0/charts/hami/values.yaml'
curl --fail --silent --show-error "$url" |
rg -n -C 5 '^(global:|image:|devicePlugin:)|imageRegistry|repository|tolerations|gpu=on' |
head -260Repository: Project-HAMi/website Length of output: 11693 Use the documented values file and chart keys.
📍 Affects 4 files
🤖 Prompt for AI Agents |
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,6 @@ | ||
| --- | ||
| title: HAMi on AWS | ||
| sidebar_label: Amazon Web Services (AWS) | ||
|
Contributor
There was a problem hiding this comment. #804 does this same aws move with a different sidebar label, cloud platforms vs platform guides. duplicate work, close one or rebase on the other.
Contributor
Author
There was a problem hiding this comment. Understood |
||
| translated: true | ||
| --- | ||
|
|
||
|
|
@@ -26,7 +27,7 @@ tar xf $(pwd)/* && find $(pwd) -maxdepth 1 -type f -delete | |
| helm install --generate-name --namespace <ENTER_NAMESPACE_HERE> ./* | ||
| ``` | ||
|
|
||
| You can customize the installation by adjusting the [configuration](../../userguide/configure.md). | ||
|
|
||
| ## Install with AWS Add-on | ||
|
|
||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Keep the platform list and platform links consistent.
Each online-installation page mentions AWS and AKS but links only to the AKS guide. Link both concrete guides, or link the Platform Guides index.
docs/installation/online-installation.md#L33-L33: link AWS and AKS guides.i18n/zh/docusaurus-plugin-content-docs/current/installation/online-installation.md#L34-L34: mirror both platform links in Chinese.i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/installation/online-installation.md#L34-L34: mirror both versioned platform links in Chinese.versioned_docs/version-v2.9.0/installation/online-installation.md#L33-L33: link both versioned platform guides.📍 Affects 4 files
docs/installation/online-installation.md#L33-L33(this comment)i18n/zh/docusaurus-plugin-content-docs/current/installation/online-installation.md#L34-L34i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/installation/online-installation.md#L34-L34versioned_docs/version-v2.9.0/installation/online-installation.md#L33-L33🤖 Prompt for AI Agents