Enhance troubleshooting guide for HAMi installation

[RohiniVishu]

Added troubleshooting documentation for HAMi installation and scheduler testing on GPU-less nodes, including the real NVIDIA device-plugin failure, the mock-device-plugin health-check bootstrap issue, and an observed scheduler image-pull/TLS failure.

What type of PR is this?

/kind documentation

--

What this PR does / why we need it:

This PR updates documentation to enable HAMi development on GPU-less nodes by addressing mock device plugin health checks and providing troubleshooting for the real device plugin. Key findings include fixing mock-device-plugin bootstrap failures by seeding non-zero vendor count resources and documenting prerequisites for the real NVIDIA plugin.
Troubleshooting and the Workarounds around it:

• GPU-less Testing: Use mock-device-plugin for scheduler testing on nodes without physical GPUs.
• Fix Unhealthy Mock Devices: If CheckHealthy() fails due to missing node.Status.Capacity, manually patch the node to add a non-zero vendor count and apply the mock-device-plugin annotation.
• Real Plugin Failure: If the real plugin enters CrashLoopBackOff on GPU-less nodes, verify the host-side NVIDIA driver stack or use the mock plugin.
• Diagnostic Commands: Review node capacity (kubectl get node -o yaml) and mock-device registration annotations.

The investigation was performed on a Fedora Linux machine with an Intel Core i3-5005U, Intel HD Graphics 5500 integrated graphics, 16 GB RAM, and a single-node kind cluster running Kubernetes v1.36.1. HAMi chart hami-charts/hami with image docker.io/projecthami/hami:v2.9.0 was used, with mock-device-plugin cloned from main at the time of testing.

The standard HAMi Quick Start exposed the following findings in detail:

1. Real HAMi device plugin fails on GPU-less nodes

On a node without a supported NVIDIA GPU and NVIDIA container runtime, the real device plugin entered CrashLoopBackOff with errors including:

Incompatible strategy detected auto
If this is a GPU node, did you configure the NVIDIA Container Toolkit?
If this is not a GPU node, you should set up a toleration or nodeSelector
to only deploy this plugin on GPU nodes
error starting plugins: ... invalid device discovery strategy

The troubleshooting documentation now explains that the real NVIDIA device plugin requires the corresponding host-side NVIDIA driver/runtime stack and that GPU-less contributors should use the mock device plugin for scheduler testing instead.

2. mock-device-plugin can repeatedly report devices as unhealthy

After deploying the official mock-device-plugin, the DaemonSet ran but repeatedly logged:

device NVIDIA is unhealthy on this node
device Ascend910A is unhealthy on this node
device Ascend910B2 is unhealthy on this node

No corresponding GPU-related resources appeared in the node's Allocatable resources. Source inspection traced this behavior to CheckHealthy() in:

internal/pkg/api/device/device.go

which checks whether the node already has a non-zero count resource in node.Status.Capacity:

func CheckHealthy(n *corev1.Node, cardResourceName string) bool {
    capacity, exists := n.Status.Capacity[corev1.ResourceName(cardResourceName)]
    if !exists {
        return false
    }
    return !capacity.IsZero()
}

The device implementation returns before registering resources when this check fails.This creates a bootstrap dependency on a fresh mock-only node:

CheckHealthy() -> count resource absent from node.Status.Capacity -> device considered unhealthy -> resource registration does not proceed -> count resource remains absent-> CheckHealthy() fails again

This is documented as a testing workaround, not a production configuration or a code-level fix [it worked for me but must be tested with different hardware]

3. Scheduler pod image-pull/TLS failure observed during validation

During the same installation, the scheduler pod also entered ImagePullBackOff while pulling:

registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.36.1

with:

tls: failed to verify certificate:
x509: certificate signed by unknown authority

The HAMi scheduler extender image from Docker Hub pulled successfully in the same environment. This failure was reproduced in the validation environment, but its broader scope is not established since it may be environment-specific, so the documentation records the observation and diagnostic commands without presenting it as a confirmed HAMi-wide issue or providing an unverified universal workaround. A separate investigation in ipsitapp8/hami-placement-lab identified a different mock-device-plugin issue involving the scheduler node lock during Allocate(). That issue occurs at a later stage, after device registration, and is separate from the registration/bootstrap problem documented here. This investigation independently reproduced the behavior on Fedora + kind and traced the bootstrap condition to the CheckHealthy() logic that occurs before it.

docs/troubleshooting/troubleshooting.md now includes:

• GPU-less HAMi device-plugin troubleshooting & explanation of the NVIDIA device-plugin failure and prerequisites;
• mock-device-plugin unhealthy-device symptoms;
• diagnostic commands for node Capacity, Allocatable, and registration annotations;
• explanation of the count-resource health gate;
• the GPU-less mock-device bootstrap workaround;
• verification commands and expected resources;
• limitations of mock-based scheduler testing;
• the observed scheduler image-pull/TLS failure and its current environment-specific status;

Another thing observed: device health could be tracked independently of pre-existing node capacity rather than using node.Status.Capacity as the bootstrap health gate. This was not implemented or tested in this documentation PR and is included only as a maintainer-facing observation for future code-level work.

Which issue(s) this PR fixes:

Fixes #656

Related:

• Mock devices never report healthy — Allocatable never shows registered GPU resources mock-device-plugin#21
• Unable to see GPU allocation using mock-device-plugin on Kind cluster (Mac) mock-device-plugin#16
• mock device plugin show "device xxx is unhealthy on this node" mock-device-plugin#14

Checklist:

• npm run lint and npm run format:check pass
• npm run build succeeds for both en and zh
• Chinese translation updated if English docs changed (or noted why not)
• Commits are signed off (git commit -s)

[netlify]

✅ Deploy Preview for project-hami ready!

Name	Link
🔨 Latest commit	844670a
🔍 Latest deploy log	https://app.netlify.com/projects/project-hami/deploys/6a8499a5ff4cbc00081da29f
😎 Deploy Preview	https://deploy-preview-791--project-hami.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
🤖 Make changes	Run an agent on this branch

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[hami-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: RohiniVishu
Once this PR has been reviewed and has the lgtm label, please assign archlitchi for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

Important

Review available on request

• 🔍 Trigger review

Reviews should be triggered manually for repositories with fewer than 10 stars. Select Trigger review above or comment @coderabbitai review to review the latest changes. For a full review, comment @coderabbitai full review.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 1e7d74b9-1abd-490d-b90c-322a04767602

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[RohiniVishu]

@rootsongjc @windsonsea Please take a look! troubleshooting documented with commands and time, will share if required by maintainers

[mesutoezdil]

the checkhealthy analysis is solid and matches the linked mock-device-plugin issues. but the text needs a cleanup pass and the checks were not run: all four checklist boxes are empty, the docs health ci did not run, and lint would likely fail on the issues below. fixes #656 closes the lfx umbrella issue, use part of. no zh update and it is not noted.

[mesutoezdil]

editor leftover: [code block 1 and 2] renders as literal text, and later the page says by doing codeblock 1 and 2. remove both and just repeat the commands or link the section.

[mesutoezdil]

many missing spaces after periods in this pr: testing.The, host.A, hardware.This, environment.This, plus doesnt. one proofread pass please.

[mesutoezdil]

this environment block is tab separated plain text, not a markdown table, and the file ends with a stray --. format it as a real table.

[mesutoezdil]

heading has a trailing colon and space, markdownlint will flag it.

[mesutoezdil]

coordination: #773 covers the same gpu-less mock topic, #788 documents the same node-nvidia-register annotation fix, and #741, #779, #783 also append to this file. worth merging efforts instead of five parallel sections.