Add initial single-sandbox-executing-at-a-time support for Hypervisor.framework - #1674
Open
syntactically wants to merge 42 commits into
Open
Add initial single-sandbox-executing-at-a-time support for Hypervisor.framework#1674syntactically wants to merge 42 commits into
syntactically wants to merge 42 commits into
Conversation
syntactically
requested review from
andreiltd,
danbugs,
dblnz,
devigned,
jprendes,
jsturtevant,
ludfjig,
simongdavies and
squillace
as code owners
July 21, 2026 15:27
syntactically
added
kind/enhancement
Contributor
There was a problem hiding this comment.
Pull request overview
Adds initial Hypervisor.framework (HVF) support to hyperlight-host on macOS, along with a snapshot ABI/version bump and memory-layout/page-size changes needed for platforms with 16K host pages.
Changes:
- Introduce an HVF
VirtualMachinebackend and macOS-specific build/runtime glue (bindgen + codesigning runner + interrupt handling). - Bump snapshot ABI/goldens versions and update snapshot file metadata/config handling for the new hypervisor tag and ABI.
- Adjust host/guest memory layout assumptions (page size usage, base addresses, alignment) and tune tests to avoid unsupported parallelism on HVF.
Reviewed changes
Copilot reviewed 39 out of 41 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| src/hyperlight_host/tests/snapshot_goldens/platform.rs | Add HVF hypervisor tag handling in snapshot golden platform selection. |
| src/hyperlight_host/tests/snapshot_goldens/goldens_version.rs | Bump goldens version and set arch-specific compatibility list. |
| src/hyperlight_host/tests/sandbox_host_tests.rs | Serialize/scale down thread-heavy tests under HVF; adjust scratch sizing. |
| src/hyperlight_host/tests/integration_test.rs | Reduce stress-test thread counts under HVF. |
| src/hyperlight_host/src/sandbox/snapshot/tripwires.rs | Bump expected snapshot ABI and update page-size/base-address invariants. |
| src/hyperlight_host/src/sandbox/snapshot/mod.rs | Align snapshot memory to host page size; update tests to use runtime page size. |
| src/hyperlight_host/src/sandbox/snapshot/file/media_types.rs | Bump SNAPSHOT_ABI_VERSION for snapshot memory blob. |
| src/hyperlight_host/src/sandbox/snapshot/file/config.rs | Add HVF hypervisor tag and macOS/aarch64 CPU vendor behavior. |
| src/hyperlight_host/src/sandbox/snapshot/file_tests.rs | Accept hvf annotation/tag in snapshot file tests. |
| src/hyperlight_host/src/sandbox/initialized_multi_use.rs | Use runtime page size in test helpers and overlap calculations. |
| src/hyperlight_host/src/sandbox/file_mapping.rs | Broaden mmap-backed file mapping resource handling to unix. |
| src/hyperlight_host/src/sandbox/config.rs | Enable interrupt retry delay config for HVF as well as Linux hypervisors. |
| src/hyperlight_host/src/mem/shared_mem.rs | Generalize mmap code to unix and switch guard sizing to runtime page size. |
| src/hyperlight_host/src/mem/shared_mem_tests.rs | Update tests to use runtime page size. |
| src/hyperlight_host/src/mem/mgr.rs | Account for host page alignment when locating snapshot PT tail. |
| src/hyperlight_host/src/mem/memory_region.rs | Use vmem page size constants and runtime page size where needed. |
| src/hyperlight_host/src/mem/layout.rs | Move base address to 0x4000; align sizes for mixed guest/host page sizes. |
| src/hyperlight_host/src/hypervisor/virtual_machine/whp.rs | Make register setters require &mut self to match new VM trait. |
| src/hyperlight_host/src/hypervisor/virtual_machine/mshv/x86_64.rs | Make register setters require &mut self to match new VM trait. |
| src/hyperlight_host/src/hypervisor/virtual_machine/mod.rs | Add HVF module/type and selection; extend error enums for HVF sync/errors. |
| src/hyperlight_host/src/hypervisor/virtual_machine/kvm/x86_64.rs | Make register setters require &mut self to match new VM trait. |
| src/hyperlight_host/src/hypervisor/virtual_machine/kvm/aarch64.rs | Make register setters require &mut self to match new VM trait. |
| src/hyperlight_host/src/hypervisor/virtual_machine/hvf/mod.rs | New HVF backend implementing VM lifecycle, memory mapping, and state sync. |
| src/hyperlight_host/src/hypervisor/virtual_machine/hvf/fp_abi.c | C ABI shims for SIMD FP register access not supported in stable Rust FFI. |
| src/hyperlight_host/src/hypervisor/virtual_machine/hvf/bindings.h | Header for bindgen + shim declarations, includes Hypervisor.framework. |
| src/hyperlight_host/src/hypervisor/surrogate_process.rs | Use runtime page size for guard pages in Windows surrogate mapping. |
| src/hyperlight_host/src/hypervisor/mod.rs | Refactor interrupt handle state tracking; add HVF cancel mechanism support. |
| src/hyperlight_host/src/hypervisor/hyperlight_vm/x86_64.rs | Switch to new interrupt handle constructors; adjust VM mutability. |
| src/hyperlight_host/src/hypervisor/hyperlight_vm/mod.rs | Route interrupt state through the new state machine in the VM run loop. |
| src/hyperlight_host/src/hypervisor/hyperlight_vm/aarch64.rs | Add HVF VM construction path and HVF interrupt handle wiring. |
| src/hyperlight_host/Cargo.toml | Add hvf feature, bindgen/cc build deps, parking_lot, and default-enable HVF. |
| src/hyperlight_host/build.rs | Generate HVF bindings and compile the C shim; add HVF cfg alias. |
| src/hyperlight_common/src/mem.rs | Remove fixed page-size constants from mem (shifted to vmem). |
| src/hyperlight_common/src/arch/aarch64/layout.rs | Adjust fixed AArch64 addresses to accommodate 16K pages. |
| flake.nix | Add aarch64-darwin target support and adjust tooling availability per platform. |
| flake.lock | Update pinned nixpkgs revision/hash. |
| dev/macos-sign-and-run.sh | Add macOS runner that codesigns binaries with hypervisor entitlement before exec. |
| dev/macos-entitlements.plist | Add com.apple.security.hypervisor entitlement for signed binaries. |
| CHANGELOG.md | Document the snapshot incompatibility caused by the address/layout change. |
| Cargo.lock | Record new dependencies (parking_lot, bindgen, cc) in the lockfile. |
| .cargo/config.toml | Configure macOS Cargo runner to use the codesign wrapper script. |
|
I submitted #1681, which has full surrogate support for parallel VMs. Would you be able to take a look at that before merging this? (Also, your copyright is out of date in several files) |
ludfjig
reviewed
Jul 23, 2026
ludfjig
left a comment
ludfjig
left a comment
Contributor
There was a problem hiding this comment.
Looks really good. I haven't reviewed the changes to InterruptHandle yet. Btw do you know if just snapshot-goldens-pull will work on mac once golden snapshto images areadded? I haven't tested it but I think there mgiht some hardcoded stuff there
| { | ||
| use std::os::unix::process::ExitStatusExt; | ||
| status.signal() == Some(libc::SIGSEGV) | ||
| let expected_signal = if cfg!(target_os = "macos") { |
Contributor
There was a problem hiding this comment.
maybe stale comment on top should be updated
Comment on lines
+735
to
+737
| fn simple_pt_base() -> usize { | ||
| page_size::get() + SandboxMemoryLayout::BASE_ADDRESS | ||
| } |
Contributor
There was a problem hiding this comment.
What is the initial chunk of page size for? Isn't this already included in base_address?
Member
Author
There was a problem hiding this comment.
This is matching what the test was doing before, I think; IIRC the tests here lay out the guest memory with 1 page of real data/code and then n pages of page tables.
Comment on lines
+553
to
+557
| snapshot_memory.resize( | ||
| snapshot_memory.len().next_multiple_of(page_size::get()), | ||
| 0u8, | ||
| ); | ||
|
|
Member
Author
There was a problem hiding this comment.
Needed to make sure that the visible-to-guest-when-pt-tail-is-unmapped portion is page aligned.
jsturtevant
reviewed
Jul 31, 2026
jsturtevant
left a comment
jsturtevant
left a comment
Contributor
There was a problem hiding this comment.
Amazing work! We should be able to run this on CI now?
jsturtevant
reviewed
Jul 31, 2026
jsturtevant
reviewed
Jul 31, 2026
jsturtevant
reviewed
Jul 31, 2026
jsturtevant
reviewed
Jul 31, 2026
syntactically
force-pushed
the
lm/hvf
branch
3 times, most recently
from
August 1, 2026 17:31
2484dc7 to
1f35d52
Compare
syntactically
force-pushed
the
lm/hvf
branch
2 times, most recently
from
August 1, 2026 17:49
d77b523 to
9f409ec
Compare
Remember to call `os_release` on `hv_vcpu_config_t`. Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
It will be used in hyperlight-host in the Hypervisor.framework implementation. Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
Use infrastructure for parsing ESR_ELx from hyperlight-common Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
Remove stale TODO comment. Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
Fix merge conflict Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
Fix merge conflict Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
Fix merge conflict Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
Fix merge conflict Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
On MacOS, libclang.dylib does not usually know how to find the correct SDK. (There's an exception for Nix-installed libclang.dylibs, which is why this was working before). Use xcrun to set the sdk root so that we can find the hvf bindings. Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
Handle srt==31 correctly. It turns out that in release builds, the compiler actually does end up doing enough inlining that it matters. Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
Update snapshot loading sanity checks to deal with the fact that the total memory size should be rounded up to the host page size. Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
[tests] hvf: support identifiers and adjust for hvf This adds new variants of the existing create_200_sandboxes test that stress-test the behaviour of the hvf code in situations with many more threads than vcpus. It also adjusts one test which relies on the preservation of system register state whose preservation Hyperlight does not guarantee; while that state is preserved on other backends (up to a snapshot restore) it can be lost on hvf if the vcpu is destroyed and recreated. Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
Transparently manage the limit on the number of vcpus per process (as reported by hv_vm_get_max_vcpu_count()). Previously, we left it up to the user to make sure that they did not have too many live threads that had used a sandbox. However, despite some attempts to reduce the thread-heaviness of the testing framework, it turns out that in some circumstances, we would in fact create too many parallel threads and have issues. This commit changes the hvf code to do its best to make sure that 1 vcpu is always available for new workloads, by having the thread that gets the "last" vcpu destroy it as soon as it is done running. Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
It will be used from the hvf code, which needs to return a structured error itself. Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
APFS does not accept non-UTF-8 paths at all, so creating one from which to test loading is impossible. Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
Member
Author
|
@ludfjig @jsturtevant There are a few substantive changes in the last few commits (starting from 83b696a Convert mem::shared_mem module to use thiserror structured errors), so please take a quick skim if you have a chance. |
Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
ludfjig
approved these changes
Aug 11, 2026
ludfjig
left a comment
ludfjig
left a comment
Contributor
There was a problem hiding this comment.
I like the typed error changes. Was there something particular that made you change it?
Comment on lines
+1078
to
+1085
| fn map_memory(&mut self, region: (u32, &MemoryRegion)) -> Result<(), HypervisorError> { | ||
| let slot = region.0 as usize; | ||
| if self.mappings.len() <= slot { | ||
| self.mappings.resize(slot + 1, None); | ||
| } | ||
| self.mappings[slot] = Some(region.1.clone()); | ||
| self.opportunistically_sync_slot(slot) | ||
| } |
Contributor
There was a problem hiding this comment.
Should we do self.opportunistically_sync_slot(slot) before updating self.mappings in case it fails? Same for fn unmap_memory. Or do some roll back if it fails
| //! The Hyperlight public API constraints sandboxes (and, by | ||
| //! extension, hypervisor API implementations) to implement | ||
| //! [`core::marker::Send`]. Other hypervisors have one vCPU but allow | ||
| //! the vCPU handle to be mgirated across threads, although this comes |
| /// For the memory region with a TLBI in it used in tlbi_vmalle1 | ||
| /// below, we really want the semantics of a global | ||
| /// [`OnceLock`]. Unfortunately, [`OnceLock::get_or_try_init`] is | ||
| /// not stable, so we can'} use it, so we would have to use an |
| } | ||
| } | ||
|
|
||
| /// This function is used both as a performance optimisation e(when |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.