Add initial single-sandbox-executing-at-a-time support for Hypervisor.framework by syntactically · Pull Request #1674 · hyperlight-dev/hyperlight

syntactically · 2026-07-21T15:27:10Z

No description provided.

Copilot

Pull request overview

Adds initial Hypervisor.framework (HVF) support to hyperlight-host on macOS, along with a snapshot ABI/version bump and memory-layout/page-size changes needed for platforms with 16K host pages.

Changes:

Introduce an HVF VirtualMachine backend and macOS-specific build/runtime glue (bindgen + codesigning runner + interrupt handling).
Bump snapshot ABI/goldens versions and update snapshot file metadata/config handling for the new hypervisor tag and ABI.
Adjust host/guest memory layout assumptions (page size usage, base addresses, alignment) and tune tests to avoid unsupported parallelism on HVF.

Reviewed changes

Copilot reviewed 39 out of 41 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
src/hyperlight_host/tests/snapshot_goldens/platform.rs	Add HVF hypervisor tag handling in snapshot golden platform selection.
src/hyperlight_host/tests/snapshot_goldens/goldens_version.rs	Bump goldens version and set arch-specific compatibility list.
src/hyperlight_host/tests/sandbox_host_tests.rs	Serialize/scale down thread-heavy tests under HVF; adjust scratch sizing.
src/hyperlight_host/tests/integration_test.rs	Reduce stress-test thread counts under HVF.
src/hyperlight_host/src/sandbox/snapshot/tripwires.rs	Bump expected snapshot ABI and update page-size/base-address invariants.
src/hyperlight_host/src/sandbox/snapshot/mod.rs	Align snapshot memory to host page size; update tests to use runtime page size.
src/hyperlight_host/src/sandbox/snapshot/file/media_types.rs	Bump `SNAPSHOT_ABI_VERSION` for snapshot memory blob.
src/hyperlight_host/src/sandbox/snapshot/file/config.rs	Add HVF hypervisor tag and macOS/aarch64 CPU vendor behavior.
src/hyperlight_host/src/sandbox/snapshot/file_tests.rs	Accept `hvf` annotation/tag in snapshot file tests.
src/hyperlight_host/src/sandbox/initialized_multi_use.rs	Use runtime page size in test helpers and overlap calculations.
src/hyperlight_host/src/sandbox/file_mapping.rs	Broaden mmap-backed file mapping resource handling to `unix`.
src/hyperlight_host/src/sandbox/config.rs	Enable interrupt retry delay config for HVF as well as Linux hypervisors.
src/hyperlight_host/src/mem/shared_mem.rs	Generalize mmap code to `unix` and switch guard sizing to runtime page size.
src/hyperlight_host/src/mem/shared_mem_tests.rs	Update tests to use runtime page size.
src/hyperlight_host/src/mem/mgr.rs	Account for host page alignment when locating snapshot PT tail.
src/hyperlight_host/src/mem/memory_region.rs	Use vmem page size constants and runtime page size where needed.
src/hyperlight_host/src/mem/layout.rs	Move base address to 0x4000; align sizes for mixed guest/host page sizes.
src/hyperlight_host/src/hypervisor/virtual_machine/whp.rs	Make register setters require `&mut self` to match new VM trait.
src/hyperlight_host/src/hypervisor/virtual_machine/mshv/x86_64.rs	Make register setters require `&mut self` to match new VM trait.
src/hyperlight_host/src/hypervisor/virtual_machine/mod.rs	Add HVF module/type and selection; extend error enums for HVF sync/errors.
src/hyperlight_host/src/hypervisor/virtual_machine/kvm/x86_64.rs	Make register setters require `&mut self` to match new VM trait.
src/hyperlight_host/src/hypervisor/virtual_machine/kvm/aarch64.rs	Make register setters require `&mut self` to match new VM trait.
src/hyperlight_host/src/hypervisor/virtual_machine/hvf/mod.rs	New HVF backend implementing VM lifecycle, memory mapping, and state sync.
src/hyperlight_host/src/hypervisor/virtual_machine/hvf/fp_abi.c	C ABI shims for SIMD FP register access not supported in stable Rust FFI.
src/hyperlight_host/src/hypervisor/virtual_machine/hvf/bindings.h	Header for bindgen + shim declarations, includes Hypervisor.framework.
src/hyperlight_host/src/hypervisor/surrogate_process.rs	Use runtime page size for guard pages in Windows surrogate mapping.
src/hyperlight_host/src/hypervisor/mod.rs	Refactor interrupt handle state tracking; add HVF cancel mechanism support.
src/hyperlight_host/src/hypervisor/hyperlight_vm/x86_64.rs	Switch to new interrupt handle constructors; adjust VM mutability.
src/hyperlight_host/src/hypervisor/hyperlight_vm/mod.rs	Route interrupt state through the new state machine in the VM run loop.
src/hyperlight_host/src/hypervisor/hyperlight_vm/aarch64.rs	Add HVF VM construction path and HVF interrupt handle wiring.
src/hyperlight_host/Cargo.toml	Add `hvf` feature, bindgen/cc build deps, parking_lot, and default-enable HVF.
src/hyperlight_host/build.rs	Generate HVF bindings and compile the C shim; add HVF cfg alias.
src/hyperlight_common/src/mem.rs	Remove fixed page-size constants from `mem` (shifted to `vmem`).
src/hyperlight_common/src/arch/aarch64/layout.rs	Adjust fixed AArch64 addresses to accommodate 16K pages.
flake.nix	Add aarch64-darwin target support and adjust tooling availability per platform.
flake.lock	Update pinned nixpkgs revision/hash.
dev/macos-sign-and-run.sh	Add macOS runner that codesigns binaries with hypervisor entitlement before exec.
dev/macos-entitlements.plist	Add `com.apple.security.hypervisor` entitlement for signed binaries.
CHANGELOG.md	Document the snapshot incompatibility caused by the address/layout change.
Cargo.lock	Record new dependencies (parking_lot, bindgen, cc) in the lockfile.
.cargo/config.toml	Configure macOS Cargo runner to use the codesign wrapper script.

siennathesane · 2026-07-22T19:23:55Z

I submitted #1681, which has full surrogate support for parallel VMs. Would you be able to take a look at that before merging this?

(Also, your copyright is out of date in several files)

ludfjig

Looks really good. I haven't reviewed the changes to InterruptHandle yet. Btw do you know if just snapshot-goldens-pull will work on mac once golden snapshto images areadded? I haven't tested it but I think there mgiht some hardcoded stuff there

ludfjig · 2026-07-23T17:30:09Z

                let read_len = std::cmp::min(
                    data.len(),
-                    (PAGE_SIZE - (gpa & (PAGE_SIZE - 1))).try_into().unwrap(),
+                    page_size::get() - (gpa as usize & (page_size::get() - 1)),


I assume page_size is cached?

Yes, it's a OnceLock internally

ludfjig · 2026-07-23T17:32:23Z

    /// Offset of the PEB struct within the snapshot region.
    pub(crate) fn peb_offset(&self) -> usize {
-        self.code_size.next_multiple_of(PAGE_SIZE_USIZE)
+        self.code_size.next_multiple_of(PAGE_SIZE)


My first instinct was why not use page_size::get() here then I realize it's the guest's page size. Do you think we should rename variable to indicate this?

The import path is in hyperlight_common::vmem which is where all the other stuff that is related to the guest memory translation stuff lives, which I think is disambiguating enough? We have a tonne of other constants in there that we would probably want to rename too if we changed this one, and I think it would get a bit clunky.

ludfjig · 2026-07-23T17:33:59Z

            {
                use std::os::unix::process::ExitStatusExt;
-                status.signal() == Some(libc::SIGSEGV)
+                let expected_signal = if cfg!(target_os = "macos") {


maybe stale comment on top should be updated

ludfjig · 2026-07-23T17:36:36Z

+            .context("Couldn't write hvf bindings")?;
+
+        cc::Build::new()
+            .opt_level(3)


should we unconditionally do opt level 3?

It's a tiny file with ABI wrapper functions, so I think unconditionally optimising is fine.

ludfjig · 2026-07-23T17:38:28Z

+    fn simple_pt_base() -> usize {
+        page_size::get() + SandboxMemoryLayout::BASE_ADDRESS
+    }


What is the initial chunk of page size for? Isn't this already included in base_address?

This is matching what the test was doing before, I think; IIRC the tests here lay out the guest memory with 1 page of real data/code and then n pages of page tables.

ludfjig · 2026-07-23T17:38:47Z

+                snapshot_memory.resize(
+                    snapshot_memory.len().next_multiple_of(page_size::get()),
+                    0u8,
+                );
+


Why this change?

jsturtevant

Amazing work! We should be able to run this on CI now?

jsturtevant · 2026-07-31T00:30:13Z

+        let mut vcpu: MaybeUninit<bindings::hv_vcpu_t> = MaybeUninit::zeroed();
+        let mut exit: *mut bindings::hv_vcpu_exit_t = core::ptr::null_mut();
+        unsafe {
+            let config = bindings::hv_vcpu_config_create();


do we need to release this when done since these are FFI?

https://developer.apple.com/documentation/hypervisor/hv_vcpu_config_create()#Discussion

Thanks, fixed in d12c2b1.

jsturtevant · 2026-07-31T00:54:58Z

+                                    VmExit::Halt()
+                                } else {
+                                    let srt = bits::<20, 16>(esr);
+                                    let data = self.get_vcpu_regs()?.x[srt as usize];


this looks like srt could be 11111(31) and this indexes into

hyperlight/src/hyperlight_host/src/hypervisor/regs/aarch64/common_regs.rs

Line 19 in f45ee4b

pub(crate) x: [u64; 31],

which could cause a panic?

Good catch! 31 would usually mean the zero register in this context (though for some instructions it can be the stack pointer). We don't do anything that could make that happen in hyperlight-guest, so I've changed it to fall back to reporting it as a non-hyperlight fault in fe60ed4.

Actually, with some more changes, we now do run into this in release builds which are optimised to use xzr, so fixed in fa1367f.

jsturtevant · 2026-07-31T15:56:08Z

+                    }
+                } else {
+                    unknown_exit
+                }


would be it be possible to flip these if statements to exit early instead of nesting and make it easier to read?

Even though there is only one syndrome that we bother to decode more fully right now, I don't think I like the idea of giving structural primacy to that code path in that way, because it's not really privileged as the "right path"---there's no reason why we mightn't switch to using a mix of different exits if we thought it was more performant or easier for some reason. If we were to add another case for parsing ec, right now that would look like just going from

if ec == ... { ... } else { unknown_exit }

to

if ec == ... { ... } else if ec == ... { ... } else { unknown_exit }

which is the correct idea (we're just adding a 2nd case to the 1 we already supported), whereas going to that from

if ec != ... { return unknown_exit; } ...

feels like you are totally changing the character of the logic.

However, I did realise that we already have some code parsing essentially the same thing in hyperlight-guest-bin in the exception handler, which parses into some nice enum structures which are easy and clear to destructure on. So, I've moved that into hyperlight-common (b38d5cd), and simplified this decode logic a little bit using it (fe60ed4).

jsturtevant · 2026-07-31T15:58:24Z

+            self.interrupt_handle.set_vcpu(vcpu.id);
+            vcpu.sync_state_from_vm(self)
+                .map_err(RunVcpuError::HvfSync)?;
+            // TODO: replace unwrap()


looks like this was taken care of and comment needs to be removed?

Thanks, fixed in 0d018f4.

jsturtevant · 2026-07-31T16:01:17Z

+
+    #[cfg(target_arch = "aarch64")]
+    fn reset_vcpu(&mut self) -> std::result::Result<(), ResetVcpuError> {
+        self.id.epoch += 1;


out of curiosity how does this reset the vcpu?

It means that when we sync_state_from the sandbox to the vcpu that will actually be used for executing the sandbox, even if that vcpu previously executed the same sandbox, it will see the CPU-state-epoch mismatch and call reset_vcpu().

jsturtevant · 2026-07-31T16:03:58Z

-    /// Bit layout:
-    /// - Bit 2: DEBUG_INTERRUPT_BIT - set when debugger interrupt is requested
-    /// - Bit 1: RUNNING_BIT - set when vcpu is actively running
-    /// - Bit 0: CANCEL_BIT - set when cancellation has been requested


did we loose this doc?

It seemed to me that this was basically a duplicate of the constant names above, probably because the same pattern was being used in so many places. Since this refactor pushes the management of these actual bits all into one smaller place where the constants are right there, I thought it was likely no longer necessary. I can add it back if you prefer, though.

Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

Previously, the Linux and WHP interrupt handle implementations used exactly the same logic to update three bits of atomic state, and further interrupt handle implementations would likely need the same. This commit extracts the state machine logic into a single shared implementation used by both interrupt handles. It also refactors `WindowsInterruptHandle` to pull the logic around locking into another separate type, which will be shared with other interrupt handle implementations in the future Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

This commit moves a bunch of code (primarily mmap-based implementations of things) from cfg(target_os = "linux") to cfg(unix), allowing it to compile on MacOS (where it will be useful). It also moves the kvm and mshv package dependencies from cfg(unix) to cfg(target_os = "linux"), so that Cargo does not try (and fail) to build them on MacOS. Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

todo: squash: hvf interrupt handle implementation Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

On AArch64, there is no particular reason to expect the page size of the host to match the page size of the guest, but the host previously used some constants (`hyperlight_common::mem::{PAGE_SIZE, PAGE_SIZE_USIZE, PAGE_SHIFT}`) both when computing guest memory layouts and when constructing host->guest mappings. This commit removes all the uses of those ambiguous constants, replacing them with `page_size::get()` when the host page size is needed and `hyperlight_common::vmem::PAGE_SIZE` when the guest page size is required. It also makes a few tweaks to remove assumptions that the page sizes are the same, allowing (at the very least) operation on a host with 16k pages while allowing guests to continue using 4k pages. Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

These conceptually mutate the state of the guest, so it makes sense that they should take &mut. This is useful for HVF, which stores a copy of the regs when they are set, and in this manner can avoid an extra Rust lock. Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

Other operating systems do not support its use in EL0 Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

On MacOS, communicating with the hypervisor requires that a binary be signed with an "entitlement" `com.apple.security.hypervisor`. This commit adds a script that locally signs and then runs a binary, and configures Cargo to use that script to run tests/exmaples/etc. Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

Remember to call `os_release` on `hv_vcpu_config_t`. Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

It will be used in hyperlight-host in the Hypervisor.framework implementation. Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

Use infrastructure for parsing ESR_ELx from hyperlight-common Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

Remove stale TODO comment. Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

Fix merge conflict Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

On MacOS, libclang.dylib does not usually know how to find the correct SDK. (There's an exception for Nix-installed libclang.dylibs, which is why this was working before). Use xcrun to set the sdk root so that we can find the hvf bindings. Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

Handle srt==31 correctly. It turns out that in release builds, the compiler actually does end up doing enough inlining that it matters. Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

Update snapshot loading sanity checks to deal with the fact that the total memory size should be rounded up to the host page size. Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

Copilot AI review requested due to automatic review settings July 21, 2026 15:27

syntactically requested review from andreiltd, danbugs, dblnz, devigned, jprendes, jsturtevant, ludfjig, simongdavies and squillace as code owners July 21, 2026 15:27

syntactically added kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. ready-for-review PR is ready for (re-)review labels Jul 21, 2026

Copilot started reviewing on behalf of syntactically July 21, 2026 15:27 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

syntactically added the regen-goldens Regenerate snapshot golden fixtures label Jul 21, 2026

syntactically force-pushed the lm/hvf branch from 2cab1eb to 64be23b Compare July 21, 2026 15:54

ludfjig reviewed Jul 23, 2026

View reviewed changes

syntactically mentioned this pull request Jul 23, 2026

feat: macOS support via Hypervisor.framework (aarch64) #1681

Open

jsturtevant reviewed Jul 31, 2026

View reviewed changes

syntactically added 6 commits August 1, 2026 18:01

nix: Support developing on macOS

4f79454

Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

Abstract out interrupt handle retry logic

7c99038

Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

hvf: add cfg items and detection stub

9fd78b0

Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

hvf: add Rust bindings

dc84b0c

Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

syntactically added 8 commits August 1, 2026 18:01

hvf: add interrupt handle using hv_vcpus_exit

4cced20

todo: squash: hvf interrupt handle implementation Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

Change guest layout to support 16k host pages

5b5ae86

Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

[tests] hvf: support identifiers and reduce thread counts

f5a27f4

Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

Restrict usage of MIDR_EL1 to Linux

a81433e

Other operating systems do not support its use in EL0 Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

Initial single-address-space support for Hypervisor.framework

7a6e2fd

Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

syntactically force-pushed the lm/hvf branch 3 times, most recently from 2484dc7 to 1f35d52 Compare August 1, 2026 17:31

syntactically changed the base branch from main to lm/enable-a64kvm-ci August 1, 2026 17:34

syntactically force-pushed the lm/hvf branch from 1f35d52 to d77b523 Compare August 1, 2026 17:38

syntactically added 9 commits August 1, 2026 18:48

fixup! Initial single-address-space support for Hypervisor.framework

cac4dc7

Remember to call `os_release` on `hv_vcpu_config_t`. Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

aarch64: Move machinery for decoding ESR_ELx to hyperlight-common

35f5830

It will be used in hyperlight-host in the Hypervisor.framework implementation. Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

fixup! Initial single-address-space support for Hypervisor.framework

1324b1a

Use infrastructure for parsing ESR_ELx from hyperlight-common Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

fixup! Initial single-address-space support for Hypervisor.framework

9a412e3

Remove stale TODO comment. Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

fixup! Abstract out interrupt handle retry logic

0600270

Fix merge conflict Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

fixup! Reduce duplication of interrupt handle state machines

6ac5585

Fix merge conflict Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

fixup! Disentangle host and guest page sizes

91e4b98

Fix merge conflict Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

fixup! hvf: add Rust bindings

1ac40f8

Fix merge conflict Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

ci: enable running on MacOS/HVF

9f409ec

Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

syntactically force-pushed the lm/hvf branch from d77b523 to 9f409ec Compare August 1, 2026 17:49

syntactically force-pushed the lm/hvf branch from f842699 to 5663bfb Compare August 1, 2026 19:28

syntactically added 2 commits August 1, 2026 21:30

fixup! Initial single-address-space support for Hypervisor.framework

fa1367f

Handle srt==31 correctly. It turns out that in release builds, the compiler actually does end up doing enough inlining that it matters. Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

fixup! Disentangle host and guest page sizes

42993eb

Update snapshot loading sanity checks to deal with the fact that the total memory size should be rounded up to the host page size. Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

Uh oh!

Conversation

syntactically commented Jul 21, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

siennathesane commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ludfjig left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jsturtevant left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

siennathesane commented Jul 22, 2026 •

edited

Loading