DragonFly VMM API and virtualization stack design

vmm-api.md

VMM Core API And Frontends

DragonFly Virtualization Stack

DragonFly virtualization is one layered stack, not a replacement for every virtual-machine use case. vmm(4) is the common kernel backend. nvmm(4) and vmmfs(5) are deliberately different management frontends over that same backend: the former preserves an exit-driven userspace-monitor model; the latter provides a declarative, modern PCIe-only kernel-managed model.

The table describes the target architecture. Entries that are not yet shipped or migrated are design commitments, not claims about the current tree.

Component	Role and boundary
vmm(4)	Unified virtual-machine backend. It exposes kernel KPI only and owns VM creation, vCPU execution, guest memory, VM exits, interrupt routing and hardware acceleration. Its implementation is to be derived by extending the existing nvmm(4) hardware code, then separating the reusable runtime core.
nvmm(4)	Virtual-machine management frontend. It exposes /dev/nvmm, preserves the NetBSD-compatible base ioctl set, and adds DragonFly-specific extension ioctls. It calls vmm(4) rather than owning a separate virtualization backend.
libnvmm(3)	Userspace wrapper for nvmm(4). It preserves a fully NetBSD-compatible libnvmm(3) ABI and adds DragonFly extensions, all implemented through nvmm(4).
vmmfs(5)	Declarative filesystem management frontend over vmm(4). It presents VM lifecycle and configuration as files, is PCI-centred, and intentionally provides no legacy-device platform.
vmmld_linux(7)	Linux loader for vmmfs(5).
vmmld_dragonfly(7)	DragonFly loader for vmmfs(5), intended to boot the local DragonFly kernel directly.
microvm(7)	VM-isolated lightweight DragonFly container manager built on libnvmm(3). It supports only a limited virtio-mmio device set. Its goals are to replace vkernel(7) with a better development/test environment and to provide stronger isolation than jail(8) at comparable startup time and resource cost.
emulators/qemu	Full-system emulator accelerated through libnvmm(3). It remains the compatibility path for broad legacy and device emulation rather than competing with the restricted vmmfs(5) platform.
sysutils/vmmld_efi	UEFI loader for vmmfs(5), loading firmware supplied by sysutils/edk2.
sysutils/runv	VM-isolated OCI container runtime built on vmmfs(5). It supports a limited virtio-pci device set, implements the OCI and containerd interfaces, uses Kata Containers today, and is intended to support DragonFly containers later.
sysutils/virtiod	Virtio-pci backend service for block, network, vsock, RNG and filesystem devices; GPU, sound and input backends are later work.
sysutils/containerd	Container management system exposing Docker and CRI/CNI/CSI management interfaces. It defaults to runv, stores images as raw blocks, and is adapted for hammer2(8).
sysutils/docker	Docker-compatible container service, using containerd by default.
sysutils/nerdctl	Container management CLI and frontend.
sysutils/kubelet	Kubernetes node service using containerd by default. It is intended to provide Kata Container-compatible service on DragonFly and join Linux Kubernetes clusters.
sysutils/flannel	CNI-compatible virtual routed network.
sysutils/calico	CNI-compatible BGP virtual network.

The intended dependency direction is:

vmm(4) <- nvmm(4) <- libnvmm(3) <- microvm(7), qemu
vmm(4) <- vmmfs(5) <- vmmld_*, runv <- containerd <- docker, nerdctl, kubelet
runv <- virtiod
containerd <- flannel, calico

Status

This is a proposed next-generation VMM API. It does not change the current vmmfs ABI or claim that a new /dev/vmm ioctl ABI exists today.

The purpose is to keep one hardware virtualization core while supporting two execution frontends:

1. a kernel-owned vCPU LWKT loop for vmmfs and the modern vPCIe platform;
2. an exit-driven LWP loop for existing libnvmm, QEMU, and a future native userspace monitor.

The core owns NPT/EPT, guest physical memory, VM exits, I/O traps and interrupt delivery. A frontend owns only object presentation, caller context and the policy for an exit which the core did not claim.

Design Rules

• Normal guest RAM is mapped by NPT/EPT and never exits.
• A registered MMIO or PIO range traps and invokes a short, non-blocking kernel callback.
• A callback never calls userspace code and never waits for backend I/O.
• An unclaimed exit is returned to the current vmm_vcpu_run() caller.
• All device interrupt sources converge on vmm_irq_raise_gsi() or vmm_irq_raise_msi(); vmm_vcpu_inject_legacy() remains a direct, compatibility-only per-vCPU injection operation.
• Interrupt delivery is selected automatically by the active backend below those three interrupt ingress functions: AVIC/APICv/posted-interrupt hardware delivery is used when available; otherwise the kernel maintains software virtual-interrupt pending state and injects at a safe guest entry point. No frontend selects this policy per interrupt.
• A vmmfs provider is a capability consumer, not a userspace machine monitor. It never owns the vCPU run loop.

Core C API

The declarations below are the intended semantic API. Exact structure layouts, ownership annotations and header placement remain implementation work. vmm_* is kernel-only C KPI; it is not a userspace ABI.

Objects

struct vmm_machine;
struct vmm_vcpu;
struct vmm_memory;
struct vmm_io;
struct vmm_exit;

vmm_machine is a short-lived powered runtime instance. It owns its vCPU set, guest physical address space, active I/O registrations and interrupt routes. For vmmfs, a separate long-lived vmm_machine_decl owns configured CPU, memory, loader and PCIe slot declarations; cold start snapshots it into a new vmm_machine. An NVMM owner/machine ID or future native machine fd can own a runtime directly without adopting vmmfs declaration semantics.

Module Boundary

The current single sys/vmm module is transitional. The target is sys/dev/vmm for this runtime API and sys/fs/vmmfs for the declaration/VFS frontend. vmmfs.ko statically depends on vmm.ko; runtime objects must not depend on vnodes, namecache, or any vmmfs declaration representation.

Machine And VCPU Lifecycle

int vmm_machine_create(struct vmm_machine **);
void vmm_machine_destroy(struct vmm_machine *);

int vmm_machine_start(struct vmm_machine *, unsigned int timeout_in_ms,
    bool force);
int vmm_machine_stop(struct vmm_machine *, unsigned int timeout_in_ms,
    bool force);
int vmm_machine_reset(struct vmm_machine *, unsigned int timeout_in_ms,
    bool force);

int vmm_vcpu_create(struct vmm_machine *, unsigned int vcpu_id,
    struct vmm_vcpu **);
void vmm_vcpu_destroy(struct vmm_vcpu *);
int vmm_vcpu_set_state(struct vmm_vcpu *, const struct vmm_vcpu_state *);
int vmm_vcpu_get_state(struct vmm_vcpu *, struct vmm_vcpu_state *);
int vmm_vcpu_run(struct vmm_vcpu *, struct vmm_exit *);

vmm_vcpu_run() is synchronous and blocking. It enters the guest and returns only after an exit which cannot be consumed by the common kernel dispatcher. The vmmfs vCPU LWKT calls it in a loop. An NVMM LWP calls it from NVMM_IOC_VCPU_RUN; that ioctl returns to userspace only when this call returns an unclaimed exit.

The synchronous machine operations wait for a stable RUNNING or STOPPED state. They are MPSAFE and idempotent. STARTING and DRAINING remain internal states.

Guest Memory

int vmm_memory_alloc(struct vmm_machine *, size_t size,
    struct vmm_memory **);
int vmm_memory_map_gpa(struct vmm_memory *, uint64_t gpa, size_t size,
    int prot);
void vmm_memory_unmap_gpa(struct vmm_memory *, uint64_t gpa, size_t size);
void vmm_memory_free(struct vmm_memory *);

vmm_memory_alloc() creates a machine-owned guest RAM vm_object; it does not automatically create a permanent userspace mapping. map_gpa() places a range of that object in the machine address space. The backend turns the machine pmap into NPT/EPT mappings and resolves missing pages through the host VM system.

Userspace access is always a frontend-created capability mapping of this same object:

• loader fd 3: one start epoch, revoked on loader exit or VM stop;
• vPCIe DMA capability: one powered provider generation, revoked on provider detach, cold device power-off, VM stop or machine destroy; provider detach does not unmap the still-running guest BAR or guest RAM mapping;
• NVMM compatibility HVA mapping: owned by the NVMM owner process according to the legacy ABI.

No path copies guest RAM between loader, provider and vCPU execution.

I/O Trap Registration

typedef int vmm_mmio_callback_t(struct vmm_io *,
    struct vmm_mmio_access *, void *);
typedef int vmm_pio_callback_t(struct vmm_io *,
    struct vmm_pio_access *, void *);

int vmm_io_register_mmio(struct vmm_machine *, uint64_t gpa, size_t size,
    vmm_mmio_callback_t *, void *arg, struct vmm_io **);
int vmm_io_register_pio(struct vmm_machine *, uint16_t port, size_t size,
    vmm_pio_callback_t *, void *arg, struct vmm_io **);
void vmm_io_unregister(struct vmm_io *);

An MMIO registration reserves a GPA range from ordinary RAM mapping. A guest access causes a nested-page trap and dispatches to the callback. A PIO registration reserves an I/O port range; port is not a GPA.

The access structures carry direction, width, address/port, data and the calling vCPU. A callback handles a read by supplying data, handles a write by recording small kernel state or notifying an endpoint, then returns. It must not sleep waiting for userspace. A registration is owned by the platform or device object which created it and is removed before that object is destroyed.

Not every device access uses a registration. Direct shared BAR mappings are not trapped. Conversely, a synchronous control page or queue doorbell can be registered and handled by the vPCIe callback. A GPA/port without RAM or a registered kernel callback remains an unclaimed exit for an exit-driven frontend.

Interrupt Delivery

void vmm_irq_raise_gsi(struct vmm_machine *, unsigned int gsi, bool level);
void vmm_irq_raise_msi(struct vmm_machine *, uint64_t address,
    uint32_t data);
void vmm_vcpu_inject_legacy(struct vmm_vcpu *, uint8_t vector);

vmm_irq_raise_gsi() is the compatibility ingress for an asserted/deasserted legacy interrupt line. The machine interrupt router translates GSI to its IOAPIC/PIC-compatible destination where such a platform exists.

vmm_irq_raise_msi() is the modern PCIe ingress. The caller supplies the current MSI or MSI-X message address and data; the router derives the target virtual APIC and vector from that message and current machine topology.

vmm_vcpu_inject_legacy() bypasses GSI/MSI routing. It exists for the NVMM event ABI, debugging and architecturally direct events; ordinary PCIe devices must use MSI/MSI-X instead.

All three calls reach one backend-independent pending/delivery path. The active backend chooses the fastest supported mechanism automatically:

AVIC / APICv / posted interrupt available
  -> hardware virtual-APIC delivery

otherwise
  -> kernel software pending state
  -> kick a running target when necessary
  -> inject at a safe guest entry point

The source API does not change. A production vmmfs platform profile may make AVIC/APICv admission mandatory; a compatibility frontend may permit the software path. This is a machine/backend admission decision, never a per-provider or per-interrupt userspace choice.

Common Exit Dispatch

vmm_vcpu_run()
    -> raw VMEXIT translated by SVM/VMX backend
    -> common memory, I/O and interrupt dispatcher
       -> guest RAM fault: resolve host VM fault and re-enter
       -> registered MMIO/PIO: invoke callback and re-enter
       -> core-owned platform event: handle and re-enter
       -> unclaimed exit: return struct vmm_exit to caller

The core does not infer a userspace device model. It only invokes callbacks registered by a kernel-owned platform object or returns an unclaimed access. An unassigned PCIe range follows the machine's PCIe/bus policy; it is not blindly turned into #GP.

Existing NVMM Frontend

The current NVMM ABI remains an adapter. It is not required to expose the vmmfs PCIe fabric.

Existing NVMM ioctl	Core composition
NVMM_IOC_CAPABILITY	Query the selected backend and common VMM capabilities.
NVMM_IOC_MACHINE_CREATE	vmm_machine_create(), then install the legacy machine ID under the NVMM owner.
NVMM_IOC_MACHINE_DESTROY	Stop/destroy the core machine after legacy owner teardown rules.
NVMM_IOC_MACHINE_CONFIGURE	Translate supported legacy machine options into core machine configuration. Unsupported legacy-only options keep their existing error semantics.
NVMM_IOC_VCPU_CREATE	vmm_vcpu_create() plus the legacy shared nvmm_comm_page mapping.
NVMM_IOC_VCPU_DESTROY	vmm_vcpu_destroy().
NVMM_IOC_VCPU_CONFIGURE	Translate supported per-vCPU configuration to the core/backend.
NVMM_IOC_VCPU_SETSTATE	vmm_vcpu_set_state(), using the existing comm-page commit rules.
NVMM_IOC_VCPU_GETSTATE	vmm_vcpu_get_state(), using the existing comm-page cache rules.
NVMM_IOC_VCPU_INJECT	vmm_vcpu_inject_legacy().
NVMM_IOC_VCPU_RUN	Call vmm_vcpu_run(). Kernel-claimed exits re-enter; an unclaimed vmm_exit is translated to nvmm_vcpu_exit and returned to the calling LWP.
NVMM_IOC_HVA_MAP	vmm_memory_alloc() plus a legacy owner HVA mapping of the resulting object.
NVMM_IOC_GPA_MAP	Find the legacy HVA object's vmm_memory, then call vmm_memory_map_gpa().
NVMM_IOC_HVA_UNMAP	Remove the legacy owner HVA mapping and release its mapping reference.
NVMM_IOC_GPA_UNMAP	vmm_memory_unmap_gpa().
NVMM_IOC_CTL	Core machine/backend query and control operations where the existing operation has a VMM equivalent.

The legacy HVA/GPA split remains important: HVA_MAP creates the object and maps it into the owner address space; GPA_MAP maps that same object into the machine. It is a zero-copy compatibility path, not a second guest-memory implementation.

Proposed NVMM Extensions

These extensions are optional. They are only needed when an exit-driven monitor wants KVM-style I/O fast paths; ordinary QEMU compatibility works without them because unclaimed accesses already return through NVMM_IOC_VCPU_RUN.

Proposed ioctl	Endpoint ABI and purpose	Core composition
NVMM_IOC_IOEVENT_BIND	Create and return an anonymous duplex VMM endpoint fd. Bind a specified PIO or MMIO write match to its read direction. A matching guest write makes the fd readable and generates EVFILT_READ; read(2) receives a fixed doorbell record. It does not return from VCPU_RUN.	Create and own a vmm_io_register_mmio() or vmm_io_register_pio() registration whose kernel callback publishes a doorbell record to the endpoint.
NVMM_IOC_IOEVENT_UNBIND	Remove a prior fast write binding.	vmm_io_unregister().
NVMM_IOC_SET_GSI_ROUTING	Define the compatibility machine's legacy GSI routes.	Install/update the machine interrupt-router GSI table used by vmm_irq_raise_gsi().
NVMM_IOC_IRQ_BIND	Take the endpoint fd returned by IOEVENT_BIND and bind its write direction to a GSI or MSI source. write(2) submits a fixed interrupt record; it is the DragonFly capability equivalent of KVM irqfd. It returns status, not a second fd.	Endpoint input validates the record then calls vmm_irq_raise_gsi() or vmm_irq_raise_msi().
NVMM_IOC_IRQ_UNBIND	Remove a prior endpoint interrupt binding.	Detach the endpoint from the selected interrupt ingress.

The endpoint is an anonymous VMM capability fd with kqueue readiness and fixed record semantics:

guest MMIO/PIO write -> endpoint readable -> read doorbell record
backend completion   -> write interrupt record -> vmm_irq_raise_*

One endpoint therefore carries the equivalent of both KVM ioeventfd and irqfd; the two ioctls bind its two directions and do not create two unrelated event objects. It replaces the KVM pattern of separately creating global eventfds and registering them through ioctl. It may be implemented with the same private socket/capability machinery used by vPCIe, but NVMM does not gain vmmfs naming or PCIe ownership semantics merely by using it.

vmmfs Frontend

vmmfs remains declarative. Its VOPs validate filesystem state and invoke synchronous core operations or enqueue the existing machine command; they do not run a VCPU or wait for provider I/O themselves.

vmmfs action	Frontend action	Core composition
mkdir <machine>	Create a VFS node with a stopped machine declaration.	Create vmm_machine_decl; no guest RAM, vCPU or runtime instance exists yet.
Close vcpu, mem, loader config write	Validate and store the next cold-start configuration.	Update vmm_machine_decl; an already powered vmm_machine is unchanged.
rm <machine>/stopped	Declare desired running and enqueue start under the caller credential. When following cold stop, this completes a cold restart and activates a new declaration snapshot.	Materialize a runtime vmm_machine, allocate vmm_memory, create loader fd 3, receive/verify launch state, then start kernel LWKT loops around vmm_vcpu_run().
touch <machine>/stopped	Declare desired stopped and enqueue immediate cold stop. touch; rm is the only topology activation boundary.	Drain vCPUs; call pmap_del_all_cpus() before unmapping runtime AVIC/BAR/memory; then notify remaining providers with STOP, revoke runtime capabilities and free the instance.
echo reset > <machine>/events	Enqueue immediate warm reset.	Recreate the current instance's COW runtime from its accepted loader boot snapshot. Do not rerun loader or activate declaration changes.
open(O_CREAT) <machine>/devices/<name>/provider	Create and exclusively hold the absent provider node, inserting a card into a declared slot; it may occur while running.	The vPCIe function owns runtime MMIO registrations and MSI-X routing. No vCPU waits for provider I/O.
Provider REGISTER after START	Accept static function profile, BAR layout and notifier needs.	vPCIe creates capability mappings and calls vmm_io_register_mmio() for synchronous/trapped ECAM or BAR pages. Direct BAR pages remain shared mappings.
Provider doorbell event	Consume a guest MMIO write without entering a userspace vCPU monitor.	The vPCIe MMIO callback marks the per-function doorbell endpoint readable.
Provider MSI/MSI-X record	Receive a completion notification from provider userspace.	Read current vPCIe MSI-X state and call vmm_irq_raise_msi().
Last provider close	Surprise card removal and node deletion.	Revoke provider BAR/DMA/event capabilities, stop I/O and MSI-X, mark the active function failed, then remove the provider name. Retain a direct guest BAR mapping until cold stop; active ECAM config may read as absent.
mv <from>/provider <to>/provider	Live provider attachment transfer.	Require an existing target slot with no provider; transfer the active provider attachment without moving either slot declaration.
rmdir devices/<name> / slot move	Future topology change.	Update the declaration and reject future provider creation. Retain the current runtime slot and provider generation until cold stop.
Last close of lease	Force ownership cleanup.	Force stop, close/revoke providers, revoke machine vnodes, then vmm_machine_destroy() after the gate drains.
rmdir <machine>	Delete a non-leased, fully stopped machine.	Reject with EBUSY unless stopped. Revoke machine vnode views and providers, then vmm_machine_destroy().

rmdir does not silently become an asynchronous power-off request. The ordinary stop action is touch stopped; deletion requires the stable stopped state. A leased machine is deleted only through the final lease close.

Execution Frontend Contrast

vmmfs:
  machine-owned vCPU LWKT
    -> vmm_vcpu_run()
    -> claimed exits re-enter
    -> unclaimed exits use fixed modern-platform policy
    -> never returns a device exit to userspace

NVMM:
  monitor-owned LWP
    -> NVMM_IOC_VCPU_RUN
    -> vmm_vcpu_run()
    -> claimed exits re-enter
    -> unclaimed exits become nvmm_vcpu_exit
    -> QEMU/microvm emulates its own device model

Both paths share memory backing, NPT/EPT, MMIO/PIO trap dispatch, IRQ routing, AVIC/APICv delivery and host CPU state protection. They differ only in who owns the unclaimed exit.

Explicit Non-Goals

• Do not require vmmfs to expose a generic ioctl control plane.
• Do not require NVMM/QEMU to use vmmfs or its kernel-owned PCIe fabric.
• Do not model legacy i440, PIIX, IDE, VGA, PIC or PIT in the VMM core.
• Do not expose a raw callback pointer or a raw guest-RAM mapping to userspace.
• Do not make provider I/O execute on a vCPU LWKT.
• Do not make AVIC/APICv a userspace ABI. It is an implementation detail of the common interrupt-delivery path.