| title | author |
|---|---|
Jailhouse as type-1 hypervisor on ARM platforms |
Nikhil Devshatwar |
Jailhouse is a lightweight hypervisor suitable for industrial and automotive applications. It's a partitioning based hypervisor; it tries to partition the resources and avoids any sharing of the resources.
One problem with Jailhouse has been that it is a type-2 hypervisor. Which means, it is loaded with assistance from a host operating system. After the bootloader, an operating system boots, then it enables Jailhouse hypervisor and then other VMs can be created and started eventually.
- Boot time for other VMs is higher because of the serialized boot
- 1st VM(root_cell) => Jailhouse => 2nd VM
- Root cell dependency is a problem for safety certifying Jailhouse
- Failure in the root_cell bootup leads to failure in all VMs
- Extra overhead of removing and adding resources (CPU, memory, interrupts)
- Whenever VMs are created, the resources are removed from root cell
- Whenever VMs are destroyed, the resources are added back to root cell
On ARM64 platforms, typically there is ARM trusted firmware (aka ATF) running at EL3 and u-boot running at EL2. u-boot loads all the binaries required for kernel boot and jumps to the kernel. This is the first Linux kernel that boots up. When Jailhouse is enabled, this becomes the root cell kernel. To convert jailhouse as a type-1 hypervisor, we need to be able to perform all the essential tasks that a root cell kernel does via a small baremetal app.
- Bringup all cores using PSCI calls
- psci_cpu_on call from cpu_psci_cpu_boot while booting kernel
- Setup the hypervisor traps in EL2 and then lift to EL1 (each core)
- __hyp_stub_vectors called from head.S
- Setup SGI support for interrupting between cores
- gic_dist_init in the GIC v3 driver
- Load hypervisor firmware from storage media to the reserved memory
- request firmware call from jailhouse driver when jailhouse is enabled
- Fill up the hypervisor firmware and root cell config headers
- Done as part of the jailhouse_cmd_enable in Jailhouse driver
- Call arch_entry for Jailhouse on each core
- Done as part of Jailhouse driver's
jailhouse enablecmd implementation
- Done as part of Jailhouse driver's
- Issue hypercalls to create cells
- Implemented in the Jailhouse driver for the
jailhouse cell createcmds
- Implemented in the Jailhouse driver for the
- Make the memory loadable and copy the VM's boot binaries in it
- Issue a SET_LOADABLE hypercall for opening access, then copy binaries
- Issue hypercall to start cells
Now, do we really need a full fledged kernel to do this?
This can be implemented as part of a baremetal application with little bit of assembly and few linker scrips. Of course we will take some shortcuts. In embedded environment, the hardware partitioning is pretty much static, we exactly know how many VMs to be created and the binaries used to boot OS inside the VMs.
e.g. for the proof-of-concept I tried to package all the binaries in one image instead of having to load them from storage media. Optionally, all the binaries can be loaded by u-boot into the RAM before it starts jumping to the baremetal app
The baremetal app is written to do all of the kernel's tasks as mentioned above. In details, it does following:
- Bringup all the ARM cores in the system (Issue PSCI CPU_ON calls using SMC calling convention)
- Do basic setup for each core once brought up
- Setup the hypervisor stubs, switch to EL1
- EL1 switching code copied from u-boot
- Setup stack for each core
- No need to enable MMU for the baremetal app to run
- Find out the addresses for all the binaries
- Hypervisor firmware, VM configs, boot binaries, DTBs
- If u-boot loads it, it will be hard-coded
- For PoC, I packaged all binaries as sections of the single image
- Fill up the hypervisor header and setup the root config
- Fill up struct jailhouse_header with details like no of CPUs Find out hypervisor entrypoint and jump all cores in the arch_entry
- At this stage, hypervisor is enabled
- Create all the required VMs using the config files
- Fill up id numbers for the new cells in the config header
- Issue HVC calls to create new cells
- Issue HVC calls to make the cells loadable
- Copy the binaries into the new cell's physcial address space
- Hardcode the load addresses of binaries in physical memory
- Copy just the baremetal binary for a simple inmate
- Copy linux-loader, arguments, kernel, dtb, initramfs, etc for a Linux inmate
- Start all the non-root cell VMs
- Issue HVC calls to start all the non-root cells
- Finally load and run the root cell
- Since the root cell now only has fewer resources, an updated DT reflecting the correct device description is required
- Root cell DTB should only describe all the resources that the root cell VM will use after all the cells are created
- e.g. If the root cell config describes 4 CPUs and inmate config owns 2 out of the 4 CPUs, then only the remaining 2 CPUs are usable on the root cell after all partitioning
| Type-1 hypervisor feature | Jailhouse feasibility | Status | Details |
|---|---|---|---|
| Boot the hypervisor before any of the VMs boot | Possible | POC will demonstrate this | After ATF/u-boot, Jailhouse is to be loaded, then VMs will be created and all VMs will boot in parallel |
| Hypervisor has storage drivers to load the VMs config and boot images | Not required | We don't need this. u-boot can load all the images required. For embedded use cases, fixed no of VMs will be instantiated. All can be done at boot time | |
| Failure to boot any VM won't impact hypervisor | Feasible | POC will enable this | No dependency on successful boot of root cell on Jailhouse |
| Ability to reboot other VMs from one of the VM | Feasible | TBD | Need to modify the current driver to run jailhouse enable without loading the hypervisor firmware. |
| All VMs should be able to issue hypercalls | Feasible | TBD | For any communication with the hypervisor, an OS needs a kernel driver. Once the driver is modified for not loading hypervisor, it can be run on other non-root cells as well. |
| Provide a dedicated interface for control and logging | Feasible | PoR | Use a dedicated UART for hypervisor logs. Control a timer to measure the latencies and for detection of VM failures |
| Recovery dependency on root cell - Check if a VM fails, take the decision and reboot it w/o waiting for some VM to tell you to | Maybe | TBD | If the VM failure detection is failproof, this can be implemented. Since all the boot binaries are available all the time, Jailhouse can reboot VMs w/o root cell's dependency. So even if there is failure in root cell, other cells can still recover. |
A couple of typos:
You may also want to elaborate a bit further the needed modifications/new features of U-Boot (if any).
Thanks.