Skip to content

Instantly share code, notes, and snippets.

@nikhildevshatwar

nikhildevshatwar/jailhouse-type-1-poc-for-ARM.md

Last active December 6, 2019 12:13
Show Gist options
  • Save nikhildevshatwar/e17121c1355422a61fddaa52615fc7a9 to your computer and use it in GitHub Desktop.
Save nikhildevshatwar/e17121c1355422a61fddaa52615fc7a9 to your computer and use it in GitHub Desktop.
Download ZIP
Proof of concept to boot Jailhouse as
Raw
jailhouse-type-1-poc-for-ARM.md
title author
Jailhouse as type-1 hypervisor on ARM platforms
Nikhil Devshatwar

Jailhouse as type-1 hypevisor on ARM platforms

Jailhouse is a lightweight hypervisor suitable for industrial and automotive applications. It's a partitioning based hypervisor; it tries to partition the resources and avoids any sharing of the resources.

One problem with Jailhouse has been that it is a type-2 hypervisor. Which means, it is loaded with assistance from a host operating system. After the bootloader, an operating system boots, then it enables Jailhouse hypervisor and then other VMs can be created and started eventually.

Problems with Jailhouse being type-2

  • Boot time for other VMs is higher because of the serialized boot
    • 1st VM(root_cell) => Jailhouse => 2nd VM
  • Root cell dependency is a problem for safety certifying Jailhouse
    • Failure in the root_cell bootup leads to failure in all VMs
  • Extra overhead of removing and adding resources (CPU, memory, interrupts)
    • Whenever VMs are created, the resources are removed from root cell
    • Whenever VMs are destroyed, the resources are added back to root cell

Converting Jailhouse to type-1 for ARM platforms

On ARM64 platforms, typically there is ARM trusted firmware (aka ATF) running at EL3 and u-boot running at EL2. u-boot loads all the binaries required for kernel boot and jumps to the kernel. This is the first Linux kernel that boots up. When Jailhouse is enabled, this becomes the root cell kernel. To convert jailhouse as a type-1 hypervisor, we need to be able to perform all the essential tasks that a root cell kernel does via a small baremetal app.

What are essential tasks root cell kernel does to start multiple VMs

  • Bringup all cores using PSCI calls
    • psci_cpu_on call from cpu_psci_cpu_boot while booting kernel
  • Setup the hypervisor traps in EL2 and then lift to EL1 (each core)
    • __hyp_stub_vectors called from head.S
  • Setup SGI support for interrupting between cores
    • gic_dist_init in the GIC v3 driver
  • Load hypervisor firmware from storage media to the reserved memory
    • request firmware call from jailhouse driver when jailhouse is enabled
  • Fill up the hypervisor firmware and root cell config headers
    • Done as part of the jailhouse_cmd_enable in Jailhouse driver
  • Call arch_entry for Jailhouse on each core
    • Done as part of Jailhouse driver's jailhouse enable cmd implementation
  • Issue hypercalls to create cells
    • Implemented in the Jailhouse driver for the jailhouse cell create cmds
  • Make the memory loadable and copy the VM's boot binaries in it
    • Issue a SET_LOADABLE hypercall for opening access, then copy binaries
  • Issue hypercall to start cells

Now, do we really need a full fledged kernel to do this?

This can be implemented as part of a baremetal application with little bit of assembly and few linker scrips. Of course we will take some shortcuts. In embedded environment, the hardware partitioning is pretty much static, we exactly know how many VMs to be created and the binaries used to boot OS inside the VMs.

e.g. for the proof-of-concept I tried to package all the binaries in one image instead of having to load them from storage media. Optionally, all the binaries can be loaded by u-boot into the RAM before it starts jumping to the baremetal app

Extending bootloader to load multiple kernels instead of single one

The baremetal app is written to do all of the kernel's tasks as mentioned above. In details, it does following:

  • Bringup all the ARM cores in the system (Issue PSCI CPU_ON calls using SMC calling convention)
  • Do basic setup for each core once brought up
    • Setup the hypervisor stubs, switch to EL1
    • EL1 switching code copied from u-boot
    • Setup stack for each core
  • No need to enable MMU for the baremetal app to run
  • Find out the addresses for all the binaries
    • Hypervisor firmware, VM configs, boot binaries, DTBs
    • If u-boot loads it, it will be hard-coded
    • For PoC, I packaged all binaries as sections of the single image
  • Fill up the hypervisor header and setup the root config
    • Fill up struct jailhouse_header with details like no of CPUs Find out hypervisor entrypoint and jump all cores in the arch_entry
    • At this stage, hypervisor is enabled
  • Create all the required VMs using the config files
    • Fill up id numbers for the new cells in the config header
    • Issue HVC calls to create new cells
    • Issue HVC calls to make the cells loadable
  • Copy the binaries into the new cell's physcial address space
    • Hardcode the load addresses of binaries in physical memory
    • Copy just the baremetal binary for a simple inmate
    • Copy linux-loader, arguments, kernel, dtb, initramfs, etc for a Linux inmate
  • Start all the non-root cell VMs
    • Issue HVC calls to start all the non-root cells
  • Finally load and run the root cell
    • Since the root cell now only has fewer resources, an updated DT reflecting the correct device description is required
    • Root cell DTB should only describe all the resources that the root cell VM will use after all the cells are created
    • e.g. If the root cell config describes 4 CPUs and inmate config owns 2 out of the 4 CPUs, then only the remaining 2 CPUs are usable on the root cell after all partitioning

Type-1 Hypervisor Features

Type-1 hypervisor feature Jailhouse feasibility Status Details
Boot the hypervisor before any of the VMs boot Possible POC will demonstrate this After ATF/u-boot, Jailhouse is to be loaded, then VMs will be created and all VMs will boot in parallel
Hypervisor has storage drivers to load the VMs config and boot images Not required We don't need this. u-boot can load all the images required. For embedded use cases, fixed no of VMs will be instantiated. All can be done at boot time
Failure to boot any VM won't impact hypervisor Feasible POC will enable this No dependency on successful boot of root cell on Jailhouse
Ability to reboot other VMs from one of the VM Feasible TBD Need to modify the current driver to run jailhouse enable without loading the hypervisor firmware.
All VMs should be able to issue hypercalls Feasible TBD For any communication with the hypervisor, an OS needs a kernel driver. Once the driver is modified for not loading hypervisor, it can be run on other non-root cells as well.
Provide a dedicated interface for control and logging Feasible PoR Use a dedicated UART for hypervisor logs. Control a timer to measure the latencies and for detection of VM failures
Recovery dependency on root cell - Check if a VM fails, take the decision and reboot it w/o waiting for some VM to tell you to Maybe TBD If the VM failure detection is failproof, this can be implemented. Since all the boot binaries are available all the time, Jailhouse can reboot VMs w/o root cell's dependency. So even if there is failure in root cell, other cells can still recover.
@claudioscordino
Copy link

Hi Nikhil,
any news about this development ?
Best regards.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment