Get the ARM Foundation Model

Download ARM Foundation Model: https://developer.arm.com/products/system-design/fixed-virtual-platforms.

Try the model: ./models/Linux64_GCC-4.9/Foundation_Platform --image examples/hello.axf

Fast Models [11.4.37 (Jun 19 2018)]
Copyright 2000-2018 ARM Limited.
All Rights Reserved.

terminal_0: Listening for serial connection on port 5000
terminal_1: Listening for serial connection on port 5001
terminal_2: Listening for serial connection on port 5002
terminal_3: Listening for serial connection on port 5003
Hello, 64-bit world!

ARM Software Stack

ARM Architecture has two worlds / execution environment: Trusted World and Non-Trusted World. These two execution environments run at the same time.
Non-Trusted world cannot access Trusted World. It protects trusted information (such as payment details) to be accessed by malicious software running in Non-Trusted world - although risks exist if the Trusted World is not configured properly or the interface between both Worlds are not correctly designed.

Most ARM (and non-ARM) platforms have different types of memory:

Trusted ROM: This memory is read-only. It contains the code that is executed at the early stage of the boot process.
Trusted SRAM: This memory is non-persistent (ie: the memory does not contain valid data at boot time) and can only contain data from the Trusted World.
DRAM: This memory is also non-persistent it can be configured as partially Trusted and Non-Trusted.
Non-Volatile memory (NVM): This memory is persistent; it can be eXecute-In-Place (XIP - ie: you can execute code located in the memory - NOR Flash technology can be used for XIP code) or Non-XIP (such as NAND, eMMC, SD card).

The Open-Source ARM Software stack contains these following modules:

ARM Trusted Firmware: it provides a reference implementation of secure world software for ARMv8-A cores.
UEFI: UEFI is a specification for Boot firmware. Tianocore (or EDK2) is an Open-Source implementation of this specification
Linux Kernel
Device Tree: Contain a static description of the platform. The device tree is passed to Linux kernel at boot time by UEFI. The aim of the device tree is to allow a same Linux kernel binary to run on various platforms of a same CPU architecture.
File System: Any Linux file system - Debian, Ubuntu, Android, ...

Run the full stack from pre-built binaries (Using u-boot)

Download the archive that contains ARM Trusted Firmware + u-boot (note: it looks Linaro ARM langing team does not support UEFI as the default boot loader on AArch64 anymore...), Linux Kernel and Device Tree at http://releases.linaro.org/members/arm/platforms/latest/fvp-latest-oe-uboot.zip (5.9 MB)
Download OpenEmbedded Minimal filesystem at https://releases.linaro.org/archive/15.05/openembedded/aarch64/vexpress64-openembedded_minimal-armv8-gcc-4.9_20150522-720.img.gz (83.9M) and decompress it with gzip``

Launch it!

 export MODEL=$PWD/Foundation_Platformpkg/models/Linux64_GCC-4.9/Foundation_Platform 
 export DISK=$PWD/vexpress64-openembedded_minimal-armv8-gcc-4.9_20150522-720.img
 ./run_model.sh

Run the full stack from pre-built binaries (Using UEFI)

Download the archive that contains ARM Trusted Firmware + u-boot (note: it looks Linaro ARM langing team does not support UEFI as the default boot loader on AArch64 anymore...), Linux Kernel and Device Tree at http://releases.linaro.org/members/arm/platforms/latest/fvp-latest-oe-uboot.zip (5.9 MB)
Download the archive that contains ARM Trusted Firmware + UEFI at http://releases.linaro.org/members/arm/platforms/latest/fvp-uefi.zip (834 KB)
Download OpenEmbedded Minimal filesystem at https://releases.linaro.org/archive/15.05/openembedded/aarch64/vexpress64-openembedded_minimal-armv8-gcc-4.9_20150522-720.img.gz (83.9M) and decompress it with gzip``

Launch it!

 export MODEL=$PWD/Foundation_Platformpkg/models/Linux64_GCC-4.9/Foundation_Platform
 export DISK=$PWD/vexpress64-openembedded_minimal-armv8-gcc-4.9_20150522-720.img
 ./run_model.sh

Stop when EFI Shell starts and go to the FS2 file system (EFI partition on OpenEmbedded image) with the EFI command: fs2:
Edit startup.nsh (EFI script called at the start of EFI Shell) with the command edit startup.nsh
Replace the Linux command line part dtb=fvp-base-gicv2-psci.dtb by dtb=fvp-foundation-gicv3-psci.dtb. Save your change with CTRL+S and quit with CTRL+Q
And start startup.nsh by calling the script from EFI Shell

Build and run the full stack

Install AArch64 GCC toolchain with sudo apt install gcc-aarch64-linux-gnu.

Build the software stack in a couple of command lines

Use Linaro's script. Check the section "Initialising a workspace" of the page Arm Platforms deliverables to get the exact destination of the script. At the moment of writing this document, the script is located here:

    wget https://community.arm.com/cfs-file/__key/communityserver-wikis-components-files/00-00-00-00-10/8267.armplat_5F00_1807.py
mv 8267.armplat_5F00_1807.py armplat_1807.py

Start the script python3 armplat_1807.py and select the following options:

 ## Please select a platform:
 
  1) Development boards   
  2) Fixed Virtual Platforms (FVPs)
 
 > 2
 
 
 ## Please select a platform:
 
  1) System Guidance
  2) Armv8 architecture
 
 > 2
 
 ## Please select a platform:
 
  1) Armv8-A Base Platform         -- 11.3.30+ (Rev C)
  2) Armv8-A Foundation Platform   -- 11.3.30+
 
 > 2
 WARNING: only avail. option is Armv8-A Foundation Platform with 64-bit software stack
 
 ## Please specify whether you want to:
 
  1) Build from source
  2) Use prebuilt configuration
 
 > 1
 
 ## Please select an environment:
 
  1) Linux kernel + filesystem
  2) Firmware only
 
 > 2
 WARNING: only avail. option is EDK II UEFI

 ## Your chosen configuration is shown below:
 
     +-------------+--------------------------------------------------------+
     | Workspace   | /tmp/linaro/                                           |
     | Platform    | Armv8-A Foundation Platform with 64-bit software stack |
     | Type        | Build from source                                      |
     | Environment | Firmware only                                          |
     | Firmware    | EDK II UEFI                                            |
     +-------------+--------------------------------------------------------+
 
 ## Proceed with this configuration?:
 
  1) Yes
  2) No
 
 > 1

Build the sources

 chmod a+x build-scripts/build-all.sh
 ./build-scripts/build-all.sh all

Run the newly built binaries

 export MODEL=/opt/Foundation_Platformpkg/models/Linux64_GCC-4.9/Foundation_Platform
 export BL1=output/fvp/fvp-uefi/uefi/bl1.bin
 export FIP=output/fvp/fvp-uefi/uefi/fip.bin
 ./model-scripts/fvp/run_model.sh

Build the software stack components by components from upstream

Create the directory that will contain all the ARM Software stack components:

mkdir src && cd src
export CROSS_COMPILE=aarch64-linux-gnu-

Build UEFI

Clone the EDK2 (a UEFI Open-Source implementation) repository

 git clone https://github.com/tianocore/edk2.git
 cd edk2
 git clone [email protected]:tianocore/edk2-platforms.git

Setup EDK2 build environment:

 . edksetup.sh
 make -C BaseTools
 export PACKAGES_PATH=$PWD:$PWD/edk2-platforms

Build EDK2 for FVP Foundation Model

 GCC5_AARCH64_PREFIX=aarch64-linux-gnu- build -a AARCH64 -p Platform/ARM/VExpressPkg/ArmVExpress-FVP-AArch64.dsc -t GCC5

Build ARM Trusted Firmware

Clone the repository

 git clone https://github.com/ARM-software/arm-trusted-firmware.git
 cd arm-trusted-firmware

Build ARM Trusted Firmware in DEBUG mode:

 CROSS_COMPILE=aarch64-linux-gnu- make PLAT=fvp all fip BL33=<edk2-dir>/Build/ArmVExpress-FVP-AArch64/DEBUG_GCC5/FV/FVP_AARCH64_EFI.fd DEBUG=1

ARM Trusted Firmware does not generate DTB (Device Tree Binary) for the Foundation model, we can generate it by hand:
```
 dtc -o fvp-foundation-gicv3-psci.dtb fdts/fvp-foundation-gicv3-psci.dts
```

Build Linux kernel

Clone the repository and use the branch linux-linaro-lsk-v4.14

 git clone https://git.linaro.org/kernel/linux-linaro-stable.git -b linux-linaro-lsk-v4.14
 cd linux-linaro-stable

Build the Linux kernel

 $ export CROSS_COMPILE=aarch64-linux-gnu-
 $ make ARCH=arm64 defconfig
 $ make ARCH=arm64 -j8 Image

 (...)
   LD      vmlinux
   SORTEX  vmlinux
   SYSMAP  System.map
   OBJCOPY arch/arm64/boot/Image

Run the newly built software stack

Set environment variables to let the model knows where to find the binaries

 export BL1=arm-trusted-firmware/build/fvp/debug/bl1.bin
 export FIP=arm-trusted-firmware/build/fvp/debug/fip.bin
 export KERNEL=linux-linaro-stable/arch/arm64/boot/Image
 export DTB=arm-trusted-firmware/fvp-foundation-gicv3-psci.dtb
 export DISK=vexpress64-openembedded_minimal-armv8-gcc-4.9_20150522-720.img

Launch the model

 /opt/Foundation_Platformpkg/models/Linux64_GCC-4.9/Foundation_Platform \
         --cores=1 \
         --secure-memory \
         --visualization \
         --use-real-time \
         --gicv3 \
         --data=${BL1}@0x0 \
         --data=${FIP}@0x8000000 \
         --data=${KERNEL}@0x80080000 \
         --data=${DTB}@0x82000000 \
         --data={INITRD}@0x84000000 \
         --block-device=$DISK

Some notes:

Versions:
- ARM Trusted Firmware: SHA1: e52ed092cde21 (Wed 12th September 2018)
- UEFI: SHA1: cfd10276ce (Monday 3th September 2018)
- Linux kernel: linux-linaro-lsk-v4.14 (Monday 10th September 2018)
Every time you make a modification to UEFI you need to rebuild the FIP (Firmware Image Package). There are two ways to rebuild the FIP:
- Use the ARM Trusted Firmware: make PLAT=fvp all fip BL33=/home/olivier/src/edk2/Build/ArmVExpress-FVP-AArch64/DEBUG_GCC49/FV/FVP_AARCH64_EFI.fd DEBUG=1
or
- Add fip_create to your PATH (eg: export PATH=../arm-trusted-firmware/tools/fip_create:$PATH) and set FIP_BIN with the location of the Firmware Image Package built by ARM Trusted Firmware (eg: export FIP_BIN=../arm-trusted-firmware/build/fvp/debug/fip.bin) (in my version of EDK2 you also need to set WORKSPACE=$PWD).

Linux NFS Boot on the ARM Foundation Model

NFS Boot allows to directly make changes to the file system from your host machine without the need to mount the disk image every time.

Ensure your Linux kernel support NFS

To enable NFS Boot you need to ensure your ARM Foundation Model Linux kernel has support for the Ethernet SMSC 91C111 controller. Check the configuration file of your Linux kernel source tree:

$ grep SMSC .config
CONFIG_NET_VENDOR_SMSC=y
CONFIG_SMSC911X=y
# CONFIG_SMSC911X_ARCH_HOOKS is not set
# CONFIG_SMSC9420 is not set
# CONFIG_SMSC_PHY is not set
# CONFIG_FB_SMSCUFX is not set

CONFIG_SMSC911X is set so it means you can use the ARM Foundation Model network controller with your current kernel binaries. If it is not set then you need to enable it in your kernel configuration and rebuild the kernel.

Configure Network support on the ARM Foundation Model

Start the model with the bridge network:

sudo ../Foundation_Platformpkg/models/Linux64_GCC-4.7/Foundation_Platform --cores=4 --secure-memory --visualization --gicv3 --data="$PWD/arm-trusted-firmware/build/fvp/debug/bl1.bin"@0x0 --data="$PWD/arm-trusted-firmware/build/fvp/debug/fip.bin"@0x08000000 --block-device="/home/olivier/prebuilt/lt-vexpress64-openembedded_minimal-armv8-gcc-4.9_20150912-729.img" --network=bridged &
sleep 5
sudo ifconfig ARM0 192.168.1.1 netmask 255.255.255.0

On the target:

sudo ifconfig eth0 192.168.1.2 netmask 255.255.255.0

Note: NAT network with the option --network=nat instead of --network=bridged also works.

More information on the network configuration here.

Setting up the NFS server on your host machine

We will need to copy the file system of your disk image onto your local disk. Let's have a look at the partitions present on your disk image and their locations - fdisk will help us:

$ /sbin/fdisk -lu ../prebuilt/lt-vexpress64-openembedded_minimal-armv8-gcc-4.9_20150912-729.img

Disk ../prebuilt/lt-vexpress64-openembedded_minimal-armv8-gcc-4.9_20150912-729.img: 3221 MB, 3221225472 bytes
255 heads, 63 sectors/track, 391 cylinders, total 6291456 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
        
                      Device Boot      Start         End      Blocks   Id  System
../prebuilt/lt-vexpress64-openembedded_minimal-armv8-gcc-4.9_20150912-729.img1   *          63      155646       77792    e  W95 FAT16 (LBA)
../prebuilt/lt-vexpress64-openembedded_minimal-armv8-gcc-4.9_20150912-729.img2          155648     6291455     3067904   83  Linux

You can see the Linux partition starts at the sector number 155648. fidsk says the sector size is 512 bytes.

We use losetup to mount the disk image on /dev/loop0:

$ losetup /dev/loop0 ../prebuilt/lt-vexpress64-openembedded_minimal-armv8-gcc-4.9_20150912-729.img -o $((155648 * 512))

file -s /dev/loop0 confirms us the Linux partition is actually a ext4 partition:

/dev/loop0: Linux rev 1.0 ext4 filesystem data, UUID=a9c947eb-c956-4c39-af5d-1fe3509121a9, volume name "rootfs" (needs journal recovery) (extents) (large files) (huge files)

We finally can mount the device /dev/loop0 to access the partition as part of our file system:

mkdir /tmp/disk
mount /dev/loop0 /tmp/disk

Now the disk image is mounted, we can copy its content to the directory /srv/nfs/lt-vexpress64-openembedded_minimal-armv8:

sudo mkdir /srv/nfs
sudo mount -o loop,offset=$((155648 * 512)) ../prebuilt/lt-vexpress64-openembedded_minimal-armv8-gcc-4.9_20150912-729.img /tmp/disk
sudo cp -ax /tmp/disk /srv/nfs/lt-vexpress64-openembedded_minimal-armv8
umount /dev/loop0
losetup -d /dev/loop0

Setting up NFS server

Check your Linux distribution documentation. For instance a simple installation on the Ubuntu distribution can be found here.
After installing NFS server, make sure the NFS service is started. You can also test your NFS installation is correctly set up by mounting the NFS file system on your host machine:

$ mkdir /tmp/open-embedded
$ sudo mount localhost:/srv/nfs/lt-vexpress64-openembedded_minimal-armv8 /tmp/open-embedded
$ ls /tmp/open-embedded/
bin  boot  dev  EFI  etc  home  lib  lost+found  media  mnt  proc  run  sbin  sys  tmp  usr  var

Next stage is to check whether you can mount the filesystem from the ARM Foundation model.
If you have any issue you can get the NFS error by looking at /var/log/syslog on the host machine: tail -n 20 /var/log/syslog

Once you manage to access the NFS file system from the model, you can now change the Linux kernel command line in UEFI in the file ArmPlatformPkg/ArmVExpressPkg/ArmVExpress-FVP-AArch64.dsc at the line gArmPlatformTokenSpaceGuid.PcdDefaultBootArgument|(..).

Replace root=/dev/vda2 by one of the following arguments:

DHCP: ip=dhcp root=/dev/nfs nfsroot=192.168.0.10:/srv/nfs/lt-vexpress64-openembedded_minimal-armv8,vers=3 rw
Static IP: ip=192.168.1.2:::255.255.255.0::eth0:: root=/dev/nfs nfsroot=192.168.1.1:/srv/nfs/lt-vexpress64-openembedded_minimal-armv8,vers=3 rw

Now, you know how to change and rebuild every components of the ARM Software stack!

Notes

The initial version of this document has been written as a Lab A Part blog available here.

I downloaded the Foundation model, from this link for Linux, today.
After Extracting the downloaded file, I executed ./FVP_ARM_Std_Library_11.14_21/FVP_ARM_Std_Library.sh for installation.
After installation, I tried running ./models/Linux64_GCC-6.4/FVP_Base_Cortex-A78x1 --image examples/hello.axf, as suggested in this page.
But it failed with message: FVP_Base_Cortex-A78x1: unrecognized option `--image'.
Can you help me solve this?

After that, I tried executing ./models/Linux64_GCC-6.4/FVP_Base_Cortex-A78x1.
It resulted in below list of errors:

terminal_2: Listening for serial connection on port 5000
terminal_0: Listening for serial connection on port 5001
terminal_1: Listening for serial connection on port 5002
terminal_3: Listening for serial connection on port 5003

Error: license error: License checkout for feature FVP_Arm_Std_Library with version 11.14 has been denied by Flex back-end. Error code: -1
Cannot find license file.
 The license files (or license server system network addresses) attempted are 
listed below.  Use LM_LICENSE_FILE to use a different license file,
 or contact your software provider for a license file.
Feature:       FVP_Arm_Std_Library
Filename:      /opt/arm/licenses/license.dat
License path:  /opt/arm/licenses/license.dat:
FlexNet Licensing error:-1,359.  System Error: 2 "No such file or directory"
In file: /tmp/plgbuild/abs_build/1003785_53700/trunk/work/fastsim/Framework/Runtime/SCXSimulationEngine.cpp:3052

Error: license error: License checkout for feature FVP_ARM_Cortex-A78 with version 11.14 has been denied by Flex back-end. Error code: -1
Cannot find license file.
 The license files (or license server system network addresses) attempted are 
listed below.  Use LM_LICENSE_FILE to use a different license file,
 or contact your software provider for a license file.
Feature:       FVP_ARM_Cortex-A78
Filename:      /opt/arm/licenses/license.dat
License path:  /opt/arm/licenses/license.dat:
FlexNet Licensing error:-1,359.  System Error: 2 "No such file or directory"
In file: /tmp/plgbuild/abs_build/1003785_53700/trunk/work/fastsim/Framework/Runtime/SCXSimulationEngine.cpp:3052

Error: license error: License checkout for feature FM_Simulator with version 11.14 has been denied by Flex back-end. Error code: -1
Cannot find license file.
 The license files (or license server system network addresses) attempted are 
listed below.  Use LM_LICENSE_FILE to use a different license file,
 or contact your software provider for a license file.
Feature:       FM_Simulator
Filename:      /opt/arm/licenses/license.dat
License path:  /opt/arm/licenses/license.dat:
FlexNet Licensing error:-1,359.  System Error: 2 "No such file or directory"
In file: /tmp/plgbuild/abs_build/1003785_53700/trunk/work/fastsim/Framework/Runtime/SCXSimulationEngine.cpp:3052

Error: license error: License checkout for feature SG_Simulator with version 11.14 has been denied by Flex back-end. Error code: -1
Cannot find license file.
 The license files (or license server system network addresses) attempted are 
listed below.  Use LM_LICENSE_FILE to use a different license file,
 or contact your software provider for a license file.
Feature:       SG_Simulator
Filename:      /opt/arm/licenses/license.dat
License path:  /opt/arm/licenses/license.dat:
FlexNet Licensing error:-1,359.  System Error: 2 "No such file or directory"
In file: /tmp/plgbuild/abs_build/1003785_53700/trunk/work/fastsim/Framework/Runtime/SCXSimulationEngine.cpp:3052

Error: license error: License checkout for feature FM_Simulator was denied because all elements (Flex_FVP_Arm_Std_Library,Flex_FVP_ARM_Cortex-A78,Original_Licenses,Original_Licenses_Uncounted) in MapFeatureOr statements were denied
In file: /tmp/plgbuild/abs_build/1003785_53700/trunk/work/fastsim/Framework/Runtime/SCXSimulationEngine.cpp:3052
ERROR: uncaught exception occured. Exception message follows:
Error: Wrong version of armctmodel. Version 11.14.21 (API Version 1.2) of the Portfolio was used to build the model. Please use identical major.minor versions of the Portfolio and the Tools.

Can you suggest me why I am seeing these errors and how to overcome this ?

All I want to do is, compile a customized kernel for ARM Cortex-A78 and test it.
Hope, I am at correct place.

Found the issue. It was due to wrong file, downloaded from the link.
Instead of downloading "FVP Library 11.14 (Linux)" (the 1st "download" button in the link), scroll down to "Armv8-A Architecture FVPs" ---> "Armv8-A Foundation AEM FVP" and download from here (link).

It now prints "Hello, 64-bit world!".
Thanks

oliviermartin/arm-foundation-software-stack.md