Getting Started with CheriBSD 24.05

Robert N. M. Watson (University of Cambridge) and Brooks Davis (SRI International)

This is a living document that describes how to get up and running with CheriBSD on Morello and CHERI-RISC-V. Topics include how to download and install prebuilt CheriBSD images, how to build your own images, how third-party packages work, and where to find further information and support.

The document describes CheriBSD as of the 24.05 release, unless explicitly stated in sections referring to earlier or later releases.

This document is a work-in-progress. Feedback and contributions are welcomed. Please see our GitHub Repository for the source code and an issue tracker. There is a rendered version on the web, which is automatically updated when the git repository is committed to.

Acknowledgements

The authors gratefully acknowledge Brian Campbell (University of Edinburgh), Jessica Clarke (University of Cambridge), George Neville-Neil (MSB Associates), and Konrad Witaszczyk (University of Cambridge) for their contributions to this document.

This work was supported in part by the Innovate UK project Digital Security by Design (DSbD) Technology Platform Prototype, 105694, and Innovate UK project Developing and Evaluating an Open-Source Desktop for Arm Morello, 10027440.

This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR001122C0110 ("ETC") and Defense Advanced Research Projects Agency (DARPA) Contract No. HR001123C0031 ("MTSS"). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Defense Advanced Research Projects Agency (DARPA).

Introduction

CheriBSD is a version of the open-source FreeBSD operating system that has been extended to use CHERI architectural protections. CheriBSD is intended to:

  • Provide a clean, in-depth illustration and template for how CHERI support can be integrated with a general-purpose, MMU-enabled operating system;
  • Enable validation and evaluation of CHERI-extended processor architectures and microarchitectures;
  • Act as an OS research platform for how operating systems can use CHERI;
  • Support the development and evaluation of CHERI-enabled applications; and
  • Provide a foundation for CHERI-enabled software demonstrations.

This guide will support you in getting up and running with CheriBSD on the CHERI-RISC-V and Arm Morello platforms. As platforms vary substantially, from instruction-set emulators and executable formal models through to FPGA- and ASIC-based hardware implementations, there are several paths that can be taken to reach a shell prompt and start productive work. You might find that you want to build CheriBSD from scratch, or simply use one of our downloadable pre-compiled images. You may want to run directly from a live image in an emulator, or alternatively want to use an installer image to install onto an internal disk in an Arm Morello-based server or workstation.

Background

CHERI

CHERI is an extension to processor Instruction-Set Architectures (ISAs) to introduce support for fine-grained memory protection and software compartmentalization. This is done by introducing a new architectural data type, the CHERI capability, which can be used to implement pointers with strong integrity, provenance validity, spatial safety, and monotonicity. CHERI requires new microarchitectural support, including support for memory tags that track the provenance validity of capabilities in registers and memory. CHERI extensions have been specified for:

  • Arm's 64-bit Armv8-a ISA, with the prototype architecture known as Morello. The multi-GHz, multi-core, superscalar Morello design is a CHERI-extended version of the Arm Neoverse N1, available on an evaluation board from Arm.
  • The open-source 32-bit and 64-bit RISC-V ISA, with the extended ISA known as CHERI-RISC-V. CHERI-RISC-V is available via multiple open-source soft processor cores for use on FPGA.

ISA-level emulation platforms exist for both architectures (QEMU for CHERI-RISC-V and Morello, and Arm's Morello FVP for Morello). Unless you have access to a Morello board or a high-end (and supported) FPGA development platform, using QEMU-CHERI will be the easiest way to get started using CHERI. This guide covers all of these use cases.

You can learn more about CHERI by reading the technical report, An Introduction to CHERI.

CHERI C/C++

CHERI C and CHERI C++ are programming-language dialects closely tied to CHERI-based code generation and Application Binary Interfaces (ABIs) that directly support referential and spatial memory safety. These language variants compile to pure-capability code, which implements all language- and sub-language pointers using CHERI capabilities rather than architectural integers. With suitable OS support, they can also support temporal memory safety.

CHERI C/C++ support is implemented by CHERI LLVM. The Clang/LLVM compiler is able to statically identify and report as warnings many of the incompatibilites between C/C++ and CHERI C/C++ that would result in run-time CHERI exceptions. However, a correctly compiled CHERI C/C++ program might still trigger CHERI capability violations at run time (e.g., due to pointer misalignment) that can be diagnosed with GDB.

You can learn more about CHERI C/C++ and its CheriABI run-time environment by reading the technical report, CHERI C/C++ Programming Guide.

CheriBSD

CheriBSD is an extended version of the open-source FreeBSD operating system. FreeBSD is a mature UNIX-based operating system able to run on multiple hardware architectures, and widely used in industry in service, appliance, and embedded environments. CheriBSD contains various CHERI-based extensions including support for kernel and userspace memory safety, and supports multiple software compartmentalization models. CheriBSD runs on both CHERI-RISC-V and Morello with nearly identical feature sets (Morello includes hardware virtualisation extensions not present in the CHERI-RISC-V cores, and so CHERI-extended bhyve hypervisor support is not available for CHERI-RISC-V).

CheriBSD features

CheriBSD extends FreeBSD with added CHERI-enabled features, such as kernel and userspace memory safety:

Some of these features are in the main CheriBSD branch; other experimental features remain on development branches.

Kernel compilation modes

The CheriBSD kernel can be compiled as hybrid, pure-capability, or benchmark code:

  • The hybrid kernel enables capability use in userspace while making relatively little use of capabilities in the kernel's own implementation.
  • The pure-capability kernel implements strong referential and spatial memory protection internally in the kernel, protecting against memory-safety vulnerabilities in components such as the network stack and system-call layer.
  • The benchmark kernel implements the Benchmark ABI which improves performance on the Arm Morello board at the cost of reduced security. The benchmark kernel is considered appropriate for a performance evaluation of the Arm Morello architecture and third-party software executed on it. It should not be used for a security evaluation and the pure-capability kernel should be used instead.

Userlevel process environments

CheriBSD 24.05 likewise supports three different userspace execution environments:

  • Hybrid processes provide strong binary compatibility with the non-CHERI version of the same architecture -- for example, aarch64 on Morello.

  • CheriABI processes support pure-capability code execution. This process environment implements strong referential, spatial, and heap temporal memory protection through the system-call interface, dynamic linker, language runtime including heap memory allocators, and compiler-generated code. This protects against memory memory-safety vulnerabilities in both system services and applications. See Building for CheriABI for information on building for CheriABI. CheriABI is described in an ASPLOS 2019 paper.

  • Benchmark ABI processes implement a code-generation model similar to CheriABI, but with relaxed program-counter capability bounds that enable improved performance prediction on the Arm Morello platform. See Building for the Benchmark ABI. The Benchmark ABI is described in a technical report on the topic from Arm and Cambridge.

All of these environments can be used over either of the hybrid or pure-capability kernels. You can determine the process environment that a particular process is using via the procstat(1) command, whose default output includes an EMUL column indicating the process ABI:

% procstat -a
  PID  PPID  PGID   SID  TSID THR LOGIN    WCHAN     EMUL           COMM
...
 3197  3193  3193  3193     0   1 robert   select    FreeBSD ELF64C sshd
 3214  3170  3214  3170  3170   1 robert   select    FreeBSD ELF64  bash
...

In this example, the sshd daemon is using CheriABI (FreeBSD ELF64C), and the bash command is using the Hybrid ABI (FreeBSD ELF64).

Pre-compiled third-party software applications (packages) are provided for each of these ABIs, although CheriABI and Benchmark ABI packages are currently considered experimental. This is discussed further in the chapter on packages.

CheriABI desktop environment (experimental)

The installer has the option to install a desktop environment using the Mali Bifrost GPU on the Morello System-on-Chip. The option installs a basic desktop environment using KDE and Wayland compiled for CheriABI with the cheri-desktop package. It also installs a hybrid ABI Chromium web browser via the cheri-desktop-hybrid-extras package.

Userlevel heap temporal memory safety (experimental)

Since CheriBSD 23.11 there has been support for userlevel heap temporal memory safety based on a load-barrier extension to Cornucopia, which is inspired by garbage-collection techniques. A forthcoming paper at ASPLOS 2024 on the Cornucopia Reloaded technique will describe these differences in detail.

This feature involves a collaboration between the kernel (which provides asynchronous capability revocation with VM acceleration) and the userlevel heap allocator (which quarantines freed memory until revocation of any pointers to it) to ensure that memory cannot be reallocated until there are no outstanding valid capabilities lasting from its previous allocation. The userlevel memory allocator and the kernel revoker share an epoch counter that counts the number of completed atomic revocation sweeps. Memory added to quarantine in one epoch cannot be removed from quarantine until at least one complete epoch has passed -- i.e., the epoch counter has been increased by 2. More information on temporal memory safety support can be found in the mrs(3) man page:

man mrs

Checking whether temporal safety is globally enabled

Use the sysctl(8) command to inspect the value of the security.cheri.runtime_revocation_default system MIB entry:

sysctl security.cheri.runtime_revocation_default

This sysctl sets the default policy for revocation used by processes on startup. We recommend setting this in /boot/loader.conf, which is processed by the boot loader before any user processes start.

Controlling revocation by binary or process

You can forcefully enable or disable revocations for a specific binary or process with elfctl(1) or proccontrol(1) and ignore the default policy:

elfctl -e <+cherirevoke or +nocherirevoke> <binary>
proccontrol -m cherirevoke -s <enable or disable> <program with arguments>

You can read more on these commands in the mrs(3) man page.

Synchronous revocation

Revocation normally occurs asynchronously, with a memory quarantine preventing memory reuse until revocation of any pointers to that memory. You can insert a specific call in your program to await synchronous revocation:

malloc_revoke();

Synchronous revocation on every call to free() can be configured using the _RUNTIME_REVOCATION_EVERY_FREE_ENABLE environmental variable.

Note that synchronous revocation can incur extremely high expense.

Monitoring revocation in processes

Use the procstat cheri -v command to inspect the CHERI memory-safety state of a target process. For example:

# procstat cheri -v 923 1012
  PID COMM                C QUAR  RSTATE                              EPOCH
  923 seatd               P  yes    none                                  0
 1012 Xorg                P  yes    none                               0xd2

Both processes in this example use a pure-capability process environment, have quarantining enabled, and are not currently revoking. seatd has never needed to perform a revocation pass, as it remains in epoch 0, whereas X.org has a non-zero epoch and has performed multiple passes.

Userlevel software compartmentalization (experimental)

CheriBSD implements two different CHERI-enabled software compartmentalization models:

  • Colocated processes (co-processes) compartmentalization accelerates UNIX Inter-Process Communication (IPC) and context switching by colocating multiple processes in the same address space, separating them using CHERI capabilities. Support for co-processes is maintained in an experimental development branch and is not included in current software releases. For more information see the Colocation Tutorial wiki page.
  • Dynamic-linker-based compartmentalization isolates shared libraries within a process using CHERI capabilities limiting the access of attackers who have achieved arbitrary code execution within a library. The linker-based library compartmentalization model has been included since the 22.12 release of CheriBSD. See the c18n(3) man page on an installed system for more information.

Library compartmentalization

CheriBSD's library compartmentalization feature (c18n) executes each dynamic library within a compartmentalization-enabled process in its own protection domain. When c18n is enabled, the run-time linker grants libraries capabilities only to resources (global variables, APIs) declared in their ELF linkage. Function calls that cross domain boundaries are interposed on by domain-crossing shims implemented by the run-time linker.

The adversary model for these compartments is one of trusted code but untrustworthy execution: a library such as libpng or libjpeg is trusted until it begins dynamic execution -- and has potentially been exposed to malicious data. With library compartmentalization, an adversary who achieves arbitrary code execution within the library at run time will be able to reach only the resources (and further attack surfaces) declared statically through its linkage. The programmer must then harden that linkage, and any involved APIs, to make them suitable for adversarial engagement -- but the foundation of isolation, controlled access, and controlled domain transition is provided by the c18n implementation.

In addition to a modified run-time linker, modest changes have been made to the aarch64c calling convention to avoid assumptions such as implicit stack sharing between callers and callees across library boundaries when passing variadic argument lists. This modified ABI is now used by all CheriABI binaries in CheriBSD, and so off-the-shelf aarch64c binaries and libraries can be used with library compartmentalization without recompilation to the modified ABI. More information on library compartmentalization can be found in the c18n(3) man page:

man c18n

Enabling library compartmentalization system-wide

Compartmentalization can be enabled for all eligible new processes with sysctl:

sysctl security.cheri.lib_based_c18n_default=1

Enabling library compartmentalization temporarily for a particular binary

To run a particular binary with library compartmentalization enabled just for this run, use proccontrol:

proccontrol -m cheric18n -s enable ./helloworld

Enabling library compartmentalization permanently for a particular binary

To enable library compartmentalization permanently for a particular binary, run the following command to set a flag in the binary:

elfctl -e +cheric18n ./helloworld

This can be reversed by replacing +cheric18n with -cheric18n.

You can confirm whether c18n is permanently enabled for a binary by inspecting it using the elfctl command:

elfctl ./helloworld

The output will contain two lines like the following indicating whether c18n is enabled or disabled.

cheric18n       'Force Enable CHERI library-based compartmentalisation' is set.
nocheric18n     'Force Disable CHERI library-based compartmentalisation' is unset.

Note that nocheric18n disables c18n for the binary, overriding the system-wide default.

Tracing compartment-boundary crossings

The BSD ktrace(1) command is able to trace compartment-boundary crossings. To enable this feature, set the LD_UTRACE_COMPARTMENT environmental variable, which will cause the run-time linker to emit records using the utrace(2) system call. Run the program under ktrace with the -tu argument to capture only those records (and not a full system-call trace):

env LD_UTRACE_COMPARTMENT=1 ktrace -tu ./helloworld

The resulting ktrace.out file can be viewed using the kdump(1) command:

kdump

It is important to understand, however, that simply isolating running code is almost always insufficient to achieve robust sandboxing. The competent adversary will now consider further rights and attack surfaces to explore in search of further vulnerabilities. While this increased work factor of finding additional vulnerabilities is an important part of compartmentalization, internal software APIs are rarely well suited to be security boundaries without performing additional hardening. With this in mind, you can:

  • Inspect the source code, output from objdump, and output from chericat to assess the robustness of this compartmentalization. You can install objdump with pkg64 install llvm-base and chericat with pkg64c install chericat.
  • Consider larger software architectural changes that will allow a library to be used more robustly when running within a computerment.

Library compartmentalization has the potential to significantly improve software integrity and confidentiality properties in the presence of a strong adversary. However, it is also limited by the abstraction being around the current library operational model.

bhyve hypervisor (experimental)

CheriBSD implements, on an experimental development branch, CHERI extensions to FreeBSD's bhyve Type 2 hypervisor on the Morello architecture. This allows bhyve to host CHERI-enabled virtual machines, including those running CheriBSD.

Unsupported FreeBSD features

Certain FreeBSD features are currently unavailable in CheriBSD, including:

  • DTrace

Many other kernel features are not yet well validated, including:

  • Non-UFS file systems
  • Most optional modules, such as various firewall systems

Alpha ZFS support

The 24.05 release includes Alpha support for the ZFS file system. It is lightly tested, but works without known issues.

Reporting bugs

Bug reports relating to these under-validated subsystems would be appreciated, and may be submitted to the CheriBSD GitHub issue tracker.

Getting CheriBSD

CheriBSD can be installed by downloading or building one of two types of disk images:

  • Memstick images that boot and automatically run bsdinstall, the FreeBSD installer, which can be used to prepare a filesystem on a disk, and then install CheriBSD onto it. These will typically be used on Arm Morello boards.

  • Live images that boot CheriBSD to a login prompt for interactive use. These will typically be used on instruction-set emulators such as QEMU and the Arm Morello FVP, as well as on FPGA.

Downloading image files

Memstick and live images can both be downloaded from the CheriBSD website. Images are compressed using the UNIX xz command, and must be decompressed before they can be used; for example by running:

unxz cheribsd-memstick-arm64-aarch64c-24.05.img.xz

Building image files

CheriBSD can be built natively on an existing CheriBSD system (e.g., on an Arm Morello box), and it can also be cross-built from FreeBSD, Linux, and macOS. Building natively from an ISA-level emulator, such as QEMU, is not recommended for performance reasons.

  • Native builds follow the same build procedure as used for the baseline FreeBSD operating system. Suitable git command-line substitions must be made to use CheriBSD rather than FreeBSD source code.

  • CheriBSD cross-build orchestration is performed by the Python-based cheribuild tool, which is able to build CHERI-enabled toolchain, operating-system stack, and various applications and other software used by the CHERI project.

Key source repositories

CheriBSD repositories currently reside in the CTSRD-CHERI GitHub Organization:

Unless you intend to modify CheriBSD, CHERI Clang/LLVM, CHERI GDB, or QEMU-CHERI, you should not generally need to manually check out or compile most of the above repositories. Instead, existing prebuilt software images/packages should be used, or the cheribuild command, which will orchestrate software cross-build for you.

Checking out cheribuild

cheribuild is a Python-based build orchestration tool that is the preferred way to cross-build CheriBSD. It can be checked out from GitHub:

git clone git@github.com:CTSRD-CHERI/cheribuild.git

Building and running CheriBSD/RISC-V in QEMU

The following command builds a CheriBSD/RISC-V live image, and boots it in QEMU-CHERI:

% ./cheribuild.py -d build-and-run-cheribsd-riscv64-purecap

Building and running CheriBSD/Morello in QEMU

The following command builds a CheriBSD/Morello live image, and boots it in QEMU-CHERI:

% ./cheribuild.py -d build-and-run-cheribsd-morello-purecap

Building CheriBSD/Morello release images

The following command builds a CheriBSD/Morello installer (memstick) image suitable to write to a USB stick:

% ./cheribuild.py -d --clean cheribsd-release-morello-purecap

The resulting image file will be generated in a file with a name along the lines of: output/cheribsd-release-morello-purecap/FreeBSD-14.0-CURRENT-arm64-aarch64c-memstick.img. This is relative to your cheribuild destination root, which is, by default, ~/cheri.

Obtaining further information

The cheribuild README.md provides detailed information on various parameters and targets its supports, as well as its dependencies.

CheriBSD on an Arm Morello board

Arm's Morello board contains a quadcore CHERI-extended Neoverse N1 processor design, as well as two management microcontrollers. CheriBSD is able to run on, and fully utilize, Morello's CHERI support to implement fine-grained kernel and userspace memory safety, and on experimental branches, also implement scalable software compartmentalization. You can install CheriBSD into the hard disk in your Morello box using a USB stick installer image. You will first need to upgrade the firmware, as older firmware revisions contain bugs or lack features essential to running the OS. This chapter explains how to:

  • Connect to the Morello management and system consoles via USB,
  • Upgrade the board's firmware, and
  • Install CheriBSD from a USB stick.

It is also possible to run CheriBSD from a USB stick, or netboot CheriBSD on the board, but those topics are not currently covered by this guide.

It is important that the most recent Arm-provided firmware be installed on your Morello board, as the board has most likely shipped with an older firmware version. If you are not running the latest version, you may experience hard-to-diagnose behaviour or reliability problems.

Accessing the Morello console

You can manage the Morello board, including reaching its multiple serial consoles and firmware flash storage, via a USB cable connected to a second computer -- typically a workstation or remote management server. When you plug the USB cable into your workstation, a number of different devices will attach (depending on device drivers on your workstation), including:

  • Arm Morello USB debug and trace
  • Flash storage used for firmware (an on-board SD card)
  • Two 4-port USB-to-serial converters

Each controller has a unique serial number ending in the letter "A" or B". The serial-port converters attach to UARTs associated with:

  • The MCC management microcontroller (converter B port 0)
  • The PCC management microcontroller (converter B port 1)
  • The Morello system console (converter B port 2)

In normal use, only the MCC and Morello system console ports will be used. You can connect to these using a serial access tool such as cu(1), or a full terminal emulation environment such as screen(1). The precise names of the /dev device nodes appearing on the workstation depend on the operating system being used. Often access will also require running the terminal command under su(8) or sudo(8), or adding your user to a dialer or similar group to grant access to the USB device nodes.

Connecting from FreeBSD

FreeBSD names USB device nodes in the order they probe and attach. Typical command lines to access the MCC and Morello consoles are:

  • MCC console: cu -l /dev/cuaU0 -s 115200
  • Morello console: cu -l /dev/cuaU2 -s 115200

Connecting from Apple macOS

macOS creates /dev/cu.usbserial- device nodes that embed the device serial number (including A or B) and port number. Typical command lines to access the MCC and Morello consoles are:

  • MCC console: cu -l /dev/cu.usbserial-00FT41683097B0 -s 115200
  • Morello console: cu -l /dev/cu.usbserial-00FT41683097B2 -s 115200

You will need to substitute the serial numbers of the USB converters in your specific Morello board for these commands to work.

Connecting from Windows

Windows requires the installation of FTDI drivers to access the Morello board's USB ports, and the OS should identify the appropriate driver version once the board is connected. Plink is a CLI bundled with PuTTY, a third-party terminal emulator and SSH client freely available from the Windows Store and the developer's official website. After installing the drivers and PuTTY, typical command lines to access the MCC and Morello consoles are:

  • MCC console: plink -serial \\.\COM3 -sercfg 115200
  • Morello console: plink -serial \\.\COM5 -sercfg 115200

Do note that Windows assigns serial numbers at random.

Upgrading the Morello firmware

Newer Morello firmware versions contain important feature extensions (such as support for network booting) and also bug fixes (such as persistent variable support for UEFI, which is relied on for CheriBSD boot).

More detailed information from Arm on Morello firmware builds and configuration can be found here. The instructions in this guide are an abbreviated and slightly modified set of instructions better tuned to CheriBSD requirements.

Setting the system clock

Start by setting the system date and time in the MCC. Although the Arm documentation recommends setting your local time, we recommend using UTC with CheriBSD:

Cmd> debug
Debug> time
Debug> date
Debug> exit

Downloading prebuilt firmware

Arm provides prebuilt firmware images via the Morello GitLab instance, although you can also build your own. Start by downloading this directory tree, which can be done by performing a git clone:

git clone https://git.morello-project.org/morello/board-firmware.git

Reflashing the firmware

The Morello firmware is stored on a 2.0GiB SD Card on the Morello board within the case. The SD Card content can be exposed as a USB mass storage device accessible over the USB cable from your workstation. This may require first enabling USB support on the MCC console:

Cmd> USB_ON
Enabling debug USB...

The SD Card will attach as a USB disk containing a FAT filesystem. Depending on your workstation operating system, this may be automatically mounted (e.g., on macOS), or require manual mounting (e.g., on FreeBSD). Once mounted, delete the existing firmware and configuration, and then copy on the new version (substituting an appropriate mounted directory for /mnt below):

rm -rf /mnt/*
cp -R board-firmware/* /mnt
sync

Depending on your workstation OS and configuration, you may need to run as root, or use sudo(8), to execute these commands. Some shells may require confirmation before they will complete the rm command.

When done, you must unmount the mounted filesystem to ensure that all blocks in your workstation's filesystem buffer cache have been written back, and to avoid concurrent accesses leading to possible filesystem corruption. This must be done before issuing the MCC reboot command.

Rebooting the MCC

Complete the upgrade by rebooting the Morello board via the MCC:

Cmd> reboot
Rebooting...
Disabling debug USB..

Recovering the SD Card format

If you find that your SD Card no longer has a correct partition table (for example, if like the author you have written an installer image over it by accident) you will need to restore it to the original state with a FAT formated volume named M1SDP. The mechanism for creating such a filesystem will vary by OS. After you have created the FAT filesystem, follow the instruction above to copy the firmware into place and install it.

Linux

On Linux the following commands will create and format the FAT32 partition:

parted /dev/sdX mktable msdos
parted /dev/sdX mkpart primary fat32 0% 100%
mkfs.msdos /dev/sdX1

In this command /dev/sdX should be replaced by the path to the SD Card device.

MacOS

On MacOS the following command will create and format the M1SDP partition:

diskutil partitionDisk /dev/disk# 1 MBRFormat MS-DOS M1SDP 100%

In this command /dev/disk# should be replaced by the path to the SD Card device.

Installing on a Morello Board

Please ensure that you have upgraded the Morello board firmware before proceeding to CheriBSD installation.

Once you have a disk image, you will either need to write it to a USB stick to boot an Arm Morello system, or specify it as an argument to an emulator such as QEMU-CHERI or the Arm Morello FVP.

Writing an installer disk image to a USB stick

With appropriate substitutions of image filename and target device, the following command would write the image to a USB stick for use with a Morello board:

dd if=cheribsd-memstick-arm64-aarch64c-24.05.img of=/dev/DISK bs=1M

It is also possible to write a live image to a USB stick, with appropriate filename substitution.

Installing an Arm Morello system

It is possible to use Arm Morello systems both by booting the live image on a USB stick, and also by installing to the internal hard disk. In this section, we describe the how to perform an installation using the memstick image.

Starting the boot

Before powering on your Morello board, ensure that the installer USB stick is inserted into one of the USB ports on the board. Use the power switch to power the board on, which should lead to firmware output on the MCC console during a multi-minute initialization and boot cycle.

Once that has completed, you will see Morello firmware output on the Morello console. When this message is presented, press Escape to enter the UEFI boot menu:

Press ESCAPE for boot options .......

At the UEFI menu, select Boot Manager using the cursor keys, and press Enter:

   Select Language            <Standard English>         This selection will
                                                         take you to the
 > Device Manager                                        Device Manager
 > Boot Manager
 > Boot Maintenance Manager

   Continue
   Reset







  ^v=Move Highlight       <Enter>=Select Entry

Select your USB stick from the Boot Manager menu using the cursor keys, and press enter. In the example below, it is the SanDisk Ultra USB 3.0 device:

/------------------------------------------------------------------------------\
|                                Boot Manager                                  |
\------------------------------------------------------------------------------/

                                                         Device Path :
   Boot Manager Menu                                     PcieRoot(0x0)/Pci(0x0,
                                                         0x0)/Pci(0x0,0x0)/Pci(
   UEFI Shell                                            0x14,0x0)/Pci(0x0,0x0)
   UEFI Misc Device                                      /USB(0x5,0x0)
   UEFI WDC WDS240G2G0A-00JH30 2143B2455605
   UEFI PXEv4 (MAC:0002F7009856)
   UEFI HTTPv4 (MAC:0002F7009856)
   UEFI HTTPv6 (MAC:0002F7009856)
   UEFI PXEv6 (MAC:0002F7009856)
   UEFI SanDisk Ultra USB 3.0
   0101cfbb32adfd6c145cd38e7d6164cd167456963c3a71b9f157
   e574567b43d96c6000000000000000000000e49dde42009f1a00
   955581079e275d95
   UEFI Non-Block Boot Device
                                                       v
/------------------------------------------------------------------------------\
|                                                                              |
| ^v=Move Highlight       <Enter>=Select Entry      Esc=Exit                   |
\------------------------------------------------------------------------------/

This will be followed by the CheriBSD boot loader and kernel output. You can press Enter to proceed, or wait for the countdown timer to finish:

    _____ _               _ ____   _____ _____
   / ____| |             (_)  _ \ / ____|  __ \
  | |    | |__   ___ _ __ _| |_) | (___ | |  | |
  | |    | '_ \ / _ \ '__| |  _ < \___ \| |  | |
  | |____| | | |  __/ |  | | |_) |____) | |__| |
   \_____|_| |_|\___|_|  |_|____/|_____/|_____/
                                                 ```                        `
                                                s` `.....---.......--.```   -/
 /---------- Welcome to CheriBSD ----------\    +o   .--`         /y:`      +.
 |                                         |     yo`:.            :o      `+-
 |  1. Boot Installer [Enter]              |      y/               -/`   -o/
 |  2. Boot Single user                    |     .-                  ::/sy+:.
 |  3. Escape to loader prompt             |     /                     `--  /
 |  4. Reboot                              |    `:                          :`
 |  5. Cons: Serial                        |    `:                          :`
 |                                         |     /                          /
 |  Options:                               |     .-                        -.
 |  6. Kernel: default/kernel (1 of 1)     |      --                      -.
 |  7. Boot Options                        |       `:`                  `:`
 |                                         |         .--             `--.
 |                                         |            .---.....----.
 \-----------------------------------------/
   Autoboot in 10 seconds. [Space] to pause

Note that seperate instances of the installer will run on both the video and serial consoles. Care should be taken to ensure that only one is used as no mechanism is implemented to prevent one from destroying the other's work.

On the serial console, you will eventually be prompted to select a terminal emulation type:

Welcome to FreeBSD!

Please choose the appropriate terminal type for your system.
Common console types are:
   ansi     Standard ANSI terminal
   vt100    VT100 or compatible terminal
   xterm    xterm terminal emulator (or compatible)

Console type [xterm]:

In most cases, simply hitting Enter will be the right thing to do. This will enter bsdinstall, the interactive FreeBSD installer.

bsdinstall

bsdinstall uses the libdialog text interface library. Be aware that you will sometimes need to use the space bar, and not enter key, to select menu options, which many users find confusing.

Select Install at the first menu by hitting Enter:

                   ┌────────────────┤Welcome├────────────────┐
                   │ Welcome to FreeBSD! Would you like to   │
                   │ begin an installation or use the live   │
                   │ system?                                 │
                   ├─────────────────────────────────────────┤
                   │[  Install  ] [   Shell   ] [Live System]│
                   └─────────────────────────────────────────┘

Keymap selection

If you intend only to use the serial console, and not video console, select the default keymap by hitting Enter at the next menu; otherwise, use the cursor keys and space bar to select a country-specific keyboard layout:

       ┌───────────────────────┤Keymap Selection├────────────────────────┐
       │ The system console driver for FreeBSD defaults to standard "US" │
       │ keyboard map. Other keymaps can be chosen below.                │
       │ ┌─────────────────────────────────────────────────────────────┐ │
       │ │>>> Continue with default keymap                             │ │
       │ │->- Test default keymap                                      │ │
       │ │( ) Armenian phonetic layout                                 │ │
       │ │( ) Belarusian                                               │ │
       │ │( ) Belgian                                                  │ │
       │ │( ) Belgian (accent keys)                                    │ │
       │ │( ) Brazilian (accent keys)                                  │ │
       │ │( ) Brazilian (without accent keys)                          │ │
       │ │( ) Bulgarian (BDS)                                          │ │
       │ │( ) Bulgarian (Phonetic)                                     │ │
       │ │( ) Canadian Bilingual                                       │ │
       │ └─↓↓↓──────────────────────────────────────────────────── 41%─┘ │
       ├─────────────────────────────────────────────────────────────────┤
       │                      [Select]     [Cancel]                      │
       └────────────────── Press arrows, TAB or ENTER ───────────────────┘

Hostname

Press Enter to accept the default hostname, or replace it:

            ┌───────────────────┤Set Hostname├────────────────────┐
            │ Please choose a hostname for this machine.          │
            │                                                     │
            │ If you are running on a managed network, please ask │
            │ your network administrator for an appropriate name. │
            │ ┌─────────────────────────────────────────────────┐ │
            │ │cheribsd                                         │ │
            │ └─────────────────────────────────────────────────┘ │
            ├─────────────────────────────────────────────────────┤
            │                      [  OK  ]                       │
            └─────────────────────────────────────────────────────┘

Partitioning (automatic or manual)

This tutorial assumes that you are performing a fresh install, or a complete reinstall of your Morello system, with the intention of booting only CheriBSD. It also assumes that you will be using the UFS file system. If these assumpsions are not true, read the FreeBSD documentation for more information on partitioning disks before proceeding.

Press Enter to select an automated UFS install:

              ┌─────────────────┤Partitioning├──────────────────┐
              │ How would you like to partition your disk?      │
              │ ┌─────────────────────────────────────────────┐ │
              │ │Auto (UFS) Guided UFS Disk Setup             │ │
              │ │Auto (ZFS) EXPERIMENTAL Guided Root-on-ZFS   │ │
              │ │Manual     Manual Disk Setup (experts)       │ │
              │ │Shell      Open a shell and partition by hand│ │
              │ └─────────────────────────────────────────────┘ │
              ├─────────────────────────────────────────────────┤
              │              [  OK  ]     [Cancel]              │
              └─────────────────────────────────────────────────┘

Partition (entire disk)

Select Entire Disk and press Enter:

                 ┌────────────────┤Partition├────────────────┐
                 │ Would you like to use this entire disk    │
                 │ (ada0) for FreeBSD or partition it to     │
                 │ share it with other operating systems?    │
                 │ Using the entire disk will erase any data │
                 │ currently stored there.                   │
                 ├───────────────────────────────────────────┤
                 │      [Entire Disk]     [ Partition ]      │
                 └───────────────────────────────────────────┘

Partition (entire disk - confirmation)

Press Enter to confirm:

                 ┌────────────────┤Partition├────────────────┐
                 │ Would you like to use this entire disk    │
                 │ (ada┌────────┤Confirmation├─────────┐     │
                 │ shar│ This will erase the disk. Are │     │
                 │ Usin│ you sure you want to proceed? │  ta │
                 │ curr├───────────────────────────────┤     │
                 ├─────│     [ Yes  ]     [  No  ]     │  ───┤
                 │     └───────────────────────────────┘     │
                 └───────                                 ───┘

Partition Editor

Review the proposed partition scheme, select Finish, and then press Enter to proceed:

            ┌─────────────────┤Partition Editor├──────────────────┐
            │ Please review the disk setup. When complete, press  │
            │ the Finish button.                                  │
            │                                                     │
            │                                                     │
            │ ┌─────────────────────────────────────────────────┐ │
            │ │ada0       224 GB    GPT                         │ │
            │ │  ada0p1   260 MB    efi             /boot/efi   │ │
            │ │  ada0p2   220 GB    freebsd-ufs     /           │ │
            │ │  ada0p3   3.6 GB    freebsd-swap    none        │ │
            │ │da0        29 GB     GPT                         │ │
            │ │  da0p1    33 MB     efi                         │ │
            │ │  da0p2    3.0 GB    freebsd-ufs                 │ │
            │ │                                                 │ │
            │ │                                                 │ │
            │ │                                                 │ │
            │ └─────────────────────────────────────────────────┘ │
            ├─────────────────────────────────────────────────────┤
            │[Create] [Delete] [Modify] [Revert] [ Auto ] [Finish]│
            └─────────────────────────────────────────────────────┘

Partition editor - confirmation

Select Commit and press Enter to continue with writing out a new partition table:

            ┌─────────────────┤Partition Editor├──────────────────┐
            │ Please review the disk setup. When complete, press  │
            │ the Finish button.                                  │
            │                                                     │
            │                                                     │
            │ ┌─────────────────────────────────────────────────┐ │
            │ │┌────────────────┤Confirmation├─────────────────┐│ │
            │ ││ Your changes will now be written to disk. If  │  │
            │ ││ you have chosen to overwrite existing data,   │  │
            │ ││ it will be PERMANENTLY ERASED. Are you sure   │  │
            │ ││ you want to commit your changes?              │  │
            │ │├───────────────────────────────────────────────┤  │
            │ ││[   Commit    ] [Revert & Exit] [    Back     ]│  │
            │ │└───────────────────────────────────────────────┘  │
            │ │                                                   │
            │ │                                                 │ │
            │ └─────────────────────────────────────────────────┘ │
            ├─────────────────────────────────────────────────────┤
            │[Create] [Delete] [Modify] [Revert] [ Auto ] [Finish]│
            └─────────────────────────────────────────────────────┘

The installation proceeds

The installer will now proceed to check distribution files and unpack them onto the disk. Unless something goes wrong, no user interaction is required. Typical output from the installation process will look like this:

┌───────────────────────────┤Checksum Verification├──────────────────────────┐
│ base.txz                                                   [ In Progress ] │
│ kernel.txz                                                 [   Pending   ] │
│ base-dbg.txz                                               [   Pending   ] │
│ kernel-dbg.txz                                             [   Pending   ] │
│ kernel.GENERIC-MORELLO-NOCAPREVOKE-NODEBUG-dbg.txz         [   Pending   ] │
│ kernel.GENERIC-MORELLO-NOCAPREVOKE-NODEBUG.txz             [   Pending   ] │
│ kernel.GENERIC-MORELLO-NOCAPREVOKE-dbg.txz                 [   Pending   ] │
│ kernel.GENERIC-MORELLO-NOCAPREVOKE.txz                     [   Pending   ] │
│ kernel.GENERIC-MORELLO-NODEBUG-dbg.txz                     [   Pending   ] │
│ kernel.GENERIC-MORELLO-NODEBUG.txz                         [   Pending   ] │
│ kernel.GENERIC-MORELLO-PURECAP-BENCHMARK-NOCAPREVOKE-NO... [   Pending   ] │
│ kernel.GENERIC-MORELLO-PURECAP-BENCHMARK-NOCAPREVOKE-NO... [   Pending   ] │
│ kernel.GENERIC-MORELLO-PURECAP-BENCHMARK-NOCAPREVOKE-db... [   Pending   ] │
│ ...                                                                        │
│                                                                            │
│ Verifying checksums of selected distributions.                             │
│                                                                            │
│  ┌─Overall Progress─────────────────────────────────────────────────────┐  │
│  │                                   0%                                 │  │
│  └──────────────────────────────────────────────────────────────────────┘  │
└────────────────────────────────────────────────────────────────────────────┘

Updating the EFI configuration table

If you are reinstalling a Morello box with an existing OS install, you may be prompted to update the boot configuration. Press Enter to proceed:

                    ┌────────┤Boot Configuration├────────┐
                    │ There are multiple "FreeBSD" EFI   │
                    │ boot entries. Would you like to    │
                    │ remove them all and add a new one? │
                    ├────────────────────────────────────┤
                    │        [ Yes  ]     [  No  ]       │
                    └────────────────────────────────────┘

Setting a root password

Enter a new root password and press Enter. Then confirm it by typing the same password again and pressing Enter.

Please select a password for the system management account (root):
Typed characters will not be visible.
Changing local password for root
New Password:

Network configuration

If desired, configure Ethernet networking by pressing Enter.

      ┌─────────────────────┤Network Configuration├─────────────────────┐
      │ Please select a network interface to configure:                 │
      │ ┌─────────────────────────────────────────────────────────────┐ │
      │ │re0 RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet│ │
      │ └─────────────────────────────────────────────────────────────┘ │
      ├─────────────────────────────────────────────────────────────────┤
      │                      [  OK  ]     [Cancel]                      │
      └─────────────────────────────────────────────────────────────────┘

Network configuration - enabling IPv4

If desired, enable IPv4 on the Ethernet interface by selecting Yes and pressing Enter:

                        ┌───┤Network Configuration├───┐
                        │ Would you like to configure │
                        │ IPv4 for this interface?    │
                        ├─────────────────────────────┤
                        │    [ Yes  ]    [  No  ]     │
                        └─────────────────────────────┘

Network configuration - DHCP for IPv4

If you will be using DHCP, select Yes and press Enter. Otherwise select No and press Enter to perform a manual IPv4 configuration.

                       ┌────┤Network Configuration├───┐
                       │ Would you like to use DHCP   │
                       │ to configure this interface? │
                       ├──────────────────────────────┤
                       │     [ Yes  ]    [  No  ]     │
                       └──────────────────────────────┘

Network configuration - enabling IPv6

If desired, enable IPv6 on the Ethernet interface by selecting Yes and pressing Enter:

                        ┌───┤Network Configuration├───┐
                        │ Would you like to configure │
                        │ IPv6 for this interface?    │
                        ├─────────────────────────────┤
                        │    [ Yes  ]    [  No  ]     │
                        └─────────────────────────────┘

Network configuration - SLAAC for IPv6

Press Enter to use stateless address autoconfiguration for IPv6:

                      ┌─────┤Network Configuration├─────┐
                      │ Would you like to try stateless │
                      │ address autoconfiguration       │
                      │ (SLAAC)?                        │
                      ├─────────────────────────────────┤
                      │      [ Yes  ]     [  No  ]      │
                      └─────────────────────────────────┘

Network configuration - resolver configuration

Enter DNS resolution configuration information, or press Enter to confirm autoconfigured DNS configuration:

    ┌───────────────────────┤Network Configuration├───────────────────────┐
    │ Resolver Configuration                                              │
    │ ┌─────────────────────────────────────────────────────────────────┐ │
    │ │Search         localdomain                                       │ │
    │ │IPv6 DNS #1                                                      │ │
    │ │IPv6 DNS #2                                                      │ │
    │ │IPv4 DNS #1    192.168.1.1                                       │ │
    │ │IPv4 DNS #2                                                      │ │
    │ └─────────────────────────────────────────────────────────────────┘ │
    ├─────────────────────────────────────────────────────────────────────┤
    │                        [  OK  ]     [Cancel]                        │
    └─────────────────────────────────────────────────────────────────────┘

Local or UTC clock

Press Enter to select a system clock on UTC:

   ┌───────────┤Select local or UTC (Greenwich Mean Time) clock├───────────┐
   │ Is this machine's CMOS clock set to UTC?  If it is set to local time, │
   │ or you don't know, please choose NO here!                             │
   │                                                                       │
   ├───────────────────────────────────────────────────────────────────────┤
   │                          [ Yes  ]   [  No  ]                          │
   └───────────────────────────────────────────────────────────────────────┘

Timezone selection

Press Enter to select UTC as your timezone; otherwise, select your continent and press Enter.

                      ┌──────┤Time Zone Selector├───────┐
                      │ Select a region                 │
                      │ ┌─────────────────────────────┐ │
                      │ │0  UTC                       │ │
                      │ │1  Africa                    │ │
                      │ │2  America -- North and South│ │
                      │ │3  Antarctica                │ │
                      │ │4  Arctic Ocean              │ │
                      │ │5  Asia                      │ │
                      │ │6  Atlantic Ocean            │ │
                      │ │7  Australia                 │ │
                      │ │8  Europe                    │ │
                      │ │9  Indian Ocean              │ │
                      │ │10 Pacific Ocean             │ │
                      │ └─────────────────────────────┘ │
                      ├─────────────────────────────────┤
                      │      [  OK  ]     [Cancel]      │
                      └─────────────────────────────────┘

If you have not selected UTC, select your country and press Enter:

        ┌────────────────────┤Countries in Europe├────────────────────┐
        │ Select a country or region                                  │
        │ ┌─^^^─────────────────────────────────────────────────────┐ │
        │ │ 35 Poland                                               │ │
        │ │ 36 Portugal                                             │ │
        │ │ 37 Romania                                              │ │
        │ │ 38 Russian Federation                                   │ │
        │ │ 39 San Marino                                           │ │
        │ │ 40 Serbia                                               │ │
        │ │ 41 Slovakia                                             │ │
        │ │ 42 Slovenia                                             │ │
        │ │ 43 Spain                                                │ │
        │ │ 44 Svalbard and Jan Mayen                               │ │
        │ │ 45 Sweden                                               │ │
        │ │ 46 Switzerland                                          │ │
        │ │ 47 Turkey                                               │ │
        │ │ 48 Ukraine                                              │ │
        │ │ 49 United Kingdom of Great Britain and Northern Ireland │ │
        │ │ 50 Åland Islands                                        │ │
        │ └────────────────────────────────────────────────100%─────┘ │
        ├─────────────────────────────────────────────────────────────┤
        │                     [  OK  ]   [Cancel]                     │
        └─────────────────────────────────────────────────────────────┘

Confirm your choice by selecting Yes and pressing Enter:

   ┌────────────────────────────┤Confirmation├────────────────────────────┐
   │ Does the abbreviation `UTC' look reasonable?                         │
   ├──────────────────────────────────────────────────────────────────────┤
   │                         [ Yes  ]     [  No  ]                        │
   └──────────────────────────────────────────────────────────────────────┘

Setting the date and time

As desired, set the date, or select Skip and press Enter if you plan to use network time synchronization:

                   ┌─────────────┤Time & Date├────────────┐
                   │  Month            Year               │
                   │  ┌───────────────┐┌───────────────┐  │
                   │  │           July││           2024│  │
                   │  └───────────────┘└───────────────┘  │
                   │  ┌────────────────────────────────┐  │
                   │  │    Sun Mon Tue Wed Thu Fri Sat │  │
                   │  │         1   2   3   4   5   6  │  │
                   │  │     7   8   9  10  11  12  13  │  │
                   │  │    14  15  16  17  18  19  20  │  │
                   │  │    21  22  23  24  25  26  27  │  │
                   │  │    28  29  30  31              │  │
                   │  │                                │  │
                   │  └────────────────────────────────┘  │
                   ├──────────────────────────────────────┤
                   │       [Set Date]     [  Skip  ]      │
                   └──────────────────────────────────────┘

And, likewise, the time:

                   ┌─────────────┤Time & Date├────────────┐
                   │            ┌──┐ ┌──┐ ┌──┐            │
                   │            │11│:│23│:│20│            │
                   │            └──┘ └──┘ └──┘            │
                   ├──────────────────────────────────────┤
                   │       [Set Time]     [  Skip  ]      │
                   └──────────────────────────────────────┘

System configuration

As desired, enable any further services (e.g., ntpd and ntpd_sync_on_start) by selecting them and hitting Space:

┌───────────────────────────┤System Configuration├───────────────────────────┐
│ Choose the services you would like to be started at boot:                  │
│ ┌────────────────────────────────────────────────────────────────────────┐ │
│ │[ ] local_unbound      Local caching validating resolver                │ │
│ │[X] sshd               Secure shell daemon                              │ │
│ │[ ] moused             PS/2 mouse pointer on console                    │ │
│ │[ ] ntpd               Synchronize system and network time              │ │
│ │[ ] ntpd_sync_on_start Sync time on ntpd startup, even if offset is high│ │
│ │[ ] powerd             Adjust CPU frequency dynamically if supported    │ │
│ │[X] dumpdev            Enable kernel crash dumps to /var/crash          │ │
│ │[ ] debugger_on_panic  Run debugger on kernel panic                     │ │
│ └────────────────────────────────────────────────────────────────────────┘ │
├────────────────────────────────────────────────────────────────────────────┤
│                                  [  OK  ]                                  │
└────────────────────────────────────────────────────────────────────────────┘

Then press Enter to continue.

Add user accounts

If you plan to install the desktop environment in the next step, it is useful to create user accounts now. If desired, select Yes and press Enter to add non-root accounts. Otherwise, select No and press Enter

                       ┌──────┤Add User Accounts├─────┐
                       │ Would you like to add users  │
                       │ to the installed system now? │
                       ├──────────────────────────────┤
                       │     [ Yes  ]    [  No  ]     │
                       └──────────────────────────────┘

If you add non-root users, we recommend adding them to the wheel, operator, and video groups to allow them to manage the system as well as use the desktop environment.

CHERI Desktop environment

If you wish to configure your CheriBSD system with a KDE-based desktop environment you can do so now. NOTE: You must have previously configured the network and be able to reach pkg.CheriBSD.org.

                      ┌─────────┤CHERI Desktop├────────┐
                      │ Would you like to install the  │
                      │ CHERI desktop environment      │
                      │ (requires network)?            │
                      ├────────────────────────────────┤
                      │      [ Yes  ]     [  No  ]     │
                      └────────────────────────────────┘

Select Yes to install (NOTE: this may take 20-50 minutes depending on network conditions).

Virtualization support

If you wish to configure your CheriBSD system to support bhyve-based, CHERI-enabled virtual machines, press Enter:

                      ┌───────┤CHERI VM Support├────────┐
                      │ Would you like to install CHERI │
                      │ virtual machine support         │
                      │ (requires network)?             │
                      ├─────────────────────────────────┤
                      │      [ Yes  ]     [  No  ]      │
                      └─────────────────────────────────┘

Final configuration

If there are further configuration settings to change, use the up/down arrow keys to highlight an appropriate option from the menu, then use the left arrow key to select the Select button and press Enter. Otherwise, press Enter to complete the installation:

┌────────────────────────────┤Final Configuration├───────────────────────────┐
│ Setup of your FreeBSD system is nearly complete. You can now modify your   │
│ configuration choices. After this screen, you will have an opportunity to  │
│ make more complex changes using a shell.                                   │
│ ┌────────────────────────────────────────────────────────────────────────┐ │
│ │Add User         Add a user to the system                               │ │
│ │Root Password    Change root password                                   │ │
│ │Hostname         Set system hostname                                    │ │
│ │Network          Networking configuration                               │ │
│ │Services         Set daemons to run on startup                          │ │
│ │System Hardening Set security options                                   │ │
│ │Time Zone        Set system timezone                                    │ │
│ │Handbook         Install FreeBSD Handbook (requires network)            │ │
│ │CHERI Desktop    Install the CHERI desktop environment (requires network│ │
│ │CHERI VM Support Install CHERI virtual machine support (requires network│ │
│ └────────────────────────────────────────────────────────────────────────┘ │
├────────────────────────────────────────────────────────────────────────────┤
│                            [Select]     [Finish]                           │
└────────────────────────────────────────────────────────────────────────────┘

Manual configuration

If desired, select Yes and press Enter to drop to a shell before reboot. Otherwise, select No and press Enter.

                 ┌──────────┤Manual Configuration├──────────┐
                 │ The installation is now finished. Before │
                 │ exiting the installer, would you like to │
                 │ open a shell in the new system to make   │
                 │ any final manual modifications?          │
                 ├──────────────────────────────────────────┤
                 │           [ Yes  ]     [  No  ]          │
                 └──────────────────────────────────────────┘

Complete

Select Reboot and press Enter:

                  ┌───────────────┤Complete├────────────────┐
                  │ Installation of FreeBSD complete! Would │
                  │ you like to reboot into the installed   │
                  │ system now?                             │
                  ├─────────────────────────────────────────┤
                  │[  Reboot   ] [ Shutdown  ] [Live System]│
                  └─────────────────────────────────────────┘

Rebooting after installation

Remove the USB stick to prevent the installer from running after a system reboot. If you have issues with reliabity rebooting your Morello box, confirm that you are using a recent Morello firmware revision due to a known firmware bug in earlier versions that made OS-triggered reboot unreliable.

Logging in

Log in using the username root and the password you entered during the installation process:

...
Performing sanity check on sshd configuration.
Starting sshd.
Starting cron.
Starting background file system checks in 60 seconds.
Starting sddm.

Sat Jul 20 00:12:09 UTC
CheriBSD/arm64 (cheribsd) (ttyu0)

login: root
Password:
Jul 20 00:12:13 cheribsd login[968]: ROOT LOGIN (root) ON ttyu0
FreeBSD 15.0-CURRENT (GENERIC-MORELLO-PURECAP) #0 releng/24.05-b2ad856aac65: Fri Jul 19 21:17:24 UTC 2024

Welcome to CheriBSD!

CheriBSD extends FreeBSD to implement memory protection and software
compartmentalization features enabled by CHERI-extended CPUs.

The CheriBSD front page:
  https://www.cheribsd.org/

We provide support via a mailing list:
  https://www.cl.cam.ac.uk/research/security/ctsrd/cheri/cheri-lists.html

And also via the CHERI-CPU Slack:
  https://www.cl.cam.ac.uk/research/security/ctsrd/cheri/cheri-slack.html

CheriBSD source may be found at:
  https://github.com/CTSRD-CHERI/cheribsd/

Find out more about about CHERI at https://cheri-cpu.org/

WARNING: INVARIANTS kernel option defined, expect reduced performance
WARNING: WITNESS kernel option defined, expect reduced performance
WARNING: capability revocation enabled by default, this may affect performance
root@cheribsd:~ #

You can now add further users, install SSH keys for remote access, install packages, and so on.

Morello known issues

The following known issues may affect the reliability and usability of CheriBSD on Morello:

  • If a USB hub is plugged in (including embedded hubs in USB keyboards) the system will pause on boot due to a firmware bug. Some people find they can bypass this pause by pressing Enter on a USB keyboard attached to the Morello desktop system. Others find they must unplug the hub to boot.

  • Sometimes the ATA controller starts timing out. and displaying messages like this on the console:

    ahcich0: Timeout on slot 12 port 0
    ahcich0: is 00000000 cs 00003000 ss 00000000 rs 00003000 tfd 50 serr 00000000 cmd 00718c17
    (ada0:ahcich0:0:0:0): WRITE_DMA. ACB: ca 00 50 8a 8b 41 00 00 00 00 d0 00
    (ada0:ahcich0:0:0:0): CAM status: Command timeout
    (ada0:ahcich0:0:0:0): Retrying command, 3 more tries remain
    

    A reboot usually cures the problem. If this happens during install, it's generally best to start over as performance is awful once timeouts start.

  • The HDMI transmitter on the board might fail to detect if a monitor is plugged in. In such a case, the monitor will turn blank when the CheriBSD kernel starts printing messages to the console, after the CheriBSD boot loader menu. A user can configure the kernel to ignore hot plug detection and assume that the attached monitor can receive HDMI input. To do that, add the following line to /boot/loader.conf and reboot your board:

    hw.tda19988.broken_hpd="1"
    

    While the screen itself should be working with this workaround, the user can still experience EDID reading failures described below.

  • The HDMI transmitter on the board might fail to detect the monitor's resolution. The CheriBSD kernel writes the following message to the console in such a case:

    tda0: Failed to read EDID
    

    Currently, there is no workaround for this issue and it is likely to occur with the HDMI transmitter issue described above.

Issues with older firmware

The following issues affected users of older firmware revisions:

  • Due to a firmware bug, OS-triggered reboot may lead the system to hang when using older firmware. Update your firmware or use the "reboot" command on the management console to reboot the system once the OS has shut down.

  • Due to a firmware bug, some earlier firmware versions are not able to persist UEFI configuration variables. This affects, for example, the installation environment as the OS installer cannot instruct the firmware to boot only from the installed target, and not further USB sticks.

It is extremely important that you use the most recent firmware with your Morello board, to avoid these and other issues present in earlier firmware releases.

Third-party packages

CheriBSD on Morello ships with CheriABI, hybrid ABI and benchmark ABI packages (compilations) of the CheriBSD ports collection, each targeting a different form of code generation and Application Binary Interface (ABI). They have different levels of completeness, maturity, security, and support.

CheriBSD ports extends FreeBSD ports, a collection of over 34,000 third-party software adaptations to FreeBSD, with CHERI- and CheriBSD-specific patches. The releng/24.05 CheriBSD ports branch contains ports matching the packages built for the current release.

Overview

The following table presents an overview of available package managers in CheriBSD that are described in more details in consecutive sections. You can also browse package repositories at pkg.CheriBSD.org to check what packages are available for a specific ABI version.

ABI#ManagerInstall pathSuitable for
CheriABI~10,000pkg64c/usr/localSecurity evaluation
Benchmark ABI~10,000pkg64cb/usr/local64cbPerformance evaluation
Hybrid ABI~25,000pkg64/usr/local64Day-to-day use

Package managers in CheriBSD

Note: As of this writing we only provide packages for Morello systems.

CheriBSD includes three package managers:

  • pkg64c for CheriABI packages;
  • pkg64cb for benchmark ABI packages;
  • pkg64 for hybrid ABI packages.

The FreeBSD package manager pkg is not available on CheriBSD. We expect that pkg64c will be renamed to pkg in a future CheriBSD release. The intention is that, over time, the CheriABI packages will become more mature, and hence the preferred collection for day-to-day use.

The syntax of the pkg64* commands match the syntax of the pkg command from FreeBSD. You can find information on package manager commands in FreeBSD's pkg manual pages shipped with CheriBSD, e.g. for the commands pkg64 rquery and pkg64c rquery, execute man pkg-rquery.

Additionally, you can learn more about FreeBSD's package manager by reading the pkg(8) manual page online and the FreeBSD Handbook chapter on packages and ports.

CheriABI packages

CheriABI packages are compiled using pure-capability CHERI C/C++, and employ fine-grained C/C++ memory protection.

These packages are considered appropriate for experimental use and a security evaluation of CHERI-extended hardware-software stacks. Their primary function is to provide necessary dependencies for efforts to port software to CheriABI and to support CHERI demonstration and evaluation. They are suitable for research and development of software that benefit from spatial and temporal memory safety as well as software compartmentalisation. They can also be used to investigate potential memory safety issues in third-party software that are easier to detect and debug using a CHERI-enabled hardware-software stack.

There are currently ~10,000 CheriABI packages available, including:

  • Databases: sqlite3;

  • Development utilities: autoconf, chericat, cmake, git, gmake;

  • Editors: nano, vim-tiny;

  • GUI environment: cheri-desktop (seatd, sddm, wayland, plasma5-plasma-workspace, dolphin, kate, konsole, okular);

  • Hypervisor tools: cheri-vm-support (u-boot-bhyve-arm64);

  • Language interpreters: perl5, tcl86;

  • Libraries: libnghttp2, libressl, libssh2, lzlib;

  • Networking tools: rsync31;

  • Shells: bash, dash;

  • System tools: sudo, tmux.

The packages are installed in the standard /usr/local hierarchy as they match the base system ABI.

Benchmark ABI packages

Benchmark ABI packages are compiled similarly to CheriABI packages, but differ in generated code that takes into account current limitations of the Morello architecture. You should make sure to read the Benchmarking guidance section before using the benchmark ABI packages to understand the differences between CheriABI and the benchmark ABI.

These packages are considered appropriate for a performance evaluation of the Arm Morello architecture and third-party software executed on it. They should not be used for a security evaluation, in which case you should use CheriABI instead.

There are currently ~10,000 benchmark ABI packages available which should include almost all packages that are available in the CheriABI package repository.

The packages are installed in the /usr/local64cb hierarchy. By default, /usr/local64cb/bin and /usr/local64cb/sbin are included in your PATH environment variable of a shell shipped with CheriBSD. If you are planning to use a custom shell, remember to add these paths to PATH.

Different numbers of CheriABI and benchmark ABI packages is a result of using a custom local base path (/usr/local64cb) and a custom LD_LIBRARY_PATH variable name (LD_64CB_LIBRARY_PATH) for the benchmark ABI. Multiple ports do not correctly handle these differences.

Hybrid ABI packages

Hybrid ABI packages are compiled almost identically to packages in the baseline non-CHERI architecture (e.g., Armv8-a for Morello, and 64-bit RISC-V for RISC-V), and do not have improvements in memory protection or software compartmentalization.

These packages are considered appropriate for day-to-day use. They are intended to provide stable versions of tools necessary to develop software and use your CHERI system while more software is ported to CheriABI.

There are currently ~25,000 hybrid ABI packages available, including:

  • Browsers: chromium, firefox;

  • Compilers:

    • llvm-morello;

      Includes Clang (a CHERI C/C++ compiler), LLD (a linker), and the LLVM infrastructure for the Arm Morello architecture. Binaries installed with this package have the suffix -morello.

    • llvm;

      Adds links in /usr/local64/bin to allow the llvm-morello package to be used as the default LLVM package, without the suffix -morello, e.g. clang instead of clang-morello.

    • llvm-base;

      Adds links in /usr/bin to allow the llvm package to be used in place of a base-system toolchain. The cc script installed with this package adds compiler flags required to natively compile code for a CHERI-enabled architecture.

    • rust.

  • Debuggers: gdb;

  • Development utilities: meson, ninja;

  • Editors: emacs;

  • Language interpreters: lua54, python39, ruby;

  • Shells: fish, zsh.

The packages are installed in the /usr/local64 hierarchy. By default, /usr/local64/bin and /usr/local64/sbin are included in your PATH environment variable of a shell shipped with CheriBSD. If you are planning to use a custom shell, remember to add these paths to PATH.

Limitations of packages

When using or even considering third-party software delivered with CheriBSD packages, it is important to keep in mind the maturity of currently available CheriABI packages, and potential conflicts between hybrid ABI and CheriABI packages. The following sections describe some of such limitations.

Maturity of CheriABI packages

There are many projects that must still be ported to CHERI in order to provide a fully-functional CheriABI package repository. The available CheriABI packages are also not continuously tested as part of the CheriBSD package building infrastructure.

CheriABI packages compile, but many have CHERI-related warnings that have not been audited, and only a limited set have been tested. Whilst the CheriABI packages are more interesting as they use CHERI memory-safety features, they might break at run time and hence might not be suitable for critical operations. In such cases, a corresponding hybrid ABI package might be better suited than the CheriABI package.

We expect the number of available and tested packages to grow with future CheriBSD releases, but it will take quite some time until we are able to provide a CheriABI package repository that covers functionalities currently included in the hybrid ABI package repository.

We encourage everyone to report bugs related to CHERI support in the CheriBSD ports issue tracker.

Separate local bases for ABIs

Upstream FreeBSD does not have the ability to manage packages for multiple ABIs at the same time and its pkg package manager installs all packages in the default local base path (/usr/local). CheriBSD introduces a workaround for this issue by using separate local base paths for different package repositories.

The hybrid ABI packages are compiled with a local base set to /usr/local64 while the CheriABI packages use the default local base. Because the hybrid ABI packages are installed in a non-standard location, there may be bugs resulting in hybrid ABI packages looking for its dependencies in /usr/local instead of /usr/local64. For example, a CheriBSD port a package was built from might fail to configure a project's build system to use this custom local base, or an upstream project might hardcode /usr/local in its source code.

We encourage everyone to report bugs related to invalid local base paths in the hybrid ABI packages in our CheriBSD ports issue tracker.

Cross-ABI package conflicts

Hybrid ABI and CheriABI package repositories include many counterpart packages, e.g. git is available in both repositories. CheriABI packages have a higher priority than the hybrid ABI packages in default PATH environment variables in CheriBSD. If you install a CheriABI package and a hybrid ABI package with a conflicting program name, you must execute the hybrid ABI program using an absolute path by default.

When using a shell configuration not included in CheriBSD's source code, a user must remember to appropriately set the PATH environment variable to take into account both /usr/local and /usr/local64 paths. The user can set the order of the paths they prefer but it is recommended to place /usr/local before /usr/local64 in such configurations to match CheriBSD's defaults.

Cross-ABI package dependencies

At the moment, a package cannot depend at run time on another package with a different ABI, e.g. to execute a program provided by that package. Such a feature would be useful for CheriBSD ports which cannot easily be adapted to CheriABI and which include programs that could be executed by other ports already adapted to CheriABI. There are currently no plans to support this case.

Missing packages

Hybrid ABI and CheriABI package repositories might be missing third-party software due to the following reasons:

Upgrading packages

It is recommended to upgrade packages in the following situations:

  • Before building a newer CheriBSD release;

    The release might require an updated toolchain (e.g., due to ABI changes in the release that must be supported by the toolchain).

  • After updating your host to a newer CheriBSD release;

    Package repositories are compiled for the ABI version used by the corresponding release, which might differ from the ABI of the previous release.

    Where practical, we intend to support packages from the immediately prior release on the newer release, but will make no effort to ensure that package repositories from even older releases continue to work.

    Additionally, package repositories for the newer release will most likely include updated third-party software versions that are tested against that release.

  • After finding a bug in a package (e.g., a library, a toolchain).

    It might happen that the bug you discovered has already been reported and fixed. If the package upgrade does not resolve your issue, please report the bug.

Use the following commands to upgrade your packages:

CheriABIHybrid ABI
pkg64c upgradepkg64 upgrade

If you do not want perform the upgrade itself but only check if newer versions are available, use the -n flag:

CheriABIHybrid ABI
pkg64c upgrade -npkg64 upgrade -n

You can read more on the upgrade command at pkg-upgrade(8) or in your console with man pkg-upgrade.

Debugging packages

Similarly to the FreeBSD Project, CheriBSD packages are built with CheriBSD ports that are based on FreeBSD ports. By default, the packages are built with optimisation levels defined in CheriBSD ports or upstream third-party software projects. Unfortunately, the FreeBSD ports build system does not allow to build ports with optimisations and debug symbols at the same time. As a consequence of that, you might discover that debug symbols are missing when debugging a program with the GDB debugger.

In order to allow users to use optimised and debug CheriABI packages, CheriBSD provides two package repositories with them: the default repository with optimisations and without debug symbols called CheriBSD, and the debug repository without optimisations and with debug symbols called CheriBSD-debug. Note that we do not provide debug packages for the hybrid and benchmark ABIs.

You can enable the debug repository by first creating a directory for custom package repository configuration files:

mkdir -p /usr/local/etc/pkg/repos

and then creating a configuration file that enables CheriBSD-debug:

echo "CheriBSD-debug: { enabled: yes }" >/usr/local/etc/pkg/repos/CheriBSD-debug.conf

You can reinstall all previously installed default packages to replace them with debug packages:

pkg64c upgrade -fr CheriBSD-debug

Alternatively, you can selectively reinstall a package:

pkg64c upgrade -r CheriBSD-debug <pkg-name>

However, keep in mind that in such case only the package you list is reinstalled, leaving its dependencies intact.

Once you enable the CheriBSD-debug repository, the CheriBSD repository is used by default. You have to use the -r flag to switch to CheriBSD-debug in a pkg64c command, e.g.:

pkg64c install -r CheriBSD-debug <pkg-name>

If you want to use only the CheriBSD-debug repository, you can create a configuration file that disables the CheriBSD repository:

echo "CheriBSD: { enabled: no }" > /usr/local/etc/pkg/repos/CheriBSD.conf

You can also check if an installed package was fetched from the default or debug package repository:

pkg64c query '%n %R' <pkg-name>

You can omit <pkg-name> in the above command to list all packages with their repositories.

Useful commands

The following list includes useful commands to manage packages. Replace <pkg> with pkg64c to execute a command in the context of CheriABI packages and with pkg64 in the context of hybrid ABI packages.

Most pkg commands have separate man pages describing them. For a command <pkg> foo, you can read its man page by executing man pkg-foo on a CheriBSD host.

  • List available package manager commands:
    <pkg> help
    
  • Read more information on a package manager command:
    <pkg> help <command>
    
  • Search a package repository:
    <pkg> search <pattern>
    
  • Search a package repository and display full information:
    <pkg> search --full <pattern>
    
  • List installed packages:
    <pkg> info
    
  • Display information for an installed package:
    <pkg> info <pkg-name>
    
  • Install a package:
    <pkg> install <pkg-name>
    
  • Delete a package:
    <pkg> delete <pkg-name>
    
  • Delete packages that are no longer required:
    <pkg> autoremove
    
  • Check a new package version for an installed package:
    <pkg> upgrade -n <pkg-name>
    
  • Check new package versions for all installed packages:
    <pkg> upgrade -n
    
  • Upgrade a package after confirmation:
    <pkg> upgrade <pkg-name>
    
  • Upgrade all packages after confirmation:
    <pkg> upgrade
    
  • Display which package installed a file:
    <pkg> which /path/to/file
    
  • Check a number of installed packages:
    <pkg> stats -l
    
  • Check a number of available packages:
    <pkg> stats -r
    

Compiling "Hello World"

These instructions are intended for use on an Arm Morello board, and install hybrid ABI versions of key toolchain and utilities using pkg64. The commands will (eventually) work the same on CHERI-RISC-V, but output details and registers in GDB will differ.

Toolchain installation

You will need the Morello LLVM toolchain and Morello GDB for this exercise. To install LLVM, use the command:

pkg64 install llvm-base

If this is the first time you are using pkg64 on this system, you will be prompted to bootstrap the pkg package before the package database is downloaded and you are prompted to confirm before installing llvm and its dependencies:

root@cheribsd:~ # pkg64 install llvm-base
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+http://pkg.CheriBSD.org/CheriBSD:20220314:aarch64, please wait...
Verifying signature with trusted certificate pkg.cheribsd.org.2022032901... done
Installing pkg-1.17.5_1...
Extracting pkg-1.17.5_1: 100%
Updating CheriBSD repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
Fetching packagesite.pkg: 100%    4 MiB   1.4MB/s    00:03
Processing entries: 100%
CheriBSD repository update completed. 20954 packages processed.
All repositories are up to date.
Updating database digests format: 100%
The following 12 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        gettext-runtime: 0.21
        indexinfo: 0.3.1
        libedit: 3.1.20210910,1
        libffi: 3.3_1
        libxml2: 2.9.13_2
        llvm: 13,1
        llvm-base: 1
        llvm-morello: 13.0.d20220502_1
        mpdecimal: 2.5.1
        perl5: 5.32.1_1
        python38: 3.8.13
        readline: 8.1.2

Number of packages to be installed: 12

The process will require 902 MiB more space.
190 MiB to be downloaded.

Proceed with this action? [y/N]: y
[1/12] Fetching llvm-base-1.pkg: 100%    3 KiB   2.8kB/s    00:01
[2/12] Fetching llvm-13,1.pkg: 100%   11 KiB  11.2kB/s    00:01
...
[12/12] Fetching libedit-3.1.20210910,1.pkg: 100%  119 KiB 121.8kB/s    00:01
Checking integrity... done (0 conflicting)
[1/12] Installing indexinfo-0.3.1...
[1/12] Extracting indexinfo-0.3.1: 100%
...
[12/12] Installing llvm-base-1...
[12/12] Extracting llvm-base-1: 100%
=====
...

Note: By default FreeBSD ships with the vi and ee editors. You may wish to install the nano or vim package to access a more familiar editor. Currently only hybrid packages installable with pkg64 are available.

Source code

#include <stdio.h>

int
main(void)
{
	printf("Hello world\n");
}

Building for CheriABI

To build a CheriABI Hello World program use:

cc -g -O2 -Wall -o helloworld helloworld.c

You can verify this is a CheriABI binary with the file command:

root@cheribsd:~ # file helloworld
helloworld: ELF 64-bit LSB pie executable, ARM aarch64, C64, CheriABI, version 1 (SYSV), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 14.0 (1400094), FreeBSD-style, with debug_info, not stripped

Building for the Benchmark ABI

To target the Benchmark ABI, add the argument -mabi=purecap-benchmark to the cc command line:

cc -g -O2 -Wall -mabi=purecap-benchmark -o helloworld helloworld.c

You can verify this is a Benchmark ABI binary with the file command:

root@cheribsd:~ # file helloworld
helloworld: ELF 64-bit LSB pie executable, ARM aarch64, C64, CheriABI, version 1 (SYSV), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 14.0 (1400094), FreeBSD-style, pure-capability benchmark ABI, with debug_info, not stripped

Running

Run the program:

root@cheribsd:~ # ./helloworld
Hello world

Debugging

First, if it is not already installed, install the CHERI GDB debugger:

pkg64 install gdb-cheri

You can then debug helloworld by running GDB and setting a breakpoint on main:

root@cheribsd:~ # gdb ./helloworld
GNU gdb (GDB) 12.1
...
Reading symbols from ./helloworld...
...
(gdb) b main
Breakpoint 1 at 0x10a14: file helloworld.c, line 6.
(gdb) r
Starting program: /root/helloworld

Breakpoint 1, main () at helloworld.c:6
6           printf("Hello world\n");
(gdb) 

If you then step into the printf (or puts depending on optimization level) call you will see that GDB prints the capability argument with expanded information including bounds and permissions:

(gdb) s
puts (s=0x1006e0 [rR,0x1006e0-0x1006ec] "Hello world")
    at /home/bed22/cheri/cheribsd/lib/libc/stdio/puts.c:60
60      /home/bed22/cheri/cheribsd/lib/libc/stdio/puts.c: No such file or directory.

The argument can also be examined directly as either an integer or capability register:

(gdb) info reg x0
x0             0x1006e0            1050336
(gdb) info reg c0
c0             0x905f400046ec06e000000000001006e0 0x1006e0 [rR,0x1006e0-0x1006ec]
(gdb)

If a capability (either in memory or a register) contains a null-derived value, it is displayed as a simple scalar value without any attributes. If a capability is not null-derived but is untagged, the string "(invalid)" is displayed after the bounds.

Capability tags in memory can also be examined by passing the "m" flag to the eXamine command. For example, compare the output of the internal FILE structure used for stdout in libc:

(gdb) p *__stdoutp
$1 = {_p = 0x0, _r = 0, _w = 0, _flags = 8, _file = 1, _bf = {_base = 0x0, 
    _size = 0}, _lbfsize = 0, 
  _cookie = 0x403a8400 [rwRWE,0x403a8230-0x403a87a0], 
  _close = 0x402bade5 <__sclose> [rxRE,0x4018e000-0x407f0000] (sentry), 
  _read = 0x402bada5 <__sread> [rxRE,0x4018e000-0x407f0000] (sentry), 
  _seek = 0x402baddd <__sseek> [rxRE,0x4018e000-0x407f0000] (sentry), 
  _write = 0x402badc1 <__swrite> [rxRE,0x4018e000-0x407f0000] (sentry), _ub = {
    _base = 0x0, _size = 0}, _up = 0x0, _ur = 0, _ubuf = "\000\000", 
  _nbuf = "", _lb = {_base = 0x0, _size = 0}, _blksize = 0, _offset = 0, 
  _fl_mutex = 0x0, _fl_owner = 0x0, _fl_count = 0, _orientation = 0, 
  _mbstate = {__mbstate8 = '\000' <repeats 127 times>, _mbstateL = 0, 
    _mbstateP = 0x0 }, _flags2 = 0}
(gdb) x/16gxm __stdoutp
<CHERI Tag 0 for range [0x403a8400,0x403a8410)>
0x403a8400:     0x0000000000000000      0x0000000000000000
<CHERI Tag 0 for range [0x403a8410,0x403a8420)>
0x403a8410:     0x0000000000000000      0x0000000000010008
<CHERI Tag 0 for range [0x403a8420,0x403a8430)>
0x403a8420:     0x0000000000000000      0x0000000000000000
<CHERI Tag 0 for range [0x403a8430,0x403a8440)>
0x403a8430:     0x0000000000000000      0x0000000000000000
<CHERI Tag 0 for range [0x403a8440,0x403a8450)>
0x403a8440:     0x0000000000000000      0x0000000000000000
<CHERI Tag 1 for range [0x403a8450,0x403a8460)>
0x403a8450:     0x00000000403a8400      0xdc5fc00047a08230
<CHERI Tag 1 for range [0x403a8460,0x403a8470)>
0x403a8460:     0x00000000402bade5      0xb05fc000bf0618e7
<CHERI Tag 1 for range [0x403a8470,0x403a8480)>
0x403a8470:     0x00000000402bada5      0xb05fc000bf0618e7

Benchmarking guidance

CheriBSD is a research operating system designed to run on experimental hardware. In this section, we provide high-level guidance on configuring CheriBSD for software benchmarking.

Framing

The expectation of this section is that a performance comparison is being made between baseline aarch64-compiled software (64-bit Arm) and memory-safe aarch64c software (CheriABI) on the Arm Morello board.

Before proceeding to the remainder of this section, it is essential to first read Early performance results from the prototype Morello microarchitecture. That document provides detailed information on how to interpret performance measurements, including documenting known limitations of the prototype Morello microarchitecture.

A key conclusion from that work is that CheriABI software being used for performance measurement should be compiled for the aarch64cb "Benchmark ABI", and not aarch64c.

Kernel configuration

Depending on the version of CheriBSD you are using, the kernel may have a number of debugging fetures enabled. These features can substantially impact system performance, including inducing disproportionate performance overhead for specific system behaviours. For example, enabling kernel lock-order checking ("WITNESS") will introduce substantial overhead, and in particular will impact kernel-centric workloads that make more intensive use of locks, such as networking. During boot, CheriBSD will add the following lines to /etc/motd, which will be displayed at login, to warn you about the "INVARIANTS" and "WITNESS" debugging features:

WARNING: INVARIANTS kernel option defined, expect reduced performance
WARNING: WITNESS kernel option defined, expect reduced performance

Performance measurements employing a hybrid kernel should use the kernel.GENERIC-MORELLO-NODEBUG configuration when kernel memory safety is not required. The following line can be added to /boot/loader.conf:

kernel="kernel.GENERIC-MORELLO-NODEBUG"

Measurements requiring a memory-safe kernel should use the kernel.GENERIC-MORELLO-PURECAP-NODEBUG configuration. The following can be added to /boot/loader.conf:

kernel="kernel.GENERIC-MORELLO-PURECAP-NODEBUG"

You will need to reboot in order for this change to take effect.

Heap temporal memory safety

Since CheriBSD 23.11, userlevel heap temporal safety is enabled by default with jemalloc as the memory allocator. This support is experimental, and has not yet been through significant performance analysis and optimization. We recommend disabling temporal safety support for the full system during benchmarking not specifically intended to capture temporal safety performance. During boot, CheriBSD will add the following line to /etc/motd, which will be displayed at login, to warn you about the temporal safety feature:

WARNING: capability revocation enabled by default, this may affect performance

The following line can be added to /boot/loader.conf:

security.cheri.runtime_revocation_default=0

You will need to reboot in order for this change to take effect.

The Benchmark ABI

Since CheriBSD 23.11, CheriBSD supports the Benchmark ABI, a modified form of code generation improving performance on the Arm Morello board. This is required due to limitations on bounds prediction in the current Morello prototype, which would be resolved in a production microarchitecture. The performance of the Benchmark ABI is more predictive of potential future CHERI microarchitectural performance than the Morello prototype running software compiled for CheriABI, making it preferable for benchmarking. However, this comes at the cost of reduce security, and so software compiled for the Benchmark ABI should not be used for security evaluation. Programs may be compiled to the Benchmark ABI using the -mabi=purecap-benchmark command-line argument to cc.

More information on compiling with the Benchmark ABI can be found in Compiling Hello World. More information on what the Benchmark ABI is, and how to interpret performance results, can be found in Early performance results from the prototype Morello microarchitecture.

Getting help

CheriBSD is a community-supported open-source research operating system. While every effort has been made to engineer to a high quality, many aspects of OS design for CHERI processors remain active research, and there are constrained engineering resources available to build, maintain, and support the system. As you gain experience with CheriBSD, your help in supporting newcomers in the community will be greatly appreciated.

GitHub issue tracker and pull requests

CheriBSD is developed and maintained in the CheriBSD GitHub repository, which includes an issue tracker in which bugs can be reported, as well as permitting pull requests to be submitted.

This document is also maintained in a GitHub repository, and your feedback and improvements would be very much appreciated.

CheriBSD Slack on cheri-cpu.slack.com

Support for CheriBSD is provided via the CHERI CPU Slack, where a number of topic-specific channels can be found:

  • #cheribsd;

    CheriBSD, its features, releases and future plans.

  • #debuggers;

    GDB extended to support CHERI.

  • #qemu;

    QEMU-CHERI, QEMU system and user modes for Morello and CHERI-RISC-V.

  • #software-porting.

    Third-party software adaptations to CheriABI and CHERI-enabled achitectures, CheriBSD ports and Poudriere.

CheriBSD mailing lists

There are also several CheriBSD email lists available:

In-person events and meetings

You can meet with the CheriBSD developers at a number of conferences and other events, including:

  • UKRI Digital Security by Design (DSbD) All Hands meetings that take place in the UK twice annually to discuss the Morello architecture, board, and its use.

  • FreeBSD developer summits and conferences around the world, including (when it takes place) the BSDCan conference in Canada, and the EuroBSDCon conference in Europe.

There is discussion around creating further, more frequent CHERI workshops in the UK in the future.

Most CheriBSD developers are based at the University of Cambridge (UK) and SRI International (USA), and it is possible to set up meetings with them directly. Get in touch if this would be useful to you.

Resources

In addition to the CheriBSD support channels, the following resources may also be useful.

CheriBSD

CHERI

FreeBSD