Raspberry Pi 3
Sequitur Labs did the initial OP-TEE port which at the time also came with modifications in U-Boot, Trusted Firmware A and Linux kernel. Since that initial port more and more patches have found mainline trees and today the OP-TEE setup for Raspberry Pi 3 uses only upstream tree’s with the exception of Linux kernel.
Disclaimer
Warning
This port of Trusted Firmware A and OP-TEE to Raspberry Pi 3 IS NOT SECURE! Although the Raspberry Pi3 processor provides ARM TrustZone exception states, the mechanisms and hardware required to implement secure boot, memory, peripherals or other secure functions are not available. Use of OP-TEE or TrustZone capabilities within this package does not result in a secure implementation. This package is provided solely for educational purposes and prototyping.
What is expected to work?
First, note that all OP-TEE developer builds (ref, build) have rather simple overall goals:
Successfully build OP-TEE for certain devices.
Run xtest and optee_example binaries successfully with no regressions using UART(s).
I.e., it is important to understand that our “OP-TEE developer builds” shall not be compared with full Linux distributions which supports “everything”. As a couple of examples, we don’t enable any particular drivers in Linux kernel, we don’t include all sorts of daemons, we do not include an X-environment etc. At the same time this doesn’t mean that you cannot use OP-TEE in real environments. It is usually perfectly fine to run on all sorts of devices, environments etc. It’s just that for the OP-TEE developer builds we have intentionally stripped down the environment to make it rather fast to get all the source code, build it all and run xtest.
We are highlighting this here, since over the years we have had many questions at GitHub about things that people usually find working on their Raspberry Pi devices when they are using Raspbian (which this is not). The table below describes what is officially supported in the Raspberry Pi 3 OP-TEE developer builds and right after that follows sections for each of giving a bit more context to it.
Name
Supported?
Buildroot
Yes
HDMI
No
NFS
Yes
Random packages
Maybe
Raspbian
No
Secure boot
Maybe
TFTP
Yes
UART
Yes
Wi-Fi
No
Buildroot
We are using Buildroot as the tool to create a stripped down filesystem for
Linux where we also put OP-TEE binaries like Trusted Applications, client
libraries and TEE supplicant. If a user wants to add/enable additional packages,
then that is also possible by adding new lines in common.mk
in build
(search for BR2_PACKAGE_
in the git to see how it’s done).
HDMI
X isn’t enabled and we have not built nor enabled any drivers for graphics.
NFS
Works to boot up a Linux root filesystem, more on that further down.
Random packages
See the Buildroot section above. You can enable packages supported by Buildroot, but as mentioned initially in this section, lack of drivers and other daemons etc might make it impossible to run.
Raspbian
We are not using it. However, people (from Sequitur Labs) have successfully been able to add OP-TEE to Raspbian builds. But since we’re not using it and haven’t tried, we simply don’t support it.
Secure boot
First pay attention to the initial warning on this page. I.e., no matter what you are doing with Raspberry Pi and TrustZone / OP-TEE you cannot make it secure. But that doesn’t mean that you cannot “enable” secure features as such for prototyping and to learn how to build and use those. That kind of knowledge can later on be transferred and used on other devices which have all the necessary secure capabilities needed to make a secure system. We haven’t tested to enable secure boot on Raspberry Pi 3. But we believe that a good starting point would be Trusted Firmware A’s documentation about the “Authentication Framework” and RPi3 in TF-A.
TFTP
When you reach U-Boot (see Boot sequence), then you can start using
TFTP to load boot firmware etc. Note that if you overwrite armstub8.bin
for
example and that happens to be faulty, then you will need to re-mount the BOOT
partition on the SD-card and put a new working version of it. Also note that
changing early boot binaries (TF-A, OP-TEE core etc) will require you to reboot
the device see the changes.
UART
Fully supported, for more details look at the UART section further down.
Wi-Fi
Even though Raspberry Pi 3 has a Wi-Fi chip, we do not support it in our stripped down builds.
What versions of Raspberry Pi will work?
Below is a table of supported hardware in our OP-TEE developer builds. We have only used the Raspberry Pi 3 Model B, i.e., the first RPi 3 device that was released. But we know that people have successfully been able to use it with both RPi 2’s as well as the newer RPi 3 B+. But as long as we in the core team doesn’t have those at hands we cannot guarantee anything, therefore we simply say “No” below.
Hardware
Supported?
Raspberry Pi 1 Model A
No
Raspberry Pi 1 Model B
No
Raspberry Pi 1+ Model A
No
Raspberry Pi 1+ Model B
No
Raspberry Pi 2 Model B
No
Raspberry Pi 2 Model B v1.2
No
Raspberry Pi 3+ Model A
No
Raspberry Pi 3 Model B
Yes
Raspberry Pi 3+ Model B
Yes
Raspberry Pi 4
No
Zero - all versions
No
Compute module - all versions
No
Boot sequence
The GPU starts executing the first stage bootloader, which is stored in ROM on the SoC. The first stage bootloader reads the SD-card, and loads the second stage bootloader (
bootcode.bin
) into the L2 cache, and runs it.
bootcode.bin
enables SDRAM, and reads the third stage bootloaderloader.bin
from the SD-card into RAM, and runs it.
loader.bin
reads the GPU firmware (start.elf
).
start.elf
readsconfig.txt
, pre-loadsarmstub8.bin
(which contains: BL1/TF-A + BL2/TF-A + BL31/TF-A + BL32/OP-TEE + BL33/U-boot) to0x0
and jumps to the first instruction.A traditional boot sequence of TF-A -> OP-TEE -> U-boot is performed, i.e., BL1 loads BL2, then BL2 loads and run BL31(SM), BL32(OP-TEE), BL33(U-boot) (one after another)
U-Boot runs
fatload/booti
sequence to load from eMMC to RAM bothzImage
and thenDTB
and boot.
Build instructions
Start by following the Get and build the solution as described in build, but stop at the “Step 6 - Flash the device” step (i.e., don’t run the make flash command!).
Next step is to partition and format the memory card and to put the files onto the same. That is something we don’t want to automate, since if anything goes wrong, in worst case it might wipe one of your regular hard disks. Instead what we have done, is that we have created another makefile target that will tell you exactly what to do. Run that command and follow the instructions there.
$ make img-help
Note
The mention of
/dev/sdx1
and/dev/sdx2
when running the command above are just examples. You need to figure out and replace that with the correct name(s) for your computer and SD-card (typically rundmesg
and look for the device name matching your SD-card).Put the SD-card back into the Raspberry Pi 3.
Plug in the UART cable and attach to the UART
$ picocom -b 115200 /dev/ttyUSB0
Note
Install picocom if not already installed
$ sudo apt-get install picocom
.Power up the Raspberry Pi 3 and the system shall start booting which you will see on the UART (not HDMI).
When you have a shell, then it’s simply just to follow the “Step 9 - Run xtest” instructions.
NFS boot
Booting via NFS is quite useful for several reasons, but the obvious reason when working with Raspberry Pi is that you don’t have to move the SD-card back and forth between the host machine and the Raspberry Pi 3 itself when working with Normal World files, like Linux kernel and user space programs. Here we will describe how to setup NFS server, so the rootfs can be mounted via NFS.
Warning
This guide doesn’t focus on any desktop security, so eventually you would need to harden your setup.
In the description below we will use the following terminology, IP addresses and paths. The reader of this guide is supposed to update this to match his own environment.
192.168.1.100 <--- This is your desktop computer (NFS server)
192.168.1.200 <--- This is the Raspberry Pi
/srv/nfs/rpi <--- Location for the NFS share
Configure NFS
Start by installing the NFS server
$ sudo apt-get install nfs-kernel-server
Then edit the exports file,
$ sudo vim /etc/exports
In this file you shall tell where your files/folder are and the IP’s allowed to access the files. The way it’s written below will make it available to every machine on the same subnet (again, be careful about security here). Let’s add this line to the file (it’s the only line necessary in the file, but if you have several different filesystems available, then you should of course add them too, one line for each share).
/srv/nfs/rpi 192.168.1.0/24(rw,sync,no_root_squash,no_subtree_check)
Next create the folder where you are going to put the root filesystem
$ sudo mkdir /srv/nfs/rpi
After this, restart the NFS kernel server
$ service nfs-kernel-server restart
Hint
To see that your shares are correctly setup and that the NFS server is
running, you can run: $ showmount --all localhost
and you should get a
list of IP:<path>'s
based on what you have added in your exports file.
If you get nothing there, then your NFS server hasn’t been setup correctly.
uboot.env configuration
The file uboot.env
contains boot configurations that tells what binaries to
load and at what addresses. When using NFS you need to tell U-Boot where the NFS
server is located (IP and path). Since the exact IP and path varies for each
user, we must update uboot.env
accordingly.
There are two ways to update uboot.env
, one is to update
uboot.env.txt
(in build) and the other is to update directly from
the U-Boot console. Pick the one that you suits your needs. We will cover each
of them separately here.
Change uboot.env.txt
In an editor open: <rpi3-project>/build/rpi3/firmware/uboot.env.txt
and
change:
nfsserverip
to match the IP address of your NFS server.
gatewayip
to the IP address of your router.
nfspath
to the exported filesystem in your NFS share.
As an example a section of uboot.env.txt
could look like this:
# NFS/TFTP boot configuraton
gatewayip=192.168.1.1
netmask=255.255.255.0
nfsserverip=192.168.1.100
nfspath=/srv/nfs/rpi
Next, you need to re-generate uboot.env
:
$ cd <rpi3-project>/build
$ make u-boot-env-clean
$ make u-boot-env
Finally, you need to copy the updated <rpi3-project>/out/uboot.env
to the
BOOT partition of your SD-card (mount it as described in
Build instructions and then just overwrite (cp
) the file on the
BOOT partition of your SD-card).
Update u-boot.env from U-Boot console
Boot up the device until you see U-Boot running and counting down, then hit any
key and will see the U-Boot>
prompt. You can then update the
nfsserverip
, gatewayip
and nfspath
by writing
U-Boot> setenv nfsserverip '192.168.1.100'
U-Boot> setenv gatewayip '192.168.1.1'
U-Boot> setenv nfspath '/srv/nfs/rpi'
If you want those environment variables to persist between boots, then type.
U-Boot> saveenv
Boot up with NFS
With all preparations above done correctly, you should now be able to boot up the device and kernel, secure side OP-TEE and the entire root filesystem should be loaded from the network shares (NFS). Power up the Raspberry, halt in U-Boot and then type.
U-Boot> run nfsboot
If everything works, you can simply copy paste files like xtest
, Trusted
Applications and other things that usually resides on the host PC’s filesystem,
i.e., directly from your build folders to the /srv/nfs/rpi/...
folders. By
doing so you don’t have to reboot the device when doing development and testing.
Just rebuild and copy is sufficient.
Note
You cannot make symlinks in the NFS share to the built files, i.e., you must copy them!
JTAG
To enable JTAG you need to add a line saying enable_jtag_gpio=1
in
config.txt
. There are two ways you can do this, both requires that you to
mount the BOOT partition on the SD-card at your computer (see the make
img-help
step under Build instructions). After you have
mounted the BOOT partition continue with whichever way is most suitable for you.
Change config.txt directly
With your editor, open /media/boot/config.txt
and add a line
enable_jtag_gpio=1
, save the file, unmount the BOOT partition and you’re
good to go after rebooting the device.
Rebuild and untar
With your editor, open
<rpi3-project>/build/rpi3/firmware/config.txt
and add a lineenable_jtag_gpio=1
, save the file.$ cd <rpi3-project>/build && make
$ cd /media
$ sudo gunzip -cd <rpi3-project>/out-br/images/rootfs.cpio.gz | sudo cpio -idmv "boot/*"
Note
You didn’t forget to mount the BOOT partition before trying this step?
Unmount the BOOT partition and you’re good to go after rebooting the device.
JTAG/RPi3 cable
We have created our own cables that consists of a standard 20-pin JTAG connector and a 22-pin connector for the Raspberry Pi 3 itself. Then using a ribbon cable we have connected the cables according to the table below (JTAG pin <-> Raspberry Pi 3 Header pin).
JTAG pin |
Signal |
GPIO |
Mode |
RPi3 Header pin |
---|---|---|---|---|
1 |
3v3 |
N/A |
N/A |
1 |
3 |
nTRST |
GPIO22 |
ALT4 |
15 |
5 |
TDI |
GPIO26 |
ALT4 |
37 |
7 |
TMS |
GPIO27 |
ALT4 |
13 |
9 |
TCK |
GPIO25 |
ALT4 |
22 |
11 |
RTCK |
GPIO23 |
ALT4 |
16 |
13 |
TDO |
GPIO24 |
ALT4 |
18 |
18 |
GND |
N/A |
N/A |
14 |
20 |
GND |
N/A |
N/A |
20 |
Warning
Be careful and cross check the wiring as incorrect wiring might damage your device! Also be careful to connect the cable correctly at both ends (don’t flip it and don’t put it at the wrong pins in the Raspberry Pi 3 side).
UART/RPi3 cable
In addition to the JTAG connections we have also wired up the RX/TX to be able to use the UART. Note, for this you don’t need to do JTAG wirings, i.e., it’s perfectly fine to just wire up the UART only. There are many ready made cables for this on the net (eBay) and cost almost nothing. Get one of those if you don’t intend to use JTAG.
UART pin |
Signal |
GPIO |
Mode |
RPi3 Header pin |
---|---|---|---|---|
Black (GND) |
GND |
N/A |
N/A |
6 |
White (RXD) |
TXD |
GPIO14 |
ALT0 |
8 |
Green (TXD) |
RXD |
GPIO15 |
ALT0 |
10 |
Warning
Be careful and cross check the wiring as incorrect wiring might damage your device!
OpenOCD
Build OpenOCD
Before building OpenOCD, ensure that you have the libusb-dev
installed.
$ sudo apt-get install libusb-1.0-0-dev
We are using the official OpenOCD release, simply clone that to your computer and then building is like a lot of other software, i.e.,
$ git clone http://repo.or.cz/openocd.git
$ cd openocd
$ ./bootstrap
$ ./configure
$ make
Note
In recent versions of OpenOCD, the legacy ft2332 support has been depracted. All these devices now uses libftdi instead. From OpenOCD release notes: “GPL-incompatible FTDI D2XX library support dropped (Presto, OpenJTAG and USB-Blaster I are using libftdi only now)”.
We leave it up to the reader of this guide to decide if he wants to install it
properly (make install
) or if he will just run it from the tree directly.
The rest of this guide will just run it from the tree.
OpenOCD RPi3 configuration file
Unfortunately, the necessary RPi3 OpenOCD config isn’t upstreamed yet into
the official OpenOCD repository, so you should use the one stored here
<rpi3-project/build/rpi3/debugger/pi3.cfg
.
Running OpenOCD
Depending on the JTAG debugger you are using you’ll need to find and use the interface file for that particular debugger. We’ve been using J-Link debuggers and Bus Blaster successfully. To start an OpenOCD session using a J-Link device you type:
$ cd <openocd>
$ ./src/openocd -f ./tcl/interface/jlink.cfg -f <rpi3-project>/build/rpi3/debugger/pi3.cfg
For Bus Blaster type:
$ ./src/openocd -f ./tcl/interface/ftdi/dp_busblaster.cfg \ -f <rpi3_repo_dir>/build/rpi3/debugger/pi3.cfg
To be able to write commands directly to OpenOCD, you simply open up another shell and type:
$ nc localhost 4444
From there you can set breakpoints, examine memory etc (”> help
” will give
you a list of available commands). Having that said, if you connect to OpenOCD
using GDB, then there is not much incentive connecting to OpenOCD directly,
since you will be able to do the same in GDB by the monitor
command.
Use GDB
OpenOCD will by default listen to GDB connections on port 3333
. So after
starting OpenOCD, make a connection to GDB.
# Ensure that you have "gdb" in your $PATH
$ aarch64-linux-gnu-gdb -q
(gdb) target remote localhost:3333
To load symbols you just use the symbol-file <path/to/my.elf
as usual. For
convenience you can create an alias in the ~/.gdbinit
file. For TEE core
debugging this works:
define jtag_rpi3
target remote localhost:3333
symbol-file <rpi3-project>/optee_os/out/arm/core/tee.elf
end
So, when running GDB, you simply type: (gdb) jtag_rpi3
and it will both
connect and load the symbols for TEE core. For Linux kernel and other binaries
you would do the same.
Debug session example
After making an initial Raspberry Pi 3 build for OP-TEE where you’ve enabled JTAG, installed and built OpenOCD, connected the JTAG cable, then you’re ready for debugging OP-TEE using JTAG on Raspberry 3. Boot up the Raspberry Pi 3 until you are in Linux and ready to run xtest. Start a new shell (on the host machine) where you run OpenOCD:
$ cd <openocd>
$ ./src/openocd -f ./tcl/interface/jlink.cfg -f <rpi3-project>/build/rpi3/debugger/pi3.cfg
Start another shell, where you run GDB
$ <rpi3-project>/toolchains/aarch64/bin/aarch64-linux-gnu-gdb -q
(gdb) target remote localhost:3333
(gdb) symbol-file <rpi3-project>/optee_os/out/arm/core/tee.elf
Next, try to set a breakpoint for the function hmac_init
, here use
hardware breakpoints (i.e., hb
)!
(gdb) hb hmac_init
Hardware assisted breakpoint 2 at 0x1012a178: file core/lib/libtomcrypt/src/mac/hmac/hmac_init.c, line 65.
(gdb) c
Continuing.
In the UART console (RPi3/Linux), run xtest.
# xtest
And shortly thereafter you will see GDB stops on your breakpoint and from there you can debug using normal GDB commands.