Raspberry Pi 3¶
Sequitur Labs did the initial OP-TEE port which at the time also came with modifications in U-Boot, Trusted Firmware A and Linux kernel. Since that initial port more and more patches have found mainline trees and today the OP-TEE setup for Raspberry Pi 3 uses only upstream tree’s with the exception of Linux kernel.
This port of Trusted Firmware A and OP-TEE to Raspberry Pi 3 IS NOT SECURE! Although the Raspberry Pi3 processor provides ARM TrustZone exception states, the mechanisms and hardware required to implement secure boot, memory, peripherals or other secure functions are not available. Use of OP-TEE or TrustZone capabilities within this package does not result in a secure implementation. This package is provided solely for educational purposes and prototyping.
What is expected to work?¶
First, note that all OP-TEE developer builds (ref, build) have rather simple overall goals:
- Successfully build OP-TEE for certain devices.
- Run xtest and optee_example binaries successfully with no regressions using UART(s).
I.e., it is important to understand that our “OP-TEE developer builds” shall not be compared with full Linux distributions which supports “everything”. As a couple of examples, we don’t enable any particular drivers in Linux kernel, we don’t include all sorts of daemons, we do not include an X-environment etc. At the same time this doesn’t mean that you cannot use OP-TEE in real environments. It is usually perfectly fine to run on all sorts of devices, environments etc. It’s just that for the OP-TEE developer builds we have intentionally stripped down the environment to make it rather fast to get all the source code, build it all and run xtest.
We are highlighting this here, since over the years we have had many questions at GitHub about things that people usually find working on their Raspberry Pi devices when they are using Raspbian (which this is not). The table below describes what is officially supported in the Raspberry Pi 3 OP-TEE developer builds and right after that follows sections for each of giving a bit more context to it.
Name Supported? Buildroot Yes HDMI No NFS Yes Random packages Maybe Raspbian No Secure boot Maybe TFTP Yes UART Yes Wi-Fi No
We are using Buildroot as the tool to create a stripped down filesystem for
Linux where we also put OP-TEE binaries like Trusted Applications, client
libraries and TEE supplicant. If a user wants to add/enable additional packages,
then that is also possible by adding new lines in
common.mk in build
BR2_PACKAGE_ in the git to see how it’s done).
X isn’t enabled and we have not built nor enabled any drivers for graphics.
Works to boot up a Linux root filesystem, more on that further down.
See the Buildroot section above. You can enable packages supported by Buildroot, but as mentioned initially in this section, lack of drivers and other daemons etc might make it impossible to run.
We are not using it. However, people (from Sequitur Labs) have successfully been able to add OP-TEE to Raspbian builds. But since we’re not using it and haven’t tried, we simply don’t support it.
First pay attention to the initial warning on this page. I.e., no matter what you are doing with Raspberry Pi and TrustZone / OP-TEE you cannot make it secure. But that doesn’t mean that you cannot “enable” secure features as such for prototyping and to learn how to build and use those. That kind of knowledge can later on be transferred and used on other devices which have all the necessary secure capabilities needed to make a secure system. We haven’t tested to enable secure boot on Raspberry Pi 3. But we believe that a good starting point would be Trusted Firmware A’s documentation about the “Authentication Framework” and RPi3 in TF-A.
When you reach U-Boot (see Boot sequence), then you can start using
TFTP to load boot firmware etc. Note that if you overwrite
example and that happens to be faulty, then you will need to re-mount the BOOT
partition on the SD-card and put a new working version of it. Also note that
changing early boot binaries (TF-A, OP-TEE core etc) will require you to reboot
the device see the changes.
Fully supported, for more details look at the UART section further down.
Even though Raspberry Pi 3 has a Wi-Fi chip, we do not support it in our stripped down builds.
What versions of Raspberry Pi will work?¶
Below is a table of supported hardware in our OP-TEE developer builds. We have only used the Raspberry Pi 3 Model B, i.e., the first RPi 3 device that was released. But we know that people have successfully been able to use it with both RPi 2’s as well as the newer RPi 3 B+. But as long as we in the core team doesn’t have those at hands we cannot guarantee anything, therefore we simply say “No” below.
Hardware Supported? Raspberry Pi 1 Model A No Raspberry Pi 1 Model B No Raspberry Pi 1+ Model A No Raspberry Pi 1+ Model B No Raspberry Pi 2 Model B No Raspberry Pi 2 Model B v1.2 No Raspberry Pi 3+ Model A No Raspberry Pi 3 Model B Yes Raspberry Pi 3+ Model B Yes Raspberry Pi 4 No Zero - all versions No Compute module - all versions No
- The GPU starts executing the first stage bootloader, which is stored in ROM on the SoC. The first stage bootloader reads the SD-card, and loads the second stage bootloader (
bootcode.bin) into the L2 cache, and runs it.
bootcode.binenables SDRAM, and reads the third stage bootloader
loader.binfrom the SD-card into RAM, and runs it.
loader.binreads the GPU firmware (
armstub8.bin(which contains: BL1/TF-A + BL2/TF-A + BL31/TF-A + BL32/OP-TEE + BL33/U-boot) to
0x0and jumps to the first instruction.
- A traditional boot sequence of TF-A -> OP-TEE -> U-boot is performed, i.e., BL1 loads BL2, then BL2 loads and run BL31(SM), BL32(OP-TEE), BL33(U-boot) (one after another)
- U-Boot runs
fatload/bootisequence to load from eMMC to RAM both
Next step is to partition and format the memory card and to put the files onto the same. That is something we don’t want to automate, since if anything goes wrong, in worst case it might wipe one of your regular hard disks. Instead what we have done, is that we have created another makefile target that will tell you exactly what to do. Run that command and follow the instructions there.
$ make img-help
The mention of
/dev/sdx2when running the command above are just examples. You need to figure out and replace that with the correct name(s) for your computer and SD-card (typically run
dmesgand look for the device name matching your SD-card).
Put the SD-card back into the Raspberry Pi 3.
Plug in the UART cable and attach to the UART
$ picocom -b 115200 /dev/ttyUSB0
Install picocom if not already installed
$ sudo apt-get install picocom.
Power up the Raspberry Pi 3 and the system shall start booting which you will see on the UART (not HDMI).
When you have a shell, then it’s simply just to follow the “Step 9 - Run xtest” instructions.
Booting via NFS is quite useful for several reasons, but the obvious reason when working with Raspberry Pi is that you don’t have to move the SD-card back and forth between the host machine and the Raspberry Pi 3 itself when working with Normal World files, like Linux kernel and user space programs. Here we will describe how to setup NFS server, so the rootfs can be mounted via NFS.
This guide doesn’t focus on any desktop security, so eventually you would need to harden your setup.
In the description below we will use the following terminology, IP addresses and paths. The reader of this guide is supposed to update this to match his own environment.
192.168.1.100 <--- This is your desktop computer (NFS server) 192.168.1.200 <--- This is the Raspberry Pi /srv/nfs/rpi <--- Location for the NFS share
Start by installing the NFS server
$ sudo apt-get install nfs-kernel-server
Then edit the exports file,
$ sudo vim /etc/exports
In this file you shall tell where your files/folder are and the IP’s allowed to access the files. The way it’s written below will make it available to every machine on the same subnet (again, be careful about security here). Let’s add this line to the file (it’s the only line necessary in the file, but if you have several different filesystems available, then you should of course add them too, one line for each share).
Next create the folder where you are going to put the root filesystem
$ sudo mkdir /srv/nfs/rpi
After this, restart the NFS kernel server
$ service nfs-kernel-server restart
To see that your shares are correctly setup and that the NFS server is
running, you can run:
$ showmount --all localhost and you should get a
IP:<path>'s based on what you have added in your exports file.
If you get nothing there, then your NFS server hasn’t been setup correctly.
uboot.env contains boot configurations that tells what binaries to
load and at what addresses. When using NFS you need to tell U-Boot where the NFS
server is located (IP and path). Since the exact IP and path varies for each
user, we must update
There are two ways to update
uboot.env, one is to update
uboot.env.txt (in build) and the other is to update directly from
the U-Boot console. Pick the one that you suits your needs. We will cover each
of them separately here.
In an editor open:
nfsserveripto match the IP address of your NFS server.
gatewayipto the IP address of your router.
nfspathto the exported filesystem in your NFS share.
As an example a section of
uboot.env.txt could look like this:
# NFS/TFTP boot configuraton gatewayip=192.168.1.1 netmask=255.255.255.0 nfsserverip=192.168.1.100 nfspath=/srv/nfs/rpi
Next, you need to re-generate
$ cd <rpi3-project>/build $ make u-boot-env-clean $ make u-boot-env
Finally, you need to copy the updated
<rpi3-project>/out/uboot.env to the
BOOT partition of your SD-card (mount it as described in
Build instructions and then just overwrite (
cp) the file on the
BOOT partition of your SD-card).
Update u-boot.env from U-Boot console¶
Boot up the device until you see U-Boot running and counting down, then hit any
key and will see the
U-Boot> prompt. You can then update the
nfspath by writing
U-Boot> setenv nfsserverip '192.168.1.100' U-Boot> setenv gatewayip '192.168.1.1' U-Boot> setenv nfspath '/srv/nfs/rpi'
If you want those environment variables to persist between boots, then type.
Boot up with NFS¶
With all preparations above done correctly, you should now be able to boot up the device and kernel, secure side OP-TEE and the entire root filesystem should be loaded from the network shares (NFS). Power up the Raspberry, halt in U-Boot and then type.
U-Boot> run nfsboot
If everything works, you can simply copy paste files like
Applications and other things that usually resides on the host PC’s filesystem,
i.e., directly from your build folders to the
/srv/nfs/rpi/... folders. By
doing so you don’t have to reboot the device when doing development and testing.
Just rebuild and copy is sufficient.
You cannot make symlinks in the NFS share to the built files, i.e., you must copy them!
To enable JTAG you need to add a line saying
config.txt. There are two ways you can do this, both requires that you to
mount the BOOT partition on the SD-card at your computer (see the
img-help step under Build instructions). After you have
mounted the BOOT partition continue with whichever way is most suitable for you.
Change config.txt directly¶
With your editor, open
/media/boot/config.txt and add a line
enable_jtag_gpio=1, save the file, unmount the BOOT partition and you’re
good to go after rebooting the device.
Rebuild and untar¶
With your editor, open
<rpi3-project>/build/rpi3/firmware/config.txtand add a line
enable_jtag_gpio=1, save the file.
$ cd <rpi3-project>/build && make
$ cd /media
$ sudo gunzip -cd <rpi3-project>/out-br/images/rootfs.cpio.gz | sudo cpio -idmv "boot/*"
You didn’t forget to mount the BOOT partition before trying this step?
Unmount the BOOT partition and you’re good to go after rebooting the device.
We have created our own cables that consists of a standard 20-pin JTAG connector and a 22-pin connector for the Raspberry Pi 3 itself. Then using a ribbon cable we have connected the cables according to the table below (JTAG pin <-> Raspberry Pi 3 Header pin).
|JTAG pin||Signal||GPIO||Mode||RPi3 Header pin|
Be careful and cross check the wiring as incorrect wiring might damage your device! Also be careful to connect the cable correctly at both ends (don’t flip it and don’t put it at the wrong pins in the Raspberry Pi 3 side).
In addition to the JTAG connections we have also wired up the RX/TX to be able to use the UART. Note, for this you don’t need to do JTAG wirings, i.e., it’s perfectly fine to just wire up the UART only. There are many ready made cables for this on the net (eBay) and cost almost nothing. Get one of those if you don’t intend to use JTAG.
|UART pin||Signal||GPIO||Mode||RPi3 Header pin|
Be careful and cross check the wiring as incorrect wiring might damage your device!
Before building OpenOCD, ensure that you have the
$ sudo apt-get install libusb-1.0-0-dev
We are using the official OpenOCD release, simply clone that to your computer and then building is like a lot of other software, i.e.,
$ git clone http://repo.or.cz/openocd.git $ cd openocd $ ./bootstrap $ ./configure $ make
In recent versions of OpenOCD, the legacy ft2332 support has been depracted. All these devices now uses libftdi instead. From OpenOCD release notes: “GPL-incompatible FTDI D2XX library support dropped (Presto, OpenJTAG and USB-Blaster I are using libftdi only now)”.
We leave it up to the reader of this guide to decide if he wants to install it
make install) or if he will just run it from the tree directly.
The rest of this guide will just run it from the tree.
OpenOCD RPi3 configuration file¶
Depending on the JTAG debugger you are using you’ll need to find and use the interface file for that particular debugger. We’ve been using J-Link debuggers and Bus Blaster successfully. To start an OpenOCD session using a J-Link device you type:
$ cd <openocd> $ ./src/openocd -f ./tcl/interface/jlink.cfg -f <rpi3-project>/build/rpi3/debugger/pi3.cfg
For Bus Blaster type:
$ ./src/openocd -f ./tcl/interface/ftdi/dp_busblaster.cfg \ -f <rpi3_repo_dir>/build/rpi3/debugger/pi3.cfg
To be able to write commands directly to OpenOCD, you simply open up another shell and type:
$ nc localhost 4444
From there you can set breakpoints, examine memory etc (”
> help” will give
you a list of available commands). Having that said, if you connect to OpenOCD
using GDB, then there is not much incentive connecting to OpenOCD directly,
since you will be able to do the same in GDB by the
OpenOCD will by default listen to GDB connections on port
3333. So after
starting OpenOCD, make a connection to GDB.
# Ensure that you have "gdb" in your $PATH $ aarch64-linux-gnu-gdb -q (gdb) target remote localhost:3333
To load symbols you just use the
symbol-file <path/to/my.elf as usual. For
convenience you can create an alias in the
~/.gdbinit file. For TEE core
debugging this works:
define jtag_rpi3 target remote localhost:3333 symbol-file <rpi3-project>/optee_os/out/arm/core/tee.elf end
So, when running GDB, you simply type:
(gdb) jtag_rpi3 and it will both
connect and load the symbols for TEE core. For Linux kernel and other binaries
you would do the same.
Debug session example¶
After making an initial Raspberry Pi 3 build for OP-TEE where you’ve enabled JTAG, installed and built OpenOCD, connected the JTAG cable, then you’re ready for debugging OP-TEE using JTAG on Raspberry 3. Boot up the Raspberry Pi 3 until you are in Linux and ready to run xtest. Start a new shell (on the host machine) where you run OpenOCD:
$ cd <openocd> $ ./src/openocd -f ./tcl/interface/jlink.cfg -f <rpi3-project>/build/rpi3/debugger/pi3.cfg
Start another shell, where you run GDB
$ <rpi3-project>/toolchains/aarch64/bin/aarch64-linux-gnu-gdb -q (gdb) target remote localhost:3333 (gdb) symbol-file <rpi3-project>/optee_os/out/arm/core/tee.elf
Next, try to set a breakpoint for the function
hmac_init, here use
hardware breakpoints (i.e.,
(gdb) hb hmac_init Hardware assisted breakpoint 2 at 0x1012a178: file core/lib/libtomcrypt/src/mac/hmac/hmac_init.c, line 65. (gdb) c Continuing.
In the UART console (RPi3/Linux), run xtest.
And shortly thereafter you will see GDB stops on your breakpoint and from there you can debug using normal GDB commands.