Benchmark framework

Due to its nature, OP-TEE is being a solution spanning over several architectural layers, where each layer includes its own complex parts. For further optimizations of performance, there is a need of tool which will provide detailed and precise profiling information for each layer.

It is necessary to receive latency values for:

  • The roundtrip time for going from a client application in normal world, down to a Trusted Application and back again.

  • Detailed information for amount of time taken to go through each layer:

    • libTEEC -> Linux OP-TEE kernel driver
    • Linux OP-TEE kernel driver -> OP-TEE OS Core
    • OP-TEE OS Core -> TA entry point (not supported yet)
    • The same way back

Implementation details

Design overview

Benchmark framework consists of such components:

  1. Benchmark Client Application (CA): a dedicated client application, which is responsible for allocating timestamp circular buffers, registering these buffers in the Benchmark PTA and consuming all timestamp data generated by all OP-TEE layers. Finally, it puts timestamp data into appropriate file with .ts extension. Additional build details can be found at optee_benchmark.
  2. Benchmark Pseudo Trusted Application (PTA): which owns all per-cpu circular non-secure buffers from a shared memory. Benchmark PTA must be invoked (by a CA) to register the timestamp circular buffers. In turn, the Benchmark PTA invokes the OP-TEE Linux driver (through some RPC mean) to register this circular buffers in the Linux kernel layer.
  3. libTEEC and Linux kernel OP-TEE driver include functionality for handling timestamp buffer registration requests from the Benchmark PTA.

When the benchmark is enabled, all OP-TEE layers (libTEEC, Linux kernel OP-TEE driver, OP-TEE OS core) do fill the registered timestamp circular buffer with timestamp data for all invocation requests on condition that the circular buffer is allocated/registered.


Timestamp source

Arm Performance Monitor Units are used as the main source of timestamp values. The reason why this technology was chosen is that it is supported on all Armv7-A/Armv8-A cores. Besides it can provide precise pre-cpu cycle counter values, it is possible to enable EL0 access to all events, so usermode applications can directly read cpu counter values from coprocessor registers, achieving minimal latency by avoiding additional syscalls to EL1 core.

Besides CPU cycle counter values, timestamp by itself contains also information about:

  • Executing CPU core index
  • OP-TEE layer id, where this timestamp was obtained from
  • Program counter value when timestamp was logged, which can be used for getting a symbol name (a filename and line number)

Call sequence diagram


Adding custom timestamps

Currently, timestamping is done only for InvokeCommand calls, but it’s also possible to choose custom places in the supported OP-TEE layers. To add timestamp storing command to custom c source file:

  1. Include appropriate header:

    • OP-TEE OS Core: bench.h
    • Linux kernel OP-TEE module: optee_bench.h
    • libTEEC: teec_benchmark.h
  2. Invoke bm_timestamp() (for linux kmod use optee_bm_timestamp()) in the function, where you want to put timestamp from.

Build and run benchmark

Please see the instructions available at optee_benchmark.

Limitations and further steps

  • Implementation of application which will analyze timestamp data and provide statistics for different types of calls providing avg/min/max values (both CPU cycles and time values).
  • Add support for all platforms, where OP-TEE is supported.
  • Adding support of S-EL0 timestamping.
  • Attaching additional payload information to each timestamp, for example, session.
  • Timestamping within interrupt context in the OP-TEE OS Core.