Perfetto for upstream kernel development

Perfetto is super useful for understanding interactions between the kernel and applications. Outside of Android and ChromeOS, though it's use isn't as common. This doc tries to provide a basic walk through to get started using perfetto for upstream kernel development with classic linux distros, potentially running under qemu.

Install perfetto

Grab the latest linux- tarball: https://github.com/google/perfetto/releases

Often the tests I’m tracing need to run as root, so because of this, I copied the binaries in the tarball to /usr/local/bin/ and chmod +x the binaries to make them executable.

Qemu Environment

Handling illegal instruction errors:

The perfetto binaries are built with lots of optimizations enabled, If you’re running with qemu, the default virtual cpu will not necessarily have those instructions enabled, resulting in illegal instruction crashes. To fix this, pass -cpu host to qemu.

Kernel Config

Required/Suggested Kernel configs:

	CONFIG_FTRACE=y
	CONFIG_FUNCTION_TRACER=y
	CONFIG_FUNCTION_GRAPH_TRACER=y
	CONFIG_HAVE_DYNAMIC_FTRACE=y

Helper script

Here is my trace-me.sh script (which I also put in /usr/local/bin/ and mark as executable) which I use to run perfetto/tracebox to capture trace logs for specific tests

#!/bin/bash
# Copyright 2024 Google LLC.
# SPDX-License-Identifier: Apache-2.0
PERFETTO_PID=""
PERFETTO_CONFIG="perf.conf"
PERFETTO_OUTPUT_FILE="trace.out"
CMD="$@"

function start_trace {
        killall tracebox &> /dev/null
        rm -f "${PERFETTO_OUTPUT_FILE}.gz"
        rm -f "${PERFETTO_OUTPUT_FILE}"
        PERFETTO_PID=$(tracebox -c "${PERFETTO_CONFIG}" --txt --background-wait -o "${PERFETTO_OUTPUT_FILE}")
        while [ $? -ne 0 ]
        do
                killall tracebox &> /dev/null
                sleep 1
                echo "tracebox had trouble starting, trying again..."
                PERFETTO_PID=$(tracebox -c "${PERFETTO_CONFIG}" --txt --background-wait -o "${PERFETTO_OUTPUT_FILE}")
        done
        echo "tracebox: $PERFETTO_PID" 
        sleep 1;
}

function stop_trace {
        echo "tracebox: $PERFETTO_PID stopping"
        kill -INT ${PERFETTO_PID}
        while ps -p ${PERFETTO_PID}; do sleep 1; done > /dev/null
        echo "tracebox: $PERFETTO_PID stopped"
        rm -f "${PERFETTO_OUTPUT_FILE}.gz"
        chmod a+wr "${PERFETTO_OUTPUT_FILE}"
        gzip "${PERFETTO_OUTPUT_FILE}"
}

start_trace
eval $CMD
RET=$?
stop_trace
exit $RET

Trace File

Create a trace config file named perf.conf in the directory where you are going to run the test:

write_into_file: true
buffers: {
    size_kb: 40960
    fill_policy: RING_BUFFER
}
buffers: {
    size_kb: 2048
    fill_policy: RING_BUFFER
}
data_sources: {
    config {
        name: "linux.ftrace"
        ftrace_config {
            buffer_size_kb: 10240
            ftrace_events: "sched/*"
            ftrace_events: "task/*"
            ftrace_events: "irq/*"
        }
    }
}
data_sources: {
    config {
        name: "linux.process_stats"
        target_buffer: 1
        process_stats_config {
            scan_all_processes_on_start: true
        }
    }
}
duration_ms: 300000
flush_period_ms: 30000
incremental_state_config {
    clear_period_ms: 5000
}

Then run trace-me.sh <command> from the directory where you saved perf.conf This will create a trace.out.gz in the same dir.

From there, I copy the trace.out.gz file to another machine and either open it with: $(AOSP_DIR)/external/perfetto/tools/open_trace_in_ui -i trace.out.gz -n

Or open the trace.out.gz file via the ui.perfetto.dev web UI

Function Graph Tracing

If you want to get kernel call stacks in the perfetto logs as well, you can do so by enabling function graph tracing. Double check CONFIG_FUNCTION_GRAPH_TRACER=y is enabled (you’ll need to rebuild your kernel on Android, as it is off by default).

Then you can add the following chunk to your linux.ftrace ftrace_config chunk in the perfetto config:

data_sources {
  config {
    name: "linux.ftrace"
    ftrace_config {
      ftrace_events: "sched/*"
...
      symbolize_ksyms: true
      ksyms_mem_policy: KSYMS_RETAIN
      enable_function_graph: true
      function_graph_roots: "sched_ttwu_pending"
      function_graph_roots: "try_to_wake_up"
    }
  }
}

Then modify the function_graph_roots: values to filter from where the function graph will start. You can run it without the function_graph_roots, but for even very short traces it will generate so much data you will likely lose some trace data, resulting in confusing traces. So pick carefully where you start.

Userland Annotations via trace_marker

It can be useful to add annotations from userland into the perfetto tracks, so you can match changes in userland to changes in the kernel. To do so, first, add to the linux.ftrace ftrace_config: ftrace_events: ftrace/print

data_sources {
  config {
    name: "linux.ftrace"
    ftrace_config {
      ftrace_events: "sched/*"
...
      ftrace_events: "ftrace/print"
...
    }
  }
}

Then while capturing a trace, your application need to write to /sys/kernel/tracing/trace_marker with the following format:

Instant events: I|<pid>|<message>

Which can even be done via the shell:

echo "I|$$|Test Begins" > /sys/kernel/tracing/trace_marker

Other types of annotations are possible as well, using this format

More info

There are tons of other config options you can use, all documented here

The perfetto web UI also can help you generate a config file. Just go through the various options and enable what you are interested in, and then visit recording command section and review the generated config file.

Power Users

Perfettos real power comes from it keeping all the trace data in an SQL database, which you can query via the query interface and extend the visualizations using SQL. So there is a lot more you can do with it if you are SQL savvy.

Thanks to:

Ryan Savitski for many of the tricks in this doc.

Neill Kapron for review and testing

austriancoder/perfetto-for-linux.md