eBPF works lite the JVM, running code instructions in an isolated environment in kernel-space. Compile C to BPF. If the code is safe the BPF program will be loaded into the kernel. Sharing data between the kernel and user-space: BPF maps.
BPF Program Types
- Tracing - help understand what's happening in the system
- Networking - inspect and manipulate the network traffic in your system. can let you filter packets coming from the network interface, or even reject those packets completely
View possible tracepoints you can attach BPF programs at: /sys/kernel/debug/tracing/events/
Linux 60-Second Analysis Checklist can be used for any performance issue:
$ uptime
$ dmesg | rail
$ vmstat 1
$ mpstat -P ALL 1
$ pidstat 1
$ iostat -xz 1
$ free -m
$ sar -n DEV 1
$ sar -n TCP,ETCP 1
$ topuptime is a quick way to view load averages, this gives a high-level idea of resource load (or demand)
vmstat 1 is the virtual memory statistics tool. Example:
[root@bpfbook ~]# vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 0 1548 229720 3880 563312 0 0 49 9 546 30 0 4 96 0 0Columns to check: r: the number of processes on CPU and waiting for a turn. an "r" value greater than the CPU count indicates saturation si and so: swap-in and swap-outs. If these are non-zero, you are out of memory us, sy, id, wa & st: these are breakdowns of CPU time, on average across all CPUs. They are user time, system time (kernel), idle, wait IO, and stolen time. If mostly user time go to next tool to profile user-space processes
mpstat -P ALL 1 this command prints per-CPU time broken down into states. Example:
$ # mpstat -P ALL 1
Linux 5.6.13-100.fc30.x86_64 (bpfbook) 11/22/2020 _x86_64_ (2 CPU)
02:32:43 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
02:32:44 PM all 0.50 0.00 0.00 0.00 1.00 2.00 0.00 0.00 0.00 96.50
02:32:44 PM 0 0.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.01
02:32:44 PM 1 0.00 0.00 0.00 0.00 2.02 4.04 0.00 0.00 0.00 93.94iostat -xz 1 shows storage device IO metrics.
$ iostat -xz 4
Linux 5.6.13-100.fc30.x86_64 (bpfbook) 11/22/2020 _x86_64_ (2 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
0.11 0.06 3.99 0.04 0.00 95.81
Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
sda 1.43 0.42 84.70 16.94 0.01 0.07 0.77 14.15 1.45 0.66 0.00 59.43 40.17 0.80 0.15Columns to check: r/s, w/s, rkB/s, wkB/s: thse are delivered reads, writes, read Kbytes and write Kbytes per second to the device. Use these for workload characterization. await: the average time for the IO in ms. This is the time the application suffers, as it's includes both time queued and time being serviced. %util: device utilization. showing the time each second that the device was doing work. Values greater than 60% typically lead to poor performance.
sar -n DEV 1 - here we use sar to look at network device metrics. Check interface throughput rxkB/s to see if any limit may have been reached
$ sar -n DEV q
Linux 5.6.13-100.fc30.x86_64 (bpfbook) 11/22/2020 _x86_64_ (2 CPU)
02:48:27 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
02:48:31 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:48:31 PM eth0 0.25 0.25 0.01 0.02 0.00 0.00 0.00 0.00
02:48:31 PM eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00sar -n TCP,ETCP 1 here we are using sar to look at TCP metrics and TCP errors:
# sar -n TCP,ETCP 3
Linux 5.6.13-100.fc30.x86_64 (bpfbook) 11/22/2020 _x86_64_ (2 CPU)
02:51:39 PM active/s passive/s iseg/s oseg/s
02:51:42 PM 0.00 0.00 0.33 0.33
02:51:39 PM atmptf/s estres/s retrans/s isegerr/s orsts/s
02:51:42 PM 0.00 0.00 0.00 0.00 0.00Columns to check:
active/s: number of locally initiated TCP connections per second passive/s: number of remotely initiated TCP connections per second retrans/s: number of TCP retransmits per second
Active and passive connection counts are useful for workload characterization. Retransmits are a sign of network or remote host issue.