As requested in netdev, this gist contains the stacks leading into skb:kfree_skb
:
The easiest way to look at the results is the flamegraphs, where reason
is the top frame.
The results are from v6.1.38.
We see ~10k calls per second during normal operation on this machine:
$ sudo perf record -a -g --kernel-callchains -e skb:kfree_skb -- sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 8.470 MB perf.data (9373 samples) ]
$ sudo perf script | sed -e 's/skb:kfree_skb:.*reason:\(.*\)/\n\tfffff \1 (unknown)/' -e 's/^\(\w\+\)\s\+/kernel /' | /usr/bin/inferno-collapse-perf --all | perl -pe 's/.*?;//' | sed -e 's/.*irq_exit_rcu_\[k\];/irq_exit_rcu_[k];/' | /usr/bin/inferno-flamegraph --truncate-text-right --colors=java --hash --title=normal --width=1440 --minwidth=0.005 > flamegraph.normal.svg
$ sudo /usr/share/bcc/tools/stackcount -s -K -D 1 t:skb:kfree_skb > stacks.normal.txt
Spikes up to 500k calls per second can happen from time to time:
$ sudo perf record -a -g --kernel-callchains -e skb:kfree_skb -- sleep 1
[ perf record: Woken up 172 times to write data ]
Warning:
4 out of order events recorded.
[ perf record: Captured and wrote 124.722 MB perf.data (398819 samples) ]
$ sudo perf script | sed -e 's/skb:kfree_skb:.*reason:\(.*\)/\n\tfffff \1 (unknown)/' -e 's/^\(\w\+\)\s\+/kernel /' | /usr/bin/inferno-collapse-perf --all | perl -pe 's/.*?;//' | sed -e 's/.*irq_exit_rcu_\[k\];/irq_exit_rcu_[k];/' | /usr/bin/inferno-flamegraph --truncate-text-right --colors=java --hash --title=spike --width=1440 --minwidth=0.005 > flamegraph.spike.svg
$ sudo /usr/share/bcc/tools/stackcount -s -K -D 1 t:skb:kfree_skb > stacks.spike.txt