http://kernelshark.org/Documentation.html sudo dnf install trace-cmd kernelshark sudo trace-cmd record -l "ip_" -l "icmp" -p function_graph ping 8.8.8.8 -c 1 kernelshark

Bcc/ebpf

why not just pcap/tcpdump https://sdcast.ksdaemon.ru/2017/07/sdcast-58/ Павел Одинцов

xdp

https://www.youtube.com/watch?v=ZdVpKx6Wmc8 1 min

https://www.youtube.com/watch?v=GsMs3n8CB6g 15 min

https://www.youtube.com/watch?v=bj3qdEDbCD4 45 min Velocity 2017: Performance Analysis Superpowers with Linux eBPF @14:08 fast new way

https://www.youtube.com/watch?v=4SiWL5tULnQ More on ebpf

https://en.wikipedia.org/wiki/IPv4

sudo dnf install bcc sudo /usr/share/bcc/tools/trace -Ilinux/skbuff.h
'ip_send_skb(struct net *netx, struct sk_buff *skb) "ip dst: %d %d %d %d", *(skb->data+16), *(skb->data+17), *(skb->data+18), *(skb->data+19)'

sudo /usr/share/bcc/tools/execsnoop

ls and echo difference - type hash which commands

Get pids

sudo /usr/share/bcc/tools/tcplife

curl localhost:8080

Part I

https://www.youtube.com/watch?v=FJW8nGV4jxY

Part II

https://www.youtube.com/watch?v=zrr2nUln9Kk

Slides

https://www.slideshare.net/brendangregg/velocity-2015-linux-perf-tools

Perf labs

https://github.com/brendangregg/perf-labs/tree/master/src

Perf

https://perf.wiki.kernel.org/index.php/Tutorial

sudo dnf install git sudo dnf install perf sudo dnf install perl-open.noarch git clone https://github.com/brendangregg/FlameGraph # or download it from github cd FlameGraph sudo perf record -F 9999 -a -g -- git clone https://github.com/brendangregg/FlameGraph sudo perf script | ./stackcollapse-perf.pl > out.perf-folded ./flamegraph.pl out.perf-folded > perf-kernel.svg python3 -m http.server sudo rm -rf FlameGraph/ out.perf-folded perf-kernel.svg perf.data

Рассказать о стеках

sudo perf top

#sudo perf record -g -a -e skb:kfree_skb

normal is consume_skb

#sudo perf script

lukego/blog#13

https://github.com/pavel-odintsov/drop_watch

ARP

sudo trace-cmd record -l "arp" -l "icmp_rcv" -p function_graph python3 -m http.server sudo ip -s -s neigh flush all

Packet Fragmentation/Defragmentation

sudo trace-cmd record -l "ip_fragment*" -p function_graph python3 -m http.server ping 192.168.122.1 -s 2000 -c 5

Forwarding & Local Delivery

ICMP

ICMP Header Применение ICMP Приём и передача ICMP сообщений

L4

Available L4 Protocols L4 Protocol Registration L3 to L4 Delivery: ip_local_deliver_finish

Use kfree_skb to detect failures

vm_ip=192.168.122.184 curl $vm_ip sudo /usr/share/bcc/tools/trace -Ilinux/skbuff.h 'kfree_skb(struct sk_buff skb) ((skb->data+23)==80) "ip dst: %d %d %d %d %d", *(skb->data+16), *(skb->data+17), *(skb->data+18), *(skb->data+18), *(skb->data+23)'

https://sdcast.ksdaemon.ru/2019/01/sdcast-97/ Александр Тоболь

https://www.youtube.com/watch?v=L5msC6pswQs Debugging Linux Issues with eBPF

sudo trace-cmd record -l "ip_fragment" -p function_graph python3 -m http.server

https://training.linuxfoundation.org/training/a-beginners-guide-to-linux-kernel-development-lfd103/

https://lwn.net/Kernel/LDD3/

lukego/blog#13

http://brendangregg.com/ebpf.html#bcc

http://man7.org/linux/man-pages/man2/syscalls.2.html https://syscalls.kernelgrok.com/ https://linux-kernel-labs.github.io/master/lectures/syscalls.html

mmalchuk/file_raw_version.md

Need kernel to match kernel-devel

sudo reboot

BOOT UP

Program memory

Syscalls

Stack

All tools

syscall vs kernel function

Strace

How strace works

Trace-cmd

tell about ftrace

Flamescope? https://www.youtube.com/watch?v=03EC8uA30Pw

5 key issues and how netflix cloud is architected to solve them

1. autoscaling (spinnaker)

2. bad push (rollback)

3. instance failure (hystrix timeouts)

4. region failure (zuul 2 reroute traffic)

5. overlooked issue (chaos engineering)

why do root scope analysis?

high latency

growth

upgrades

methodologies http://www.brendangregg.com/methodology.html

red (RATE ERRORS DURATION) USE (utilization saturation errors)

Anti methods

bad instance anti method(just delete)

checklist methodology

zipkin

@43:26 from perf to bpf

@45:48 ftrace/perf/ebpf

books http://www.brendangregg.com/books.html