Cqshinn enixdark

Design and Implementation of K8s Services Proxy using eBpf

Goals and Priorities

Build an eBpf based implementation of Kubernetes Services (ClusterIP, NodePort, LoadBalancer) to replace Kube-proxy/ iptables and CNI based implementations of Kubernetes services.
The goal is not "use as much eBpf" as possible but rather to use eBpf selectively and opportunistically and also to leverage standard kernel datapaths as much as possible unless there is a good reason to do otherwise.
Since iptables packages are being deprecated in the Linux kernel and RHEL, it is necessary to have an implementation of kube-proxy that does not depend on iptables. See iptables deprecation
Primary design requirement is to retain end user experience for stability and debuggability when replacing the kube-proxy/ iptables based datapath. This requirement is more important that flat out data plane performanc

docker-compose.yml

version: "3"

networks:
  loki:

Interface

Ref: https://elixir.bootlin.com/linux/v2.6.39/source/kernel/trace/trace.h#L246

Examples

nop tracer

struct tracer nop_trace __read_mostly =

# First rule SNAT the packet that comes out of the gw node
-A CILIUM_POST_nat -s 10.11.1.0/24 ! -d 10.11.0.0/16 ! -o cilium_+ -m comment --comment "cilium masquerade non-cluster" -j MASQUERADE

-A CILIUM_POST_nat ! -o cilium_host -m comment --comment "exclude non-cilium_host traffic from masquerade" -j RETURN
-A CILIUM_POST_nat -m mark --mark 0xa00/0xe00 -m comment --comment "exclude proxy return traffic from masquarade" -j ACCEPT
-A CILIUM_POST_nat ! -s 10.11.1.253/32 ! -d 10.11.1.0/24 -o cilium_host -m comment --comment "cilium host->cluster masquerade" -j SNAT --to-source 10.11.1.253
-A CILIUM_POST_nat -s 127.0.0.1/32 -o cilium_host -m comment --comment "cilium host->cluster from 127.0.0.1 masquerade" -j SNAT --to-source 10.11.1.253
-A CILIUM_POST_nat -o cilium_host -m mark --mark 0xf00/0xf00 -m conntrack --ctstate DNAT -m comment --comment "hairpin traffic that originated from a local pod" -j SNAT --to-source 10.11.1.253

Node/Pod	IP
Source Pod	10.11.1.103
Source Node	192.168.11.12
Gateway Node	192.168.11.11
External Service	192.168.11.13

iptables

Added the following on both source and gateway node. Egress gateway datapath still works.

vagrant@k8s1:~/go/src/github.com/cilium/cilium$ sudo tcpdump -i enp0s8 icmp6 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp0s8, link-type EN10MB (Ethernet), capture size 262144 bytes




18:11:10.397859 IP6 k8s2 > ff02::1:ff00:2c69: ICMP6, neighbor solicitation, who has f00d::a0f:0:0:2c69, length 32
18:11:11.478191 IP6 k8s2 > ff02::1:ff00:2c69: ICMP6, neighbor solicitation, who has f00d::a0f:0:0:2c69, length 32

	require 'elasticsearch/rails/tasks/import'

	namespace :elasticsearch do
	desc 'Rebuild index with zerodown time'
	task rebuild: :environment do
	size = (ENV['size'].presence \|\| 1000).to_i
	model = ENV['model'].to_s.classify.constantize
	delete_old_index = ENV['delete'].present?
	es = model.__elasticsearch__

	from datetime import datetime
	from elasticsearch_dsl import (
	DocType,
	String,
	Integer,
	Float
	)


	def _suffix():

	对比 Apache APISIX 和 envoy 的一些性能参数：网关延迟、QPS 和是否线性扩展
	系统环境：微软云 Linux (ubuntu 18.04)， Standard D13 v2 (8 vcpus, 56 GiB memory)
	压力测试客户端：wrk