Skip to content

Instantly share code, notes, and snippets.

@metahertz

metahertz/debug-faas-on-pwk.md

Created August 9, 2017 15:29
Show Gist options
  • Save metahertz/b182b80265bc581f93cd639def2c0792 to your computer and use it in GitHub Desktop.
Save metahertz/b182b80265bc581f93cd639def2c0792 to your computer and use it in GitHub Desktop.
Download ZIP
Debugging DNS 'dos' on PWD K8S triggered by faas-netes.
Raw
debug-faas-on-pwk.md

#Debug Data [node3 /]$ tcpdump -n -vv -i any udp port 53 tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes

REQUESTS: {functions_provider_url}.default.svc.cluster.local

Should work for Default K8S deployment based on value of faas-netesd. but will fail on faas-netesd.more.of.anything as seen below. 4x A's 4x AAAA's all within .1 of a second. Current total (within a second: 8 requests)


14:58:24.929991 IP (tos 0x0, ttl 64, id 30339, offset 0, flags [DF], proto UDP (17), length 109)
    10.40.0.0.38888 > 10.32.0.2.domain: [udp sum ok] 39753+ AAAA? faas-netesd.default.svc.cluster.local.default.svc.cluster.local. (81)
14:58:24.930133 IP (tos 0x0, ttl 64, id 30339, offset 0, flags [DF], proto UDP (17), length 109)
    10.40.0.0.38888 > 10.32.0.2.domain: [udp sum ok] 39753+ AAAA? faas-netesd.default.svc.cluster.local.default.svc.cluster.local. (81)
14:58:24.930151 IP (tos 0x0, ttl 64, id 30339, offset 0, flags [DF], proto UDP (17), length 109)
    10.40.0.0.38888 > 10.32.0.2.domain: [udp sum ok] 39753+ AAAA? faas-netesd.default.svc.cluster.local.default.svc.cluster.local. (81)
14:58:24.930156 IP (tos 0x0, ttl 64, id 30340, offset 0, flags [DF], proto UDP (17), length 109)
    10.40.0.0.40055 > 10.32.0.2.domain: [udp sum ok] 4884+ A? faas-netesd.default.svc.cluster.local.default.svc.cluster.local. (81)
14:58:24.930178 IP (tos 0x0, ttl 64, id 30339, offset 0, flags [DF], proto UDP (17), length 109)
    10.40.0.0.38888 > 10.32.0.2.domain: [udp sum ok] 39753+ AAAA? faas-netesd.default.svc.cluster.local.default.svc.cluster.local. (81)
14:58:24.930293 IP (tos 0x0, ttl 64, id 30340, offset 0, flags [DF], proto UDP (17), length 109)
    10.40.0.0.40055 > 10.32.0.2.domain: [udp sum ok] 4884+ A? faas-netesd.default.svc.cluster.local.default.svc.cluster.local. (81)
14:58:24.930303 IP (tos 0x0, ttl 64, id 30340, offset 0, flags [DF], proto UDP (17), length 109)
    10.40.0.0.40055 > 10.32.0.2.domain: [udp sum ok] 4884+ A? faas-netesd.default.svc.cluster.local.default.svc.cluster.local. (81)
14:58:24.930317 IP (tos 0x0, ttl 64, id 30340, offset 0, flags [DF], proto UDP (17), length 109)
    10.40.0.0.40055 > 10.32.0.2.domain: [udp sum ok] 4884+ A? faas-netesd.default.svc.cluster.local.default.svc.cluster.local. (81)

NXDOMAINS for failed requests

14:58:24.930778 IP (tos 0x0, ttl 64, id 17314, offset 0, flags [DF], proto UDP (17), length 202)
    10.32.0.2.domain > 10.40.0.0.40055: [bad udp cksum 0x1511 -> 0x267f!] 4884 NXDomain q: A? faas-netesd.default.svc.cluster.local.default.svc.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local. 1502287200 28800 7200 604800 60 (174)
14:58:24.930799 IP (tos 0x0, ttl 64, id 17314, offset 0, flags [DF], proto UDP (17), length 202)
    10.32.0.2.domain > 10.40.0.0.40055: [bad udp cksum 0x1511 -> 0x267f!] 4884 NXDomain q: A? faas-netesd.default.svc.cluster.local.default.svc.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local. 1502287200 28800 7200 604800 60 (174)
14:58:24.930803 IP (tos 0x0, ttl 64, id 17314, offset 0, flags [DF], proto UDP (17), length 202)
    10.32.0.2.domain > 10.40.0.0.40055: [bad udp cksum 0x1511 -> 0x267f!] 4884 NXDomain q: A? faas-netesd.default.svc.cluster.local.default.svc.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local. 1502287200 28800 7200 604800 60 (174)
14:58:24.930806 IP (tos 0x0, ttl 64, id 17314, offset 0, flags [DF], proto UDP (17), length 202)
    10.32.0.2.domain > 10.40.0.0.40055: [bad udp cksum 0x1511 -> 0x267f!] 4884 NXDomain q: A? faas-netesd.default.svc.cluster.local.default.svc.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local. 1502287200 28800 7200 604800 60 (174)
14:58:24.930862 IP (tos 0x0, ttl 64, id 17315, offset 0, flags [DF], proto UDP (17), length 202)
    10.32.0.2.domain > 10.40.0.0.38888: [bad udp cksum 0x1511 -> 0x87d8!] 39753 NXDomain q: AAAA? faas-netesd.default.svc.cluster.local.default.svc.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local. 1502287200 28800 7200 604800 60 (174)
14:58:24.930871 IP (tos 0x0, ttl 64, id 17315, offset 0, flags [DF], proto UDP (17), length 202)
    10.32.0.2.domain > 10.40.0.0.38888: [bad udp cksum 0x1511 -> 0x87d8!] 39753 NXDomain q: AAAA? faas-netesd.default.svc.cluster.local.default.svc.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local. 1502287200 28800 7200 604800 60 (174)
14:58:24.930874 IP (tos 0x0, ttl 64, id 17315, offset 0, flags [DF], proto UDP (17), length 202)
    10.32.0.2.domain > 10.40.0.0.38888: [bad udp cksum 0x1511 -> 0x87d8!] 39753 NXDomain q: AAAA? faas-netesd.default.svc.cluster.local.default.svc.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local. 1502287200 28800 7200 604800 60 (174)
14:58:24.930876 IP (tos 0x0, ttl 64, id 17315, offset 0, flags [DF], proto UDP (17), length 202)
    10.32.0.2.domain > 10.40.0.0.38888: [bad udp cksum 0x1511 -> 0x87d8!] 39753 NXDomain q: AAAA? faas-netesd.default.svc.cluster.local.default.svc.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local. 1502287200 28800 7200 604800 60 (174)
14:58:24.930895 IP (tos 0x0, ttl 64, id 17314, offset 0, flags [DF], proto UDP (17), length 202)
    10.32.0.2.domain > 10.40.0.0.40055: [udp sum ok] 4884 NXDomain q: A? faas-netesd.default.svc.cluster.local.default.svc.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local. 1502287200 28800 7200 604800 60 (174)
14:58:24.931016 IP (tos 0x0, ttl 64, id 17315, offset 0, flags [DF], proto UDP (17), length 202)
    10.32.0.2.domain > 10.40.0.0.38888: [udp sum ok] 39753 NXDomain q: AAAA? faas-netesd.default.svc.cluster.local.default.svc.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local. 1502287200 28800 7200 604800 60 (174)

REQUESTS: {functions_provider_url}.svc.cluster.local.

Should work for Default K8S deployment based on value of faas-netesd.default. This is the default config in faas-netes repo faas.yml 4x A's 4x AAAA's all within .1 of a second. Current total (within a second: 16 requests)

14:58:24.931521 IP (tos 0x0, ttl 64, id 30341, offset 0, flags [DF], proto UDP (17), length 101)
    10.40.0.0.49415 > 10.32.0.2.domain: [bad udp cksum 0x14ac -> 0x9fd6!] 49522+ AAAA? faas-netesd.default.svc.cluster.local.svc.cluster.local. (73)
14:58:24.931527 IP (tos 0x0, ttl 64, id 30342, offset 0, flags [DF], proto UDP (17), length 101)
    10.40.0.0.44459 > 10.32.0.2.domain: [bad udp cksum 0x14ac -> 0x5830!] 14197+ A? faas-netesd.default.svc.cluster.local.svc.cluster.local. (73)
14:58:24.931532 IP (tos 0x0, ttl 64, id 30341, offset 0, flags [DF], proto UDP (17), length 101)
    10.40.0.0.49415 > 10.32.0.2.domain: [bad udp cksum 0x14ac -> 0x9fd6!] 49522+ AAAA? faas-netesd.default.svc.cluster.local.svc.cluster.local. (73)
14:58:24.931534 IP (tos 0x0, ttl 64, id 30341, offset 0, flags [DF], proto UDP (17), length 101)
    10.40.0.0.49415 > 10.32.0.2.domain: [bad udp cksum 0x14ac -> 0x9fd6!] 49522+ AAAA? faas-netesd.default.svc.cluster.local.svc.cluster.local. (73)
14:58:24.931535 IP (tos 0x0, ttl 64, id 30342, offset 0, flags [DF], proto UDP (17), length 101)
    10.40.0.0.44459 > 10.32.0.2.domain: [bad udp cksum 0x14ac -> 0x5830!] 14197+ A? faas-netesd.default.svc.cluster.local.svc.cluster.local. (73)
14:58:24.931536 IP (tos 0x0, ttl 64, id 30342, offset 0, flags [DF], proto UDP (17), length 101)
    10.40.0.0.44459 > 10.32.0.2.domain: [bad udp cksum 0x14ac -> 0x5830!] 14197+ A? faas-netesd.default.svc.cluster.local.svc.cluster.local. (73)
14:58:24.931544 IP (tos 0x0, ttl 64, id 30341, offset 0, flags [DF], proto UDP (17), length 101)
    10.40.0.0.49415 > 10.32.0.2.domain: [bad udp cksum 0x14ac -> 0x9fd6!] 49522+ AAAA? faas-netesd.default.svc.cluster.local.svc.cluster.local. (73)
14:58:24.931550 IP (tos 0x0, ttl 64, id 30342, offset 0, flags [DF], proto UDP (17), length 101)
    10.40.0.0.44459 > 10.32.0.2.domain: [bad udp cksum 0x14ac -> 0x5830!] 14197+ A? faas-netesd.default.svc.cluster.local.svc.cluster.local. (73)

NXDOMAINS for failed requests

14:58:24.931859 IP (tos 0x0, ttl 64, id 17316, offset 0, flags [DF], proto UDP (17), length 194)
    10.32.0.2.domain > 10.40.0.0.44459: [bad udp cksum 0x1509 -> 0x2bae!] 14197 NXDomain q: A? faas-netesd.default.svc.cluster.local.svc.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local.1502287200 28800 7200 604800 60 (166)
14:58:24.931872 IP (tos 0x0, ttl 64, id 17316, offset 0, flags [DF], proto UDP (17), length 194)
    10.32.0.2.domain > 10.40.0.0.44459: [bad udp cksum 0x1509 -> 0x2bae!] 14197 NXDomain q: A? faas-netesd.default.svc.cluster.local.svc.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local.1502287200 28800 7200 604800 60 (166)
14:58:24.931874 IP (tos 0x0, ttl 64, id 17316, offset 0, flags [DF], proto UDP (17), length 194)
    10.32.0.2.domain > 10.40.0.0.44459: [bad udp cksum 0x1509 -> 0x2bae!] 14197 NXDomain q: A? faas-netesd.default.svc.cluster.local.svc.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local.1502287200 28800 7200 604800 60 (166)
14:58:24.931876 IP (tos 0x0, ttl 64, id 17316, offset 0, flags [DF], proto UDP (17), length 194)
    10.32.0.2.domain > 10.40.0.0.44459: [bad udp cksum 0x1509 -> 0x2bae!] 14197 NXDomain q: A? faas-netesd.default.svc.cluster.local.svc.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local.1502287200 28800 7200 604800 60 (166)
14:58:24.931879 IP (tos 0x0, ttl 64, id 17316, offset 0, flags [DF], proto UDP (17), length 194)
    10.32.0.2.domain > 10.40.0.0.44459: [bad udp cksum 0x1509 -> 0x2bae!] 14197 NXDomain q: A? faas-netesd.default.svc.cluster.local.svc.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local.1502287200 28800 7200 604800 60 (166)
14:58:24.931979 IP (tos 0x0, ttl 64, id 17317, offset 0, flags [DF], proto UDP (17), length 194)
    10.32.0.2.domain > 10.40.0.0.49415: [bad udp cksum 0x1509 -> 0x7354!] 49522 NXDomain q: AAAA? faas-netesd.default.svc.cluster.local.svc.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local. 1502287200 28800 7200 604800 60 (166)
14:58:24.931986 IP (tos 0x0, ttl 64, id 17317, offset 0, flags [DF], proto UDP (17), length 194)
    10.32.0.2.domain > 10.40.0.0.49415: [bad udp cksum 0x1509 -> 0x7354!] 49522 NXDomain q: AAAA? faas-netesd.default.svc.cluster.local.svc.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local. 1502287200 28800 7200 604800 60 (166)
14:58:24.931990 IP (tos 0x0, ttl 64, id 17317, offset 0, flags [DF], proto UDP (17), length 194)
    10.32.0.2.domain > 10.40.0.0.49415: [bad udp cksum 0x1509 -> 0x7354!] 49522 NXDomain q: AAAA? faas-netesd.default.svc.cluster.local.svc.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local. 1502287200 28800 7200 604800 60 (166)
14:58:24.931991 IP (tos 0x0, ttl 64, id 17317, offset 0, flags [DF], proto UDP (17), length 194)
    10.32.0.2.domain > 10.40.0.0.49415: [bad udp cksum 0x1509 -> 0x7354!] 49522 NXDomain q: AAAA? faas-netesd.default.svc.cluster.local.svc.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local. 1502287200 28800 7200 604800 60 (166)
14:58:24.931994 IP (tos 0x0, ttl 64, id 17317, offset 0, flags [DF], proto UDP (17), length 194)
    10.32.0.2.domain > 10.40.0.0.49415: [bad udp cksum 0x1509 -> 0x7354!] 49522 NXDomain q: AAAA? faas-netesd.default.svc.cluster.local.svc.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local. 1502287200 28800 7200 604800 60 (166)

REQUESTS: {functions_provider_url}.cluster.local.

Should work for Default K8S deployment based on value of faas-netesd.default.svc. 4x A's 4x AAAA's all within .1 of a second. Current total (within a second: 24 requests)

14:58:24.932282 IP (tos 0x0, ttl 64, id 30343, offset 0, flags [DF], proto UDP (17), length 97)
    10.40.0.0.43357 > 10.32.0.2.domain: [bad udp cksum 0x14a8 -> 0x4a21!] 43184+ AAAA? faas-netesd.default.svc.cluster.local.cluster.local. (69)
14:58:24.932285 IP (tos 0x0, ttl 64, id 30343, offset 0, flags [DF], proto UDP (17), length 97)
    10.40.0.0.43357 > 10.32.0.2.domain: [bad udp cksum 0x14a8 -> 0x4a21!] 43184+ AAAA? faas-netesd.default.svc.cluster.local.cluster.local. (69)
14:58:24.932286 IP (tos 0x0, ttl 64, id 30343, offset 0, flags [DF], proto UDP (17), length 97)
    10.40.0.0.43357 > 10.32.0.2.domain: [bad udp cksum 0x14a8 -> 0x4a21!] 43184+ AAAA? faas-netesd.default.svc.cluster.local.cluster.local. (69)
14:58:24.932300 IP (tos 0x0, ttl 64, id 30343, offset 0, flags [DF], proto UDP (17), length 97)
    10.40.0.0.43357 > 10.32.0.2.domain: [bad udp cksum 0x14a8 -> 0x4a21!] 43184+ AAAA? faas-netesd.default.svc.cluster.local.cluster.local. (69)
14:58:24.932324 IP (tos 0x0, ttl 64, id 30344, offset 0, flags [DF], proto UDP (17), length 97)
    10.40.0.0.59903 > 10.32.0.2.domain: [bad udp cksum 0x14a8 -> 0xa677!] 9912+ A? faas-netesd.default.svc.cluster.local.cluster.local. (69)
14:58:24.932327 IP (tos 0x0, ttl 64, id 30344, offset 0, flags [DF], proto UDP (17), length 97)
    10.40.0.0.59903 > 10.32.0.2.domain: [bad udp cksum 0x14a8 -> 0xa677!] 9912+ A? faas-netesd.default.svc.cluster.local.cluster.local. (69)
14:58:24.932328 IP (tos 0x0, ttl 64, id 30344, offset 0, flags [DF], proto UDP (17), length 97)
    10.40.0.0.59903 > 10.32.0.2.domain: [bad udp cksum 0x14a8 -> 0xa677!] 9912+ A? faas-netesd.default.svc.cluster.local.cluster.local. (69)
14:58:24.932342 IP (tos 0x0, ttl 64, id 30344, offset 0, flags [DF], proto UDP (17), length 97)
    10.40.0.0.59903 > 10.32.0.2.domain: [bad udp cksum 0x14a8 -> 0xa677!] 9912+ A? faas-netesd.default.svc.cluster.local.cluster.local. (69)

NXDOMAINS For failed requests.

14:58:24.932642 IP (tos 0x0, ttl 64, id 17318, offset 0, flags [DF], proto UDP (17), length 190)
    10.32.0.2.domain > 10.40.0.0.59903: [bad udp cksum 0x1505 -> 0x79f5!] 9912 NXDomain q: A? faas-netesd.default.svc.cluster.local.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local. 1502287200 28800 7200 604800 60 (162)
14:58:24.932662 IP (tos 0x0, ttl 64, id 17318, offset 0, flags [DF], proto UDP (17), length 190)
    10.32.0.2.domain > 10.40.0.0.59903: [bad udp cksum 0x1505 -> 0x79f5!] 9912 NXDomain q: A? faas-netesd.default.svc.cluster.local.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local. 1502287200 28800 7200 604800 60 (162)
14:58:24.932666 IP (tos 0x0, ttl 64, id 17318, offset 0, flags [DF], proto UDP (17), length 190)
    10.32.0.2.domain > 10.40.0.0.59903: [bad udp cksum 0x1505 -> 0x79f5!] 9912 NXDomain q: A? faas-netesd.default.svc.cluster.local.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local. 1502287200 28800 7200 604800 60 (162)
14:58:24.932668 IP (tos 0x0, ttl 64, id 17318, offset 0, flags [DF], proto UDP (17), length 190)
    10.32.0.2.domain > 10.40.0.0.59903: [bad udp cksum 0x1505 -> 0x79f5!] 9912 NXDomain q: A? faas-netesd.default.svc.cluster.local.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local. 1502287200 28800 7200 604800 60 (162)
14:58:24.932671 IP (tos 0x0, ttl 64, id 17318, offset 0, flags [DF], proto UDP (17), length 190)
    10.32.0.2.domain > 10.40.0.0.59903: [bad udp cksum 0x1505 -> 0x79f5!] 9912 NXDomain q: A? faas-netesd.default.svc.cluster.local.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local. 1502287200 28800 7200 604800 60 (162)
14:58:24.932775 IP (tos 0x0, ttl 64, id 17319, offset 0, flags [DF], proto UDP (17), length 190)
    10.32.0.2.domain > 10.40.0.0.43357: [bad udp cksum 0x1505 -> 0x1d9f!] 43184 NXDomain q: AAAA? faas-netesd.default.svc.cluster.local.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local. 1502287200 28800 7200 604800 60 (162)
14:58:24.932782 IP (tos 0x0, ttl 64, id 17319, offset 0, flags [DF], proto UDP (17), length 190)
    10.32.0.2.domain > 10.40.0.0.43357: [bad udp cksum 0x1505 -> 0x1d9f!] 43184 NXDomain q: AAAA? faas-netesd.default.svc.cluster.local.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local. 1502287200 28800 7200 604800 60 (162)
14:58:24.932785 IP (tos 0x0, ttl 64, id 17319, offset 0, flags [DF], proto UDP (17), length 190)
    10.32.0.2.domain > 10.40.0.0.43357: [bad udp cksum 0x1505 -> 0x1d9f!] 43184 NXDomain q: AAAA? faas-netesd.default.svc.cluster.local.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local. 1502287200 28800 7200 604800 60 (162)
14:58:24.932787 IP (tos 0x0, ttl 64, id 17319, offset 0, flags [DF], proto UDP (17), length 190)
    10.32.0.2.domain > 10.40.0.0.43357: [bad udp cksum 0x1505 -> 0x1d9f!] 43184 NXDomain q: AAAA? faas-netesd.default.svc.cluster.local.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local. 1502287200 28800 7200 604800 60 (162)
14:58:24.932790 IP (tos 0x0, ttl 64, id 17319, offset 0, flags [DF], proto UDP (17), length 190)
    10.32.0.2.domain > 10.40.0.0.43357: [bad udp cksum 0x1505 -> 0x1d9f!] 43184 NXDomain q: AAAA? faas-netesd.default.svc.cluster.local.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local. 1502287200 28800 7200 604800 60 (162)

REQUESTS: {functions_provider_url}.ec2.internal.

This is clearly for the AWS internal DNS usecase. 4x A's 4x AAAA's all within .1 of a second. Current total (within a second: 32 requests)

14:58:24.933102 IP (tos 0x0, ttl 64, id 30345, offset 0, flags [DF], proto UDP (17), length 96)
    10.40.0.0.51741 > 10.32.0.2.domain: [bad udp cksum 0x14a7 -> 0x760a!] 32414+ AAAA? faas-netesd.default.svc.cluster.local.ec2.internal. (68)
14:58:24.933115 IP (tos 0x0, ttl 64, id 30345, offset 0, flags [DF], proto UDP (17), length 96)
    10.40.0.0.51741 > 10.32.0.2.domain: [bad udp cksum 0x14a7 -> 0x760a!] 32414+ AAAA? faas-netesd.default.svc.cluster.local.ec2.internal. (68)
14:58:24.933118 IP (tos 0x0, ttl 64, id 30345, offset 0, flags [DF], proto UDP (17), length 96)
    10.40.0.0.51741 > 10.32.0.2.domain: [bad udp cksum 0x14a7 -> 0x760a!] 32414+ AAAA? faas-netesd.default.svc.cluster.local.ec2.internal. (68)
14:58:24.933135 IP (tos 0x0, ttl 64, id 30345, offset 0, flags [DF], proto UDP (17), length 96)
    10.40.0.0.51741 > 10.32.0.2.domain: [bad udp cksum 0x14a7 -> 0x760a!] 32414+ AAAA? faas-netesd.default.svc.cluster.local.ec2.internal. (68)
14:58:24.933247 IP (tos 0x0, ttl 64, id 30346, offset 0, flags [DF], proto UDP (17), length 96)
    10.40.0.0.38266 > 10.32.0.2.domain: [bad udp cksum 0x14a7 -> 0x8f5b!] 39435+ A? faas-netesd.default.svc.cluster.local.ec2.internal. (68)
14:58:24.933257 IP (tos 0x0, ttl 64, id 30346, offset 0, flags [DF], proto UDP (17), length 96)
    10.40.0.0.38266 > 10.32.0.2.domain: [bad udp cksum 0x14a7 -> 0x8f5b!] 39435+ A? faas-netesd.default.svc.cluster.local.ec2.internal. (68)
14:58:24.933260 IP (tos 0x0, ttl 64, id 30346, offset 0, flags [DF], proto UDP (17), length 96)
    10.40.0.0.38266 > 10.32.0.2.domain: [bad udp cksum 0x14a7 -> 0x8f5b!] 39435+ A? faas-netesd.default.svc.cluster.local.ec2.internal. (68)
14:58:24.933275 IP (tos 0x0, ttl 64, id 30346, offset 0, flags [DF], proto UDP (17), length 96)
    10.40.0.0.38266 > 10.32.0.2.domain: [bad udp cksum 0x14a7 -> 0x8f5b!] 39435+ A? faas-netesd.default.svc.cluster.local.ec2.internal. (68)

More requests after 5 seconds as we have no response from the DNS server

14:58:29.933355 IP (tos 0x0, ttl 64, id 31073, offset 0, flags [DF], proto UDP (17), length 96)
    10.40.0.0.33976 > 10.32.0.2.domain: [bad udp cksum 0x14a7 -> 0xa5de!] 37935+ AAAA? faas-netesd.default.svc.cluster.local.ec2.internal. (68)
14:58:29.933355 IP (tos 0x0, ttl 64, id 31072, offset 0, flags [DF], proto UDP (17), length 96)
    10.40.0.0.42730 > 10.32.0.2.domain: [bad udp cksum 0x14a7 -> 0xf9a2!] 7764+ A? faas-netesd.default.svc.cluster.local.ec2.internal. (68)
14:58:29.933371 IP (tos 0x0, ttl 64, id 31073, offset 0, flags [DF], proto UDP (17), length 96)
    10.40.0.0.33976 > 10.32.0.2.domain: [bad udp cksum 0x14a7 -> 0xa5de!] 37935+ AAAA? faas-netesd.default.svc.cluster.local.ec2.internal. (68)
14:58:29.933371 IP (tos 0x0, ttl 64, id 31072, offset 0, flags [DF], proto UDP (17), length 96)
    10.40.0.0.42730 > 10.32.0.2.domain: [bad udp cksum 0x14a7 -> 0xf9a2!] 7764+ A? faas-netesd.default.svc.cluster.local.ec2.internal. (68)
14:58:29.933373 IP (tos 0x0, ttl 64, id 31073, offset 0, flags [DF], proto UDP (17), length 96)
    10.40.0.0.33976 > 10.32.0.2.domain: [bad udp cksum 0x14a7 -> 0xa5de!] 37935+ AAAA? faas-netesd.default.svc.cluster.local.ec2.internal. (68)
14:58:29.933373 IP (tos 0x0, ttl 64, id 31072, offset 0, flags [DF], proto UDP (17), length 96)
    10.40.0.0.42730 > 10.32.0.2.domain: [bad udp cksum 0x14a7 -> 0xf9a2!] 7764+ A? faas-netesd.default.svc.cluster.local.ec2.internal. (68)
14:58:29.933392 IP (tos 0x0, ttl 64, id 31073, offset 0, flags [DF], proto UDP (17), length 96)
    10.40.0.0.33976 > 10.32.0.2.domain: [bad udp cksum 0x14a7 -> 0xa5de!] 37935+ AAAA? faas-netesd.default.svc.cluster.local.ec2.internal. (68)
14:58:29.933392 IP (tos 0x0, ttl 64, id 31072, offset 0, flags [DF], proto UDP (17), length 96)
    10.40.0.0.42730 > 10.32.0.2.domain: [bad udp cksum 0x14a7 -> 0xf9a2!] 7764+ A? faas-netesd.default.svc.cluster.local.ec2.internal. (68)

More requests 5 seconds later as we havent heard anything back from our .cluster.local requests.

14:58:34.933591 IP (tos 0x0, ttl 64, id 31919, offset 0, flags [DF], proto UDP (17), length 83)
    10.40.0.0.60417 > 10.32.0.2.domain: [bad udp cksum 0x149a -> 0x5f7f!] 36549+ A? faas-netesd.default.svc.cluster.local. (55)
14:58:34.933592 IP (tos 0x0, ttl 64, id 31920, offset 0, flags [DF], proto UDP (17), length 83)
    10.40.0.0.48774 > 10.32.0.2.domain: [bad udp cksum 0x149a -> 0x81b6!] 32521+ AAAA? faas-netesd.default.svc.cluster.local. (55)
14:58:34.933602 IP (tos 0x0, ttl 64, id 31919, offset 0, flags [DF], proto UDP (17), length 83)
    10.40.0.0.60417 > 10.32.0.2.domain: [bad udp cksum 0x149a -> 0x5f7f!] 36549+ A? faas-netesd.default.svc.cluster.local. (55)
14:58:34.933603 IP (tos 0x0, ttl 64, id 31920, offset 0, flags [DF], proto UDP (17), length 83)
    10.40.0.0.48774 > 10.32.0.2.domain: [bad udp cksum 0x149a -> 0x81b6!] 32521+ AAAA? faas-netesd.default.svc.cluster.local. (55)
14:58:34.933604 IP (tos 0x0, ttl 64, id 31919, offset 0, flags [DF], proto UDP (17), length 83)
    10.40.0.0.60417 > 10.32.0.2.domain: [bad udp cksum 0x149a -> 0x5f7f!] 36549+ A? faas-netesd.default.svc.cluster.local. (55)
14:58:34.933606 IP (tos 0x0, ttl 64, id 31920, offset 0, flags [DF], proto UDP (17), length 83)
    10.40.0.0.48774 > 10.32.0.2.domain: [bad udp cksum 0x149a -> 0x81b6!] 32521+ AAAA? faas-netesd.default.svc.cluster.local. (55)
14:58:34.933618 IP (tos 0x0, ttl 64, id 31919, offset 0, flags [DF], proto UDP (17), length 83)
    10.40.0.0.60417 > 10.32.0.2.domain: [bad udp cksum 0x149a -> 0x5f7f!] 36549+ A? faas-netesd.default.svc.cluster.local. (55)
14:58:34.933623 IP (tos 0x0, ttl 64, id 31920, offset 0, flags [DF], proto UDP (17), length 83)
    10.40.0.0.48774 > 10.32.0.2.domain: [bad udp cksum 0x149a -> 0x81b6!] 32521+ AAAA? faas-netesd.default.svc.cluster.local. (55)

## Responses, we dont get these back for 10 seconds since the request. (DNSMASQ failed health check and died/restarted as it had reached it's 150 concurrent transactions default) (and wasnt accepting any more queries. Eachtime the gateway is accessed, 32X queries, this would only take a couple of refreshes to trigger.) Plus, DNS requests for external sources time out (not configured for external resolution, so clients looking for yum / apt updates / ntp etc will add to this number).

14:58:34.948401 IP (tos 0x0, ttl 64, id 18478, offset 0, flags [DF], proto UDP (17), length 137)
    10.32.0.2.domain > 10.40.0.0.48774: [bad udp cksum 0x14d0 -> 0x1e7e!] 32521* q: AAAA? faas-netesd.default.svc.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local. 1502287200 28800 7200 604800 60 (109)
14:58:34.948422 IP (tos 0x0, ttl 64, id 18478, offset 0, flags [DF], proto UDP (17), length 137)
    10.32.0.2.domain > 10.40.0.0.48774: [bad udp cksum 0x14d0 -> 0x1e7e!] 32521* q: AAAA? faas-netesd.default.svc.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local. 1502287200 28800 7200 604800 60 (109)
14:58:34.948427 IP (tos 0x0, ttl 64, id 18478, offset 0, flags [DF], proto UDP (17), length 137)
    10.32.0.2.domain > 10.40.0.0.48774: [bad udp cksum 0x14d0 -> 0x1e7e!] 32521* q: AAAA? faas-netesd.default.svc.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local. 1502287200 28800 7200 604800 60 (109)
14:58:34.948429 IP (tos 0x0, ttl 64, id 18478, offset 0, flags [DF], proto UDP (17), length 137)
    10.32.0.2.domain > 10.40.0.0.48774: [bad udp cksum 0x14d0 -> 0x1e7e!] 32521* q: AAAA? faas-netesd.default.svc.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local. 1502287200 28800 7200 604800 60 (109)
14:58:34.948434 IP (tos 0x0, ttl 64, id 18478, offset 0, flags [DF], proto UDP (17), length 137)
    10.32.0.2.domain > 10.40.0.0.48774: [bad udp cksum 0x14d0 -> 0x1e7e!] 32521* q: AAAA? faas-netesd.default.svc.cluster.local. 0/1/0 ns: cluster.local. SOA ns.dns.cluster.local. hostmaster.cluster.local. 1502287200 28800 7200 604800 60 (109)
14:58:34.948594 IP (tos 0x0, ttl 64, id 18479, offset 0, flags [DF], proto UDP (17), length 99)
    10.32.0.2.domain > 10.40.0.0.60417: [bad udp cksum 0x14aa -> 0x8e51!] 36549* q: A? faas-netesd.default.svc.cluster.local. 1/0/0 faas-netesd.default.svc.cluster.local. A 10.100.193.183 (71)
14:58:34.948609 IP (tos 0x0, ttl 64, id 18479, offset 0, flags [DF], proto UDP (17), length 99)
    10.32.0.2.domain > 10.40.0.0.60417: [bad udp cksum 0x14aa -> 0x8e51!] 36549* q: A? faas-netesd.default.svc.cluster.local. 1/0/0 faas-netesd.default.svc.cluster.local. A 10.100.193.183 (71)
14:58:34.948611 IP (tos 0x0, ttl 64, id 18479, offset 0, flags [DF], proto UDP (17), length 99)
    10.32.0.2.domain > 10.40.0.0.60417: [bad udp cksum 0x14aa -> 0x8e51!] 36549* q: A? faas-netesd.default.svc.cluster.local. 1/0/0 faas-netesd.default.svc.cluster.local. A 10.100.193.183 (71)
14:58:34.948613 IP (tos 0x0, ttl 64, id 18479, offset 0, flags [DF], proto UDP (17), length 99)
    10.32.0.2.domain > 10.40.0.0.60417: [bad udp cksum 0x14aa -> 0x8e51!] 36549* q: A? faas-netesd.default.svc.cluster.local. 1/0/0 faas-netesd.default.svc.cluster.local. A 10.100.193.183 (71)
14:58:34.948616 IP (tos 0x0, ttl 64, id 18479, offset 0, flags [DF], proto UDP (17), length 99)
    10.32.0.2.domain > 10.40.0.0.60417: [bad udp cksum 0x14aa -> 0x8e51!] 36549* q: A? faas-netesd.default.svc.cluster.local. 1/0/0 faas-netesd.default.svc.cluster.local. A 10.100.193.183 (71)

Solution test

IF (Testing this theory) we configure faas.yml with a function_provider_url of just faas-netesd, the first set of DNS requests will append default.svc.cluster.local and resolve correctly. this will only have been 8 requests, and as long as the code then doesnt try the other lookups when it's received a sucessful response, this should cut down the load. That said, max in flight of 150 requests for a DNS server serving a k8s cluster probably needs increasing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment