-
-
Save danehans/c48f070aa434f8d5eb6bd44dddc7ab76 to your computer and use it in GitHub Desktop.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ______________________________________________________________ | |
| k8s+contiv BGP peering setup | |
| Status: | |
| - able to peer with ASR1K route-reflector | |
| - able to see reflected routes in k8s netctl bgp output | |
| - NOT able to see k8s pod routes in ASR1K | |
| ------------------------------------------------------- | |
| BGP peering test setup: | |
| ASR1K | |
| bgp AS 64511 | |
| \ | |
| vrf harmony | |
| route-reflector | |
| / | |
| +------+-------+.................+ | |
| | | | | | |
| node1 node2 node3 openstack peer | |
| ------------ (172.16.0.200) | |
| k8s+contiv | |
| (192.168.35.x) | |
| node1 = hostname bxb-mercury-control-2 (kubemaster+minion / netmaster) | |
| node2 = hostname bxb-mercury-control-3 (minion) | |
| node3 = hostname bxb-mercury-control-1 (minion) | |
| ---------------- | |
| ASR1K config: | |
| ---------------- | |
| vrf definition harmony | |
| rd 172.16.0.1:1 | |
| ! | |
| address-family ipv4 | |
| exit-address-family | |
| ! | |
| interface GigabitEthernet0/2/0.3150 | |
| encapsulation dot1Q 3150 | |
| vrf forwarding harmony | |
| ip address 192.168.35.1 255.255.255.0 | |
| ! | |
| router bgp 64511 | |
| bgp router-id 172.16.0.1 | |
| bgp log-neighbor-changes | |
| ! | |
| address-family ipv4 vrf harmony | |
| neighbor 172.16.0.200 remote-as 64511 | |
| neighbor 172.16.0.200 activate | |
| neighbor 172.16.0.200 route-reflector-client | |
| neighbor 192.168.35.2 remote-as 64511 | |
| neighbor 192.168.35.2 activate | |
| neighbor 192.168.35.2 route-reflector-client | |
| neighbor 192.168.35.3 remote-as 64511 | |
| neighbor 192.168.35.3 activate | |
| neighbor 192.168.35.3 route-reflector-client | |
| neighbor 192.168.35.4 remote-as 64511 | |
| neighbor 192.168.35.4 activate | |
| neighbor 192.168.35.4 route-reflector-client | |
| exit-address-family | |
| ! | |
| --------------------- | |
| k8s + contiv procedure | |
| - NOTE: initially I forgot to setup the default epg after the net-create | |
| - I created epg and deleted the pod deployment and redeployed it | |
| (see the troubleshooting section II.) | |
| --------------------- | |
| git clone https://github.com/contiv/netplugin | |
| get contiv.yaml from: | |
| https://raw.githubusercontent.com/neelimamukiri/netplugin/k8s_aci_yaml/install/k8s/contiv/contiv.yaml | |
| - set netmaster IP and VLAN_IF in contiv.yaml | |
| > kubectl apply -f contiv.yaml | |
| # wait a few mins for containers to come up | |
| > netctl global set --fwd-mode routing | |
| # check global info | |
| > netctl global info | |
| # setup bgp peering to ASR1K RR (192.168.35.1) | |
| > netctl bgp create bxb-mercury-control-2 --router-ip="192.168.35.2/24" --as="64511" --neighbor-as="64511" --neighbor="192.168.35.1" | |
| # check on peering on ASR1K | |
| --- | |
| bxb-os-asr1k-2#sh ip bgp all summary | |
| For address family: VPNv4 Unicast | |
| BGP router identifier 172.16.0.1, local AS number 64511 | |
| BGP table version is 24, main routing table version 24 | |
| 1 network entries using 256 bytes of memory | |
| 1 path entries using 120 bytes of memory | |
| 1/1 BGP path/bestpath attribute entries using 264 bytes of memory | |
| 0 BGP route-map cache entries using 0 bytes of memory | |
| 0 BGP filter-list cache entries using 0 bytes of memory | |
| BGP using 640 total bytes of memory | |
| BGP activity 10/9 prefixes, 12/11 paths, scan interval 60 secs | |
| Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd | |
| 172.16.0.200 4 64511 87 92 24 0 0 00:18:20 1 | |
| 192.168.34.3 4 64511 0 0 1 0 0 6d22h Idle | |
| 192.168.34.4 4 64511 0 0 1 0 0 6d22h Idle | |
| 192.168.34.5 4 64511 0 0 1 0 0 6d22h Idle | |
| 192.168.35.2 4 64511 3 5 24 0 0 00:00:55 0 | |
| --- | |
| NOTE: 192.168.35.2 is up | |
| -------------------------- | |
| # join other minions and wait for contiv-netplugin pods to come up | |
| # setup bgp peering of other minion nodes to ASR1K RR (192.168.35.1) | |
| > netctl bgp create bxb-mercury-control-3 --router-ip="192.168.35.3/24" --as="64511" --neighbor-as="64511" --neighbor="192.168.35.1" | |
| > netctl bgp create bxb-mercury-control-1 --router-ip="192.168.35.4/24" --as="64511" --neighbor-as="64511" --neighbor="192.168.35.1" | |
| # check on peering on ASR1K | |
| --- | |
| bxb-os-asr1k-2#sh ip bgp vpnv4 all summ | |
| BGP router identifier 172.16.0.1, local AS number 64511 | |
| BGP table version is 24, main routing table version 24 | |
| 1 network entries using 256 bytes of memory | |
| 1 path entries using 120 bytes of memory | |
| 1/1 BGP path/bestpath attribute entries using 264 bytes of memory | |
| 0 BGP route-map cache entries using 0 bytes of memory | |
| 0 BGP filter-list cache entries using 0 bytes of memory | |
| BGP using 640 total bytes of memory | |
| BGP activity 10/9 prefixes, 12/11 paths, scan interval 60 secs | |
| Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd | |
| 172.16.0.200 4 64511 165 173 24 0 0 00:35:13 1 | |
| 192.168.34.3 4 64511 0 0 1 0 0 6d22h Idle | |
| 192.168.34.4 4 64511 0 0 1 0 0 6d22h Idle | |
| 192.168.34.5 4 64511 0 0 1 0 0 6d22h Idle | |
| 192.168.35.2 4 64511 37 42 24 0 0 00:17:48 0 | |
| 192.168.35.3 4 64511 3 4 24 0 0 00:00:40 0 | |
| 192.168.35.4 4 64511 2 4 24 0 0 00:00:19 0 | |
| --- | |
| NOTE: 192.168.35.2, 3, 4 are up | |
| -------------------------------- | |
| # check the VRF harmony routing table (NOTE the 192.168.2.0/24 route) | |
| bxb-os-asr1k-2#sh ip route vrf harmony | |
| Routing Table: harmony | |
| Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP | |
| D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area | |
| N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 | |
| E1 - OSPF external type 1, E2 - OSPF external type 2 | |
| i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2 | |
| ia - IS-IS inter area, * - candidate default, U - per-user static route | |
| o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP | |
| a - application route | |
| + - replicated route, % - next hop override, p - overrides from PfR | |
| Gateway of last resort is not set | |
| 172.16.0.0/16 is variably subnetted, 4 subnets, 2 masks | |
| C 172.16.0.0/24 is directly connected, GigabitEthernet0/2/0.412 | |
| L 172.16.0.1/32 is directly connected, GigabitEthernet0/2/0.412 | |
| C 172.16.1.0/24 is directly connected, GigabitEthernet0/2/0.418 | |
| L 172.16.1.1/32 is directly connected, GigabitEthernet0/2/0.418 | |
| B 192.168.2.0/24 [200/0] via 172.16.0.6, 00:23:54 | |
| 192.168.34.0/24 is variably subnetted, 2 subnets, 2 masks | |
| C 192.168.34.0/24 is directly connected, GigabitEthernet0/2/0.3260 | |
| L 192.168.34.1/32 is directly connected, GigabitEthernet0/2/0.3260 | |
| 192.168.35.0/24 is variably subnetted, 2 subnets, 2 masks | |
| C 192.168.35.0/24 is directly connected, GigabitEthernet0/2/0.3150 | |
| L 192.168.35.1/32 is directly connected, GigabitEthernet0/2/0.3150 | |
| ----------------------------------------------------- | |
| # Contiv BGP status bgp status for all the nodes | |
| ----------------------------------------------------- | |
| [root@bxb-mercury-control-2 ~]# netctl bgp inspect bxb-mercury-control-2 | |
| netctl. Inspecting bgp: bxb-mercury-control-2 | |
| { | |
| "Config": { | |
| "key": "bxb-mercury-control-2", | |
| "as": "64511", | |
| "hostname": "bxb-mercury-control-2", | |
| "neighbor": "192.168.35.1", | |
| "neighbor-as": "64511", | |
| "routerip": "192.168.35.2/24" | |
| }, | |
| "Oper": { | |
| "adminStatus": "ADMIN_STATE_UP", | |
| "neighborStatus": "established", | |
| "numRoutes": 1, | |
| "routes": [ | |
| "192.168.2.0/24" | |
| ] | |
| } | |
| } | |
| [root@bxb-mercury-control-2 ~]# netctl bgp inspect bxb-mercury-control-1 | |
| netctl. Inspecting bgp: bxb-mercury-control-1 | |
| { | |
| "Config": { | |
| "key": "bxb-mercury-control-1", | |
| "as": "64511", | |
| "hostname": "bxb-mercury-control-1", | |
| "neighbor": "192.168.35.1", | |
| "neighbor-as": "64511", | |
| "routerip": "192.168.35.4/24" | |
| }, | |
| "Oper": { | |
| "adminStatus": "ADMIN_STATE_UP", | |
| "neighborStatus": "established", | |
| "numRoutes": 1, | |
| "routes": [ | |
| "192.168.2.0/24" | |
| ] | |
| } | |
| } | |
| [root@bxb-mercury-control-2 ~]# netctl bgp inspect bxb-mercury-control-3 | |
| netctl. Inspecting bgp: bxb-mercury-control-3 | |
| { | |
| "Config": { | |
| "key": "bxb-mercury-control-3", | |
| "as": "64511", | |
| "hostname": "bxb-mercury-control-3", | |
| "neighbor": "192.168.35.1", | |
| "neighbor-as": "64511", | |
| "routerip": "192.168.35.3/24" | |
| }, | |
| "Oper": { | |
| "adminStatus": "ADMIN_STATE_UP", | |
| "neighborStatus": "established", | |
| "numRoutes": 1, | |
| "routes": [ | |
| "192.168.2.0/24" | |
| ] | |
| } | |
| } | |
| ----- | |
| NOTE: You can see the openstack 192.168.2.0/24 route was propogated from the ASR | |
| --------------------------- | |
| ______________________________________________________________ | |
| Create a default network and create a pod deployment | |
| ______________________________________________________________ | |
| # create a default-net | |
| [root@bxb-mercury-control-2 deployments]# netctl net create -t default --subnet=192.171.0.0/24 --gateway=192.171.0.254 --encap="vlan" default-net | |
| Creating network default:default-net | |
| # create a nginx pod deployment across 2 nodes (replica-set) | |
| [root@bxb-mercury-control-2 ~]# cat test/deployments/my-nginx.yaml | |
| apiVersion: extensions/v1beta1 | |
| kind: Deployment | |
| metadata: | |
| name: my-nginx | |
| spec: | |
| replicas: 2 | |
| template: | |
| metadata: | |
| labels: | |
| run: my-nginx | |
| spec: | |
| containers: | |
| - name: my-nginx | |
| image: nginx | |
| ports: | |
| - containerPort: 80 | |
| [root@bxb-mercury-control-2 deployments]# kubectl create -f /root/test/deployments/my-nginx.yaml | |
| deployment "my-nginx" created | |
| [root@bxb-mercury-control-2 deployments]# kubectl get pods -o wide | |
| NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE | |
| default my-nginx-379829228-f7w0x 0/1 ContainerCreating 0 18s <none> bxb-mercury-control-1 | |
| default my-nginx-379829228-j8b7x 0/1 ContainerCreating 0 18s <none> bxb-mercury-control-2 | |
| # ...wait until they're up... | |
| [root@bxb-mercury-control-2 deployments]# kubectl get pods -o wide | |
| NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE | |
| default my-nginx-379829228-f7w0x 1/1 Running 0 1m 192.171.0.2 bxb-mercury-control-1 | |
| default my-nginx-379829228-j8b7x 1/1 Running 0 1m 192.171.0.3 bxb-mercury-control-2 | |
| # Look for routes in the ASR1K | |
| ---- | |
| bxb-os-asr1k-2#sh ip route vrf harmony | |
| Routing Table: harmony | |
| Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP | |
| D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area | |
| N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 | |
| E1 - OSPF external type 1, E2 - OSPF external type 2 | |
| i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2 | |
| ia - IS-IS inter area, * - candidate default, U - per-user static route | |
| o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP | |
| a - application route | |
| + - replicated route, % - next hop override, p - overrides from PfR | |
| Gateway of last resort is not set | |
| 172.16.0.0/16 is variably subnetted, 4 subnets, 2 masks | |
| C 172.16.0.0/24 is directly connected, GigabitEthernet0/2/0.412 | |
| L 172.16.0.1/32 is directly connected, GigabitEthernet0/2/0.412 | |
| C 172.16.1.0/24 is directly connected, GigabitEthernet0/2/0.418 | |
| L 172.16.1.1/32 is directly connected, GigabitEthernet0/2/0.418 | |
| B 192.168.2.0/24 [200/0] via 172.16.0.6, 01:59:51 | |
| 192.168.34.0/24 is variably subnetted, 2 subnets, 2 masks | |
| C 192.168.34.0/24 is directly connected, GigabitEthernet0/2/0.3260 | |
| L 192.168.34.1/32 is directly connected, GigabitEthernet0/2/0.3260 | |
| 192.168.35.0/24 is variably subnetted, 2 subnets, 2 masks | |
| C 192.168.35.0/24 is directly connected, GigabitEthernet0/2/0.3150 | |
| L 192.168.35.1/32 is directly connected, GigabitEthernet0/2/0.3150 | |
| ---- | |
| OH NO!!! no routes for the pods 192.171.0.0/24 | |
| ----------------- | |
| # check ping between pods | |
| [root@bxb-mercury-control-2 deployments]# kubectl exec my-nginx-379829228-j8b7x -- ping -c3 192.171.0.2 | |
| PING 192.171.0.2 (192.171.0.2): 56 data bytes | |
| --- 192.171.0.2 ping statistics --- | |
| 3 packets transmitted, 0 packets received, 100% packet loss | |
| # --- NOPE! something's not right --- | |
| ________________________________________________________________ | |
| Troubleshooting | |
| ________________________________________________________________ | |
| [root@bxb-mercury-control-2 deployments]# kubectl get pods -o wide --all-namespaces | |
| NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE | |
| default my-nginx-379829228-f7w0x 1/1 Running 0 1m 192.171.0.2 bxb-mercury-control-1 | |
| default my-nginx-379829228-j8b7x 1/1 Running 0 1m 192.171.0.3 bxb-mercury-control-2 | |
| kube-system contiv-etcd-shwxq 1/1 Running 0 17h 172.19.80.22 bxb-mercury-control-2 | |
| kube-system contiv-netmaster-xd914 1/1 Running 0 17h 172.19.80.22 bxb-mercury-control-2 | |
| kube-system contiv-netplugin-4x6tl 1/1 Running 0 17h 172.19.80.22 bxb-mercury-control-2 | |
| kube-system contiv-netplugin-l5p20 1/1 Running 0 23m 172.19.80.26 bxb-mercury-control-1 | |
| kube-system contiv-netplugin-ns7m7 1/1 Running 0 23m 172.19.80.24 bxb-mercury-control-3 | |
| kube-system dummy-2088944543-05vm1 1/1 Running 0 17h 172.19.80.22 bxb-mercury-control-2 | |
| kube-system etcd-bxb-mercury-control-2 1/1 Running 0 17h 172.19.80.22 bxb-mercury-control-2 | |
| kube-system kube-apiserver-bxb-mercury-control-2 1/1 Running 0 17h 172.19.80.22 bxb-mercury-control-2 | |
| kube-system kube-controller-manager-bxb-mercury-control-2 1/1 Running 0 17h 172.19.80.22 bxb-mercury-control-2 | |
| kube-system kube-discovery-1769846148-rhpm1 1/1 Running 0 17h 172.19.80.22 bxb-mercury-control-2 | |
| kube-system kube-dns-2924299975-sglqn 3/4 Running 9 17h 192.171.0.1 bxb-mercury-control-2 | |
| kube-system kube-proxy-4t5qn 1/1 Running 0 17h 172.19.80.22 bxb-mercury-control-2 | |
| kube-system kube-proxy-7gnp1 1/1 Running 0 23m 172.19.80.24 bxb-mercury-control-3 | |
| kube-system kube-proxy-wwk8b 1/1 Running 0 23m 172.19.80.26 bxb-mercury-control-1 | |
| kube-system kube-scheduler-bxb-mercury-control-2 1/1 Running 0 17h 172.19.80.22 bxb-mercury-control-2 | |
| ----------------------------------------- | |
| Section I. Prior to default epg create | |
| ----------------------------------------- | |
| [root@bxb-mercury-control-2 deployments]# netctl net inspect default-net | |
| Inspeting network: default-net tenant: default | |
| { | |
| "Config": { | |
| "key": "default:default-net", | |
| "encap": "vlan", | |
| "gateway": "192.171.0.254", | |
| "networkName": "default-net", | |
| "nwType": "data", | |
| "subnet": "192.171.0.0/24", | |
| "tenantName": "default", | |
| "link-sets": {}, | |
| "links": { | |
| "Tenant": { | |
| "type": "tenant", | |
| "key": "default" | |
| } | |
| } | |
| }, | |
| "Oper": { | |
| "allocatedAddressesCount": 3, | |
| "allocatedIPAddresses": "192.171.0.1-192.171.0.3, 192.171.0.254", | |
| "availableIPAddresses": "192.171.0.4-192.171.0.253", | |
| "endpoints": [ | |
| { | |
| "containerName": "my-nginx-379829228-f7w0x", | |
| "endpointID": "170292e955d6ae6950a2e63b996ad1236cf32edf885dec2d27298c5275d97135", | |
| "homingHost": "bxb-mercury-control-1", | |
| "ipAddress": [ | |
| "192.171.0.2", | |
| "" | |
| ], | |
| "labels": "map[]", | |
| "macAddress": "02:02:c0:ab:00:02", | |
| "network": "default-net.default" | |
| }, | |
| { | |
| "containerName": "my-nginx-379829228-j8b7x", | |
| "endpointID": "7172dcbf73b1cf2442bf2dbcf3528232716a0f420270212d8893ac1bd34f5836", | |
| "homingHost": "bxb-mercury-control-2", | |
| "ipAddress": [ | |
| "192.171.0.3", | |
| "" | |
| ], | |
| "labels": "map[]", | |
| "macAddress": "02:02:c0:ab:00:03", | |
| "network": "default-net.default" | |
| }, | |
| { | |
| "containerName": "kube-dns-2924299975-sglqn", | |
| "endpointID": "1134d32a231fe22b4f946518a6c3987f025663893440666ab7d0b2b09c9b0430", | |
| "homingHost": "bxb-mercury-control-2", | |
| "ipAddress": [ | |
| "192.171.0.1", | |
| "" | |
| ], | |
| "labels": "map[]", | |
| "macAddress": "02:02:c0:ab:00:01", | |
| "network": "default-net.default" | |
| } | |
| ], | |
| "numEndpoints": 3, | |
| "pktTag": 1 | |
| } | |
| } | |
| ----------------------------------------------- | |
| Check inside one of the nginx containers | |
| [root@bxb-mercury-control-2 deployments]# kubectl exec -it my-nginx-379829228-j8b7x -- /bin/sh | |
| # ip a | |
| 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1 | |
| link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 | |
| inet 127.0.0.1/8 scope host lo | |
| valid_lft forever preferred_lft forever | |
| inet6 ::1/128 scope host | |
| valid_lft forever preferred_lft forever | |
| 27: eth0@if26: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default | |
| link/ether 02:02:c0:ab:00:03 brd ff:ff:ff:ff:ff:ff | |
| inet 192.171.0.3/24 scope global eth0 | |
| valid_lft forever preferred_lft forever | |
| inet6 fe80::2:c0ff:feab:3/64 scope link | |
| valid_lft forever preferred_lft forever | |
| # ip route | |
| default via 192.171.0.254 dev eth0 | |
| 192.171.0.0/24 dev eth0 proto kernel scope link src 192.171.0.3 | |
| # ping 192.171.0.254 | |
| PING 192.171.0.254 (192.171.0.254): 56 data bytes | |
| ^C--- 192.171.0.254 ping statistics --- | |
| 4 packets transmitted, 0 packets received, 100% packet loss | |
| # ^C | |
| # exit | |
| --------------------------------------------------------- | |
| Check inside one of the contiv netplugin containers | |
| [root@bxb-mercury-control-2 deployments]# kubectl exec -it contiv-netplugin-4x6tl -n kube-system -- /bin/sh | |
| # ip a | |
| 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1 | |
| link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 | |
| inet 127.0.0.1/8 scope host lo | |
| valid_lft forever preferred_lft forever | |
| inet6 ::1/128 scope host | |
| valid_lft forever preferred_lft forever | |
| 2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond_lacp state UP group default qlen 1000 | |
| link/ether 58:ac:78:02:9c:89 brd ff:ff:ff:ff:ff:ff | |
| 3: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond_lacp state UP group default qlen 1000 | |
| link/ether 58:ac:78:02:9c:89 brd ff:ff:ff:ff:ff:ff | |
| 4: p0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master p state UP group default qlen 1000 | |
| link/ether 58:ac:78:02:9c:8d brd ff:ff:ff:ff:ff:ff | |
| 5: p1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master p state UP group default qlen 1000 | |
| link/ether 58:ac:78:02:9c:8d brd ff:ff:ff:ff:ff:ff | |
| 6: mx0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master mx state UP group default qlen 1000 | |
| link/ether 58:ac:78:02:9c:8f brd ff:ff:ff:ff:ff:ff | |
| 7: eno1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000 | |
| link/ether 70:e4:22:83:90:d2 brd ff:ff:ff:ff:ff:ff | |
| 8: mx1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master mx state UP group default qlen 1000 | |
| link/ether 58:ac:78:02:9c:8f brd ff:ff:ff:ff:ff:ff | |
| 9: t0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master t state UP group default qlen 1000 | |
| link/ether 58:ac:78:02:9c:91 brd ff:ff:ff:ff:ff:ff | |
| 10: t1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master t state UP group default qlen 1000 | |
| link/ether 58:ac:78:02:9c:91 brd ff:ff:ff:ff:ff:ff | |
| 11: eno2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000 | |
| link/ether 70:e4:22:83:90:d3 brd ff:ff:ff:ff:ff:ff | |
| 12: bond0: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN group default qlen 1000 | |
| link/ether e6:c6:2d:f9:7a:c5 brd ff:ff:ff:ff:ff:ff | |
| 13: bond_lacp: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 | |
| link/ether 58:ac:78:02:9c:89 brd ff:ff:ff:ff:ff:ff | |
| inet6 fe80::5aac:78ff:fe02:9c89/64 scope link | |
| valid_lft forever preferred_lft forever | |
| 14: mx: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 | |
| link/ether 58:ac:78:02:9c:8f brd ff:ff:ff:ff:ff:ff | |
| inet 172.19.80.22/24 brd 172.19.80.255 scope global mx | |
| valid_lft forever preferred_lft forever | |
| inet6 fe80::5aac:78ff:fe02:9c8f/64 scope link | |
| valid_lft forever preferred_lft forever | |
| 15: p: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000 | |
| link/ether 58:ac:78:02:9c:8d brd ff:ff:ff:ff:ff:ff | |
| inet6 fe80::5aac:78ff:fe02:9c8d/64 scope link | |
| valid_lft forever preferred_lft forever | |
| 16: t: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue master ovs-system state UP group default qlen 1000 | |
| link/ether 58:ac:78:02:9c:91 brd ff:ff:ff:ff:ff:ff | |
| inet6 fe80::5aac:78ff:fe02:9c91/64 scope link | |
| valid_lft forever preferred_lft forever | |
| 17: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default | |
| link/ether 02:42:a6:69:b9:f9 brd ff:ff:ff:ff:ff:ff | |
| inet 172.17.0.1/16 scope global docker0 | |
| valid_lft forever preferred_lft forever | |
| 18: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 | |
| link/ether ae:45:31:26:db:e2 brd ff:ff:ff:ff:ff:ff | |
| 21: contivh0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 | |
| link/ether 02:02:ac:14:ff:fe brd ff:ff:ff:ff:ff:ff | |
| inet 172.20.255.254/16 scope global contivh0 | |
| valid_lft forever preferred_lft forever | |
| inet6 fe80::2:acff:fe14:fffe/64 scope link | |
| valid_lft forever preferred_lft forever | |
| 22: inb01: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 | |
| link/ether de:29:77:4d:61:af brd ff:ff:ff:ff:ff:ff | |
| inet 192.168.35.2/24 scope global inb01 | |
| valid_lft forever preferred_lft forever | |
| inet6 fe80::dc29:77ff:fe4d:61af/64 scope link | |
| valid_lft forever preferred_lft forever | |
| 23: vxlan_sys_4789: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65470 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000 | |
| link/ether a2:88:ea:7a:e7:6a brd ff:ff:ff:ff:ff:ff | |
| 24: vvport1@if25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP group default | |
| link/ether d2:ca:84:3f:d7:96 brd ff:ff:ff:ff:ff:ff link-netnsid 0 | |
| inet6 fe80::d0ca:84ff:fe3f:d796/64 scope link | |
| valid_lft forever preferred_lft forever | |
| 26: vvport2@if27: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP group default | |
| link/ether 1e:77:7f:c1:07:94 brd ff:ff:ff:ff:ff:ff link-netnsid 1 | |
| inet6 fe80::1c77:7fff:fec1:794/64 scope link | |
| valid_lft forever preferred_lft forever | |
| # ip route | |
| default via 172.19.80.1 dev mx | |
| 169.254.0.0/16 dev bond_lacp scope link metric 1013 | |
| 169.254.0.0/16 dev mx scope link metric 1014 | |
| 169.254.0.0/16 dev p scope link metric 1015 | |
| 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 | |
| 172.19.80.0/24 dev mx proto kernel scope link src 172.19.80.22 | |
| 172.20.0.0/16 dev contivh0 proto kernel scope link src 172.20.255.254 | |
| 192.168.35.0/24 dev inb01 proto kernel scope link src 192.168.35.2 | |
| # exit | |
| ----------------------------------- | |
| ----------------------------------------- | |
| Section II. After default epg create | |
| > netctl group create -t default default-net default-epg | |
| ----------------------------------------- | |
| [root@bxb-mercury-control-2 ~]# netctl net inspect default-net | |
| Inspeting network: default-net tenant: default | |
| { | |
| "Config": { | |
| "key": "default:default-net", | |
| "encap": "vlan", | |
| "gateway": "192.171.0.254", | |
| "networkName": "default-net", | |
| "nwType": "data", | |
| "subnet": "192.171.0.0/24", | |
| "tenantName": "default", | |
| "link-sets": { | |
| "EndpointGroups": { | |
| "default:default-epg": { | |
| "type": "endpointGroup", | |
| "key": "default:default-epg" | |
| } | |
| } | |
| }, | |
| "links": { | |
| "Tenant": { | |
| "type": "tenant", | |
| "key": "default" | |
| } | |
| } | |
| }, | |
| "Oper": { | |
| "allocatedAddressesCount": 3, | |
| "allocatedIPAddresses": "192.171.0.1-192.171.0.3, 192.171.0.254", | |
| "availableIPAddresses": "192.171.0.4-192.171.0.253", | |
| "endpoints": [ | |
| { | |
| "containerName": "kube-dns-2924299975-sglqn", | |
| "endpointID": "1134d32a231fe22b4f946518a6c3987f025663893440666ab7d0b2b09c9b0430", | |
| "homingHost": "bxb-mercury-control-2", | |
| "ipAddress": [ | |
| "192.171.0.1", | |
| "" | |
| ], | |
| "labels": "map[]", | |
| "macAddress": "02:02:c0:ab:00:01", | |
| "network": "default-net.default" | |
| }, | |
| { | |
| "containerName": "my-nginx-379829228-f7w0x", | |
| "endpointID": "170292e955d6ae6950a2e63b996ad1236cf32edf885dec2d27298c5275d97135", | |
| "homingHost": "bxb-mercury-control-1", | |
| "ipAddress": [ | |
| "192.171.0.2", | |
| "" | |
| ], | |
| "labels": "map[]", | |
| "macAddress": "02:02:c0:ab:00:02", | |
| "network": "default-net.default" | |
| }, | |
| { | |
| "containerName": "my-nginx-379829228-j8b7x", | |
| "endpointID": "7172dcbf73b1cf2442bf2dbcf3528232716a0f420270212d8893ac1bd34f5836", | |
| "homingHost": "bxb-mercury-control-2", | |
| "ipAddress": [ | |
| "192.171.0.3", | |
| "" | |
| ], | |
| "labels": "map[]", | |
| "macAddress": "02:02:c0:ab:00:03", | |
| "network": "default-net.default" | |
| } | |
| ], | |
| "numEndpoints": 3, | |
| "pktTag": 1 | |
| } | |
| } | |
| ----- | |
| # still didn't see routes in ASR | |
| # still couldn't ping between pods in the deployment | |
| # delete the deployment | |
| [root@bxb-mercury-control-2 ~]# kubectl delete deployment my-nginx | |
| deployment "my-nginx" deleted | |
| [root@bxb-mercury-control-2 ~]# kubectl get pods -o wide --all-namespaces | |
| NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE | |
| kube-system contiv-etcd-shwxq 1/1 Running 0 18h 172.19.80.22 bxb-mercury-control-2 | |
| kube-system contiv-netmaster-xd914 1/1 Running 0 18h 172.19.80.22 bxb-mercury-control-2 | |
| kube-system contiv-netplugin-4x6tl 1/1 Running 0 18h 172.19.80.22 bxb-mercury-control-2 | |
| kube-system contiv-netplugin-l5p20 1/1 Running 0 1h 172.19.80.26 bxb-mercury-control-1 | |
| kube-system contiv-netplugin-ns7m7 1/1 Running 0 1h 172.19.80.24 bxb-mercury-control-3 | |
| kube-system dummy-2088944543-05vm1 1/1 Running 0 18h 172.19.80.22 bxb-mercury-control-2 | |
| kube-system etcd-bxb-mercury-control-2 1/1 Running 0 18h 172.19.80.22 bxb-mercury-control-2 | |
| kube-system kube-apiserver-bxb-mercury-control-2 1/1 Running 0 18h 172.19.80.22 bxb-mercury-control-2 | |
| kube-system kube-controller-manager-bxb-mercury-control-2 1/1 Running 0 18h 172.19.80.22 bxb-mercury-control-2 | |
| kube-system kube-discovery-1769846148-rhpm1 1/1 Running 0 18h 172.19.80.22 bxb-mercury-control-2 | |
| kube-system kube-dns-2924299975-sglqn 2/4 CrashLoopBackOff 73 18h 192.171.0.1 bxb-mercury-control-2 | |
| kube-system kube-proxy-4t5qn 1/1 Running 0 18h 172.19.80.22 bxb-mercury-control-2 | |
| kube-system kube-proxy-7gnp1 1/1 Running 0 1h 172.19.80.24 bxb-mercury-control-3 | |
| kube-system kube-proxy-wwk8b 1/1 Running 0 1h 172.19.80.26 bxb-mercury-control-1 | |
| kube-system kube-scheduler-bxb-mercury-control-2 1/1 Running 0 18h 172.19.80.22 bxb-mercury-control-2 | |
| ------------------------------------------------------------- | |
| After another deploy of nginx deployment (replica-set = 2) | |
| - ping still fails | |
| ------------------------------------------------------------- | |
| [root@bxb-mercury-control-2 ~]# kubectl get pods -o wide | |
| NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE | |
| default my-nginx-379829228-w617r 1/1 Running 0 2m 192.171.0.3 bxb-mercury-control-1 | |
| default my-nginx-379829228-wvmt5 1/1 Running 0 2m 192.171.0.2 bxb-mercury-control-3 | |
| [root@bxb-mercury-control-2 ~]# kubectl exec my-nginx-379829228-w617r -- ping -c3 192.171.0.2 | |
| PING 192.171.0.2 (192.171.0.2): 56 data bytes | |
| --- 192.171.0.2 ping statistics --- | |
| 3 packets transmitted, 0 packets received, 100% packet loss | |
| [root@bxb-mercury-control-2 ~]# kubectl exec my-nginx-379829228-w617r -- ip route | |
| default via 192.171.0.254 dev eth0 | |
| 192.171.0.0/24 dev eth0 proto kernel scope link src 192.171.0.3 | |
| [root@bxb-mercury-control-2 ~]# kubectl exec my-nginx-379829228-w617r -- ip a | |
| 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1 | |
| link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 | |
| inet 127.0.0.1/8 scope host lo | |
| valid_lft forever preferred_lft forever | |
| inet6 ::1/128 scope host | |
| valid_lft forever preferred_lft forever | |
| 25: eth0@if24: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default | |
| link/ether 02:02:c0:ab:00:03 brd ff:ff:ff:ff:ff:ff | |
| inet 192.171.0.3/24 scope global eth0 | |
| valid_lft forever preferred_lft forever | |
| inet6 fe80::2:c0ff:feab:3/64 scope link | |
| valid_lft forever preferred_lft forever | |
| ------------------------------------------------- | |
| Check inside one of the netplugin containers | |
| ------------------------------------------------- | |
| [root@bxb-mercury-control-2 ~]# kubectl exec -it -n kube-system contiv-netplugin-l5p20 -- /bin/sh | |
| # ip a | |
| 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1 | |
| link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 | |
| inet 127.0.0.1/8 scope host lo | |
| valid_lft forever preferred_lft forever | |
| inet6 ::1/128 scope host | |
| valid_lft forever preferred_lft forever | |
| 2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond_lacp state UP group default qlen 1000 | |
| link/ether 58:ac:78:02:ce:3f brd ff:ff:ff:ff:ff:ff | |
| 3: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond_lacp state UP group default qlen 1000 | |
| link/ether 58:ac:78:02:ce:3f brd ff:ff:ff:ff:ff:ff | |
| 4: p0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master p state UP group default qlen 1000 | |
| link/ether 58:ac:78:02:ce:43 brd ff:ff:ff:ff:ff:ff | |
| 5: p1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master p state UP group default qlen 1000 | |
| link/ether 58:ac:78:02:ce:43 brd ff:ff:ff:ff:ff:ff | |
| 6: eno1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000 | |
| link/ether 70:e4:22:83:b2:ec brd ff:ff:ff:ff:ff:ff | |
| 7: mx0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master mx state UP group default qlen 1000 | |
| link/ether 58:ac:78:02:ce:45 brd ff:ff:ff:ff:ff:ff | |
| 8: mx1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master mx state UP group default qlen 1000 | |
| link/ether 58:ac:78:02:ce:45 brd ff:ff:ff:ff:ff:ff | |
| 9: t0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master t state UP group default qlen 1000 | |
| link/ether 58:ac:78:02:ce:47 brd ff:ff:ff:ff:ff:ff | |
| 10: t1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master t state UP group default qlen 1000 | |
| link/ether 58:ac:78:02:ce:47 brd ff:ff:ff:ff:ff:ff | |
| 11: eno2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000 | |
| link/ether 70:e4:22:83:b2:ed brd ff:ff:ff:ff:ff:ff | |
| 12: bond0: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN group default qlen 1000 | |
| link/ether 7a:22:e3:70:5c:51 brd ff:ff:ff:ff:ff:ff | |
| 13: bond_lacp: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 | |
| link/ether 58:ac:78:02:ce:3f brd ff:ff:ff:ff:ff:ff | |
| inet6 fe80::5aac:78ff:fe02:ce3f/64 scope link | |
| valid_lft forever preferred_lft forever | |
| 14: mx: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 | |
| link/ether 58:ac:78:02:ce:45 brd ff:ff:ff:ff:ff:ff | |
| inet 172.19.80.26/24 brd 172.19.80.255 scope global mx | |
| valid_lft forever preferred_lft forever | |
| inet6 fe80::5aac:78ff:fe02:ce45/64 scope link | |
| valid_lft forever preferred_lft forever | |
| 15: p: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000 | |
| link/ether 58:ac:78:02:ce:43 brd ff:ff:ff:ff:ff:ff | |
| inet6 fe80::5aac:78ff:fe02:ce43/64 scope link | |
| valid_lft forever preferred_lft forever | |
| 16: t: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue master ovs-system state UP group default qlen 1000 | |
| link/ether 58:ac:78:02:ce:47 brd ff:ff:ff:ff:ff:ff | |
| inet6 fe80::5aac:78ff:fe02:ce47/64 scope link | |
| valid_lft forever preferred_lft forever | |
| 17: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default | |
| link/ether 02:42:6c:2e:f8:75 brd ff:ff:ff:ff:ff:ff | |
| inet 172.17.0.1/16 scope global docker0 | |
| valid_lft forever preferred_lft forever | |
| 18: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 | |
| link/ether a6:c8:fc:3f:92:f3 brd ff:ff:ff:ff:ff:ff | |
| 19: contivh0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 | |
| link/ether 02:02:ac:14:ff:fe brd ff:ff:ff:ff:ff:ff | |
| inet 172.20.255.254/16 scope global contivh0 | |
| valid_lft forever preferred_lft forever | |
| inet6 fe80::2:acff:fe14:fffe/64 scope link | |
| valid_lft forever preferred_lft forever | |
| 20: vxlan_sys_4789: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65470 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000 | |
| link/ether ae:36:83:14:75:e8 brd ff:ff:ff:ff:ff:ff | |
| 21: inb01: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 | |
| link/ether 4a:ee:34:c8:98:12 brd ff:ff:ff:ff:ff:ff | |
| inet 192.168.35.4/24 scope global inb01 | |
| valid_lft forever preferred_lft forever | |
| inet6 fe80::48ee:34ff:fec8:9812/64 scope link | |
| valid_lft forever preferred_lft forever | |
| 24: vvport2@if25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP group default | |
| link/ether 7e:fe:1e:9e:30:60 brd ff:ff:ff:ff:ff:ff link-netnsid 0 | |
| inet6 fe80::7cfe:1eff:fe9e:3060/64 scope link | |
| valid_lft forever preferred_lft forever | |
| # ip netns list | |
| # ip route | |
| default via 172.19.80.1 dev mx | |
| 169.254.0.0/16 dev bond_lacp scope link metric 1013 | |
| 169.254.0.0/16 dev mx scope link metric 1014 | |
| 169.254.0.0/16 dev p scope link metric 1015 | |
| 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 | |
| 172.19.80.0/24 dev mx proto kernel scope link src 172.19.80.26 | |
| 172.20.0.0/16 dev contivh0 proto kernel scope link src 172.20.255.254 | |
| 192.168.35.0/24 dev inb01 proto kernel scope link src 192.168.35.4 | |
| # ovs-vsctl show | |
| 3ae531ff-fa47-4f48-a072-624a918ee32d | |
| Manager "ptcp:6640" | |
| is_connected: true | |
| Bridge contivVxlanBridge | |
| Controller "tcp:127.0.0.1:6633" | |
| is_connected: true | |
| fail_mode: secure | |
| Port "vxif172198022" | |
| Interface "vxif172198022" | |
| type: vxlan | |
| options: {key=flow, remote_ip="172.19.80.22", tos=inherit} | |
| Port "vxif172198024" | |
| Interface "vxif172198024" | |
| type: vxlan | |
| options: {key=flow, remote_ip="172.19.80.24", tos=inherit} | |
| Bridge contivVlanBridge | |
| Controller "tcp:127.0.0.1:6634" | |
| is_connected: true | |
| fail_mode: secure | |
| Port t | |
| Interface t | |
| Port "vvport2" | |
| tag: 1 | |
| Interface "vvport2" | |
| Port "inb01" | |
| tag: 1 | |
| Interface "inb01" | |
| type: internal | |
| Bridge contivHostBridge | |
| Controller "tcp:127.0.0.1:6635" | |
| is_connected: true | |
| fail_mode: secure | |
| Port "contivh0" | |
| tag: 2 | |
| Interface "contivh0" | |
| type: internal | |
| # |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment