Skip to content

Instantly share code, notes, and snippets.

@nerdalert

nerdalert/global-net-crash.md

Created April 12, 2022 06:55
Show Gist options
  • Save nerdalert/5fae5ac76b698768d99d2dded14e8850 to your computer and use it in GitHub Desktop.
Save nerdalert/5fae5ac76b698768d99d2dded14e8850 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
global-net-crash.md

Logs and Cluster Details from GlobalNet Crashing with active/active branch

  • Makes/Images
git clone https://github.com/astoycos/submariner-operator.git
git clone https://github.com/astoycos/submariner.git
cd submariner
git checkout remotes/origin/AS-mult-active-gw-multiple-vxlan-tunnels
make images
cd ../submariner-operator
git checkout remotes/origin/AS-multi-active-gw
make images

docker tag quay.io/submariner/submariner-gateway:devel  quay.io/networkstatic/submariner-gateway:devel
docker push quay.io/networkstatic/submariner-gateway:devel

docker tag quay.io/submariner/submariner-globalnet:devel  quay.io/networkstatic/submariner-globalnet:devel
docker push quay.io/networkstatic/submariner-globalnet:devel

docker tag quay.io/submariner/submariner-operator:devel  quay.io/networkstatic/submariner-operator:devel
docker push quay.io/networkstatic/submariner-operator:devel

docker tag  quay.io/submariner/submariner-route-agent:devel quay.io/networkstatic/submariner-route-agent:devel
docker push quay.io/networkstatic/submariner-route-agent:devel
  • Subctl Broker/Joins:
- name: Download the custom active/active subctl binary
  shell: |
    sudo curl {{ submariner_subctl_binary_url }} --output /usr/local/bin/subctl
    sudo chmod +x /usr/local/bin/subctl

- name: Start the submariner broker
  shell: |
    subctl --kubeconfig /home/{{ ansible_user }}/config deploy-broker \
     --components service-discovery,connectivity \
    --globalnet
    kubectl wait --for=condition=ready pod -l name=submariner-operator --timeout=60s -n submariner-operator --kubeconfig /home/{{ ansible_user }}/config


- name: Label the node as a gateway
  command: kubectl label node {{ hostname }} submariner.io/gateway=true --overwrite --kubeconfig /home/{{ ansible_user }}/config


# Join from the K8s Master & Broker
- name: Join the submariner broker node to the broker
  shell: |
    subctl --kubeconfig /home/{{ ansible_user }}/config \
    join /home/{{ ansible_user }}/broker-info.subm \
    --clusterid {{ hostname }} \
    --nattport 4500 \
    --cable-driver {{ submariner_cable_driver }} \
    --natt=false \
    --multi-active-gateway=true \
    --image-override=submariner-operator={{ submariner_image_override }}/submariner-operator:devel \
    --image-override=submariner-globalnet={{ submariner_image_override }}/submariner-globalnet:devel \
    --image-override=submariner-gateway={{ submariner_image_override }}/submariner-gateway:devel \
    --image-override=submariner-route-agent={{ submariner_image_override }}/submariner-route-agent:devel


- name: Join the submariner secondary gateway node to the broker
  shell: |
    subctl --kubeconfig /home/{{ ansible_user }}/config \
    join /home/{{ ansible_user }}/broker-info.subm \
    --clusterid {{ hostname }} \
    --cable-driver {{ submariner_cable_driver }} \
    --nattport 4500 \
    --natt=false \
    --multi-active-gateway=true \
    --image-override=submariner-operator={{ submariner_image_override }}/submariner-operator:devel \
    --image-override=submariner-globalnet={{ submariner_image_override }}/submariner-globalnet:devel \
    --image-override=submariner-gateway={{ submariner_image_override }}/submariner-gateway:devel \
    --image-override=submariner-route-agent={{ submariner_image_override }}/submariner-route-agent:devel
  • Globalnet worker node crash log:
ubuntu@cluster0-broker-ub3-ha1-cluster:~$ kubectl logs submariner-globalnet-555ddf579d-xfj8q -n submariner-operator
+ trap 'exit 1' SIGTERM SIGINT
+ SUBMARINER_VERBOSITY=2
+ '[' '' == true ']'
+ DEBUG=-v=2
+ exec submariner-globalnet -v=2 -alsologtostderr
W0412 06:50:53.769027       1 client_config.go:608] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0412 06:50:53.769500       1 main.go:76] Starting submariner-globalnet{cluster0-ha2-ub3-node submariner-operator [] false}
I0412 06:50:53.834598       1 gateway_monitor.go:117] Starting GatewayMonitor to monitor the active Gateway node in the cluster.
I0412 06:50:53.935613       1 gateway_monitor.go:332] Install/ensure SUBMARINER-GN-MARK chain exists
I0412 06:50:53.936468       1 gateway_monitor.go:144] In processNextEndpoint, endpoint info: {"metadata":{"name":"cluster2-ub3-node-submariner-cable-cluster2-ub3-node-172-31-22-182","namespace":"submariner-operator","uid":"8507bbfc-b9e9-4681-89d5-6c0e6f31cf44","resourceVersion":"2443","generation":1,"creationTimestamp":"2022-04-12T06:42:50Z","labels":{"submariner-io/clusterID":"cluster2-ub3-node"},"managedFields":[{"manager":"submariner-gateway","operation":"Update","apiVersion":"submariner.io/v1","time":"2022-04-12T06:42:50Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:submariner-io/clusterID":{}}},"f:spec":{".":{},"f:backend":{},"f:backend_config":{".":{},"f:natt-discovery-port":{},"f:preferred-server":{},"f:udp-port":{}},"f:cable_name":{},"f:cluster_id":{},"f:hostname":{},"f:nat_enabled":{},"f:private_ip":{},"f:public_ip":{},"f:subnets":{}}}}]},"spec":{"cluster_id":"cluster2-ub3-node","cable_name":"submariner-cable-cluster2-ub3-node-172-31-22-182","hostname":"cluster2-ub3-node","subnets":["242.3.0.0/16"],"private_ip":"172.31.22.182","public_ip":"54.175.130.218","nat_enabled":false,"backend":"vxlan","backend_config":{"natt-discovery-port":"4490","preferred-server":"false","udp-port":"4500"}}}
I0412 06:50:53.936843       1 gateway_monitor.go:147] Endpoint "cluster2-ub3-node", host: "cluster2-ub3-node" belongs to a remote cluster
I0412 06:50:53.936970       1 gateway_monitor.go:448] Marking traffic destined to remote cluster: -d 242.3.0.0/16 -j MARK --set-mark 0xC0000/0xC0000
F0412 06:50:53.938015       1 main.go:124] Error running gatewayMonitor: error while calling createGlobalNetMarkingChain: error creating iptables chain SUBMARINER-GN-MARK: error listing IP table chains: running [/usr/sbin/iptables -t nat -S --wait 5]: exit status 4: Fatal: can't open lock file /run/xtables.lock: Is a directory
  • Cluster details
ubuntu@cluster0-broker-ub3-ha1-cluster:~$ kubectl get nodes
NAME                              STATUS   ROLES                  AGE   VERSION
cluster0-broker-ub3-ha1-cluster   Ready    control-plane,master   20m   v1.22.7+k3s1
cluster0-ub3-worker-1-node        Ready    <none>                 15m   v1.22.7+k3s1
cluster0-ha2-ub3-node             Ready    <none>                 17m   v1.22.7+k3s1


ubuntu@cluster0-broker-ub3-ha1-cluster:~$ kubectl get pods --all-namespaces -o wide
NAMESPACE             NAME                                             READY   STATUS    RESTARTS      AGE    IP              NODE                              NOMINATED NODE   READINESS GATES
kube-system           local-path-provisioner-84bb864455-d7mgl          1/1     Running   0             20m    10.42.0.3       cluster0-broker-ub3-ha1-cluster   <none>           <none>
kube-system           coredns-96cc4f57d-8zqtl                          1/1     Running   0             20m    10.42.0.2       cluster0-broker-ub3-ha1-cluster   <none>           <none>
kube-system           metrics-server-ff9dbcb6c-zsl9t                   1/1     Running   0             20m    10.42.0.4       cluster0-broker-ub3-ha1-cluster   <none>           <none>
submariner-operator   submariner-operator-789c775b8-bscfg              1/1     Running   0             15m    10.42.1.2       cluster0-ha2-ub3-node             <none>           <none>
submariner-operator   submariner-gateway-k9fb7                         1/1     Running   0             13m    172.31.19.114   cluster0-broker-ub3-ha1-cluster   <none>           <none>
submariner-operator   submariner-gateway-8xfrd                         1/1     Running   0             13m    172.31.24.248   cluster0-ha2-ub3-node             <none>           <none>
submariner-operator   submariner-routeagent-bwq99                      1/1     Running   0             13m    172.31.24.248   cluster0-ha2-ub3-node             <none>           <none>
submariner-operator   submariner-routeagent-lkcfr                      1/1     Running   0             13m    172.31.19.114   cluster0-broker-ub3-ha1-cluster   <none>           <none>
submariner-operator   submariner-lighthouse-agent-6d9546dfb9-skhcj     1/1     Running   0             13m    10.42.2.9       cluster0-ub3-worker-1-node        <none>           <none>
submariner-operator   submariner-routeagent-6pnnq                      1/1     Running   0             13m    172.31.28.95    cluster0-ub3-worker-1-node        <none>           <none>
submariner-operator   submariner-lighthouse-coredns-5f7669c945-w6n9t   1/1     Running   0             13m    10.42.2.10      cluster0-ub3-worker-1-node        <none>           <none>
submariner-operator   submariner-lighthouse-coredns-5f7669c945-ts47r   1/1     Running   0             13m    10.42.1.4       cluster0-ha2-ub3-node             <none>           <none>
submariner-operator   submariner-globalnet-555ddf579d-xfj8q            0/1     Error     5 (88s ago)   3m5s   10.42.1.5       cluster0-ha2-ub3-node             <none>           <none>


ubuntu@cluster0-broker-ub3-ha1-cluster:~$ subctl show all
Cluster "default"
 ✓ Detecting broker(s)
NAMESPACE                NAME                     COMPONENTS
submariner-k8s-broker    submariner-broker        service-discovery, connectivity

 ✓ Showing Connections
GATEWAY            CLUSTER            REMOTE IP      NAT  CABLE DRIVER  SUBNETS       STATUS     RTT avg.
cluster2-ub3-node  cluster2-ub3-node  172.31.22.182  no   vxlan         242.3.0.0/16  connected
cluster1-ub3-node  cluster1-ub3-node  172.31.21.202  no   vxlan         242.2.0.0/16  connected
cluster2-ub3-node  cluster2-ub3-node  172.31.22.182  no   vxlan         242.3.0.0/16  connected
cluster1-ub3-node  cluster1-ub3-node  172.31.21.202  no   vxlan         242.2.0.0/16  connected

 ✓ Showing Endpoints
CLUSTER ID                    ENDPOINT IP     PUBLIC IP       CABLE DRIVER        TYPE
cluster0-ha2-ub3-node         172.31.19.114   54.221.51.199   vxlan               local
cluster2-ub3-node             172.31.22.182   54.175.130.218  vxlan               remote
cluster1-ub3-node             172.31.21.202   54.236.105.235  vxlan               remote
cluster0-ha2-ub3-node         172.31.24.248   100.25.216.67   vxlan               local
cluster2-ub3-node             172.31.22.182   54.175.130.218  vxlan               remote
cluster1-ub3-node             172.31.21.202   54.236.105.235  vxlan               remote

 ✓ Showing Gateways
NODE                            HA STATUS       SUMMARY
cluster0-broker-ub3-ha1-cluster active          All connections (2) are established
cluster0-ha2-ub3-node           active          All connections (2) are established

    Discovered network details via Submariner:
 ✓ Showing Network details
        Network plugin:  generic
        Service CIDRs:   [10.43.0.0/16]
        Cluster CIDRs:   [10.42.2.0/24]
        Global CIDR:     242.1.0.0/16

 ✓ Showing versions
COMPONENT                       REPOSITORY                                            VERSION
submariner                      quay.io/submariner                                    devel
submariner-operator             quay.io/networkstatic                                 devel
service-discovery               quay.io/submariner                                    devel
COMPONENT                       REPOSITORY                                            VERSION
submariner                      quay.io/submariner                                    devel
submariner-operator             quay.io/networkstatic                                 devel
service-discovery               quay.io/submariner                                    devel
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment