Skip to content

Instantly share code, notes, and snippets.

@nerdalert

nerdalert/globalnet-crash.md

Created April 21, 2022 05:44
Show Gist options
  • Save nerdalert/7820d7067a11cac39e100051d1ac1990 to your computer and use it in GitHub Desktop.
Save nerdalert/7820d7067a11cac39e100051d1ac1990 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
globalnet-crash.md 

[fedora@cluster2-ha1-perf-target-node ~]$ kubectl get pods --all-namespaces -o wide
NAMESPACE             NAME                                             READY   STATUS             RESTARTS         AGE   IP            NODE                            NOMINATED NODE   READINESS GATES
kube-system           local-path-provisioner-84bb864455-5zq8r          1/1     Running            0                76m   10.42.0.3     cluster2-ha1-perf-target-node   <none>           <none>
kube-system           coredns-96cc4f57d-bz2ns                          1/1     Running            0                76m   10.42.0.2     cluster2-ha1-perf-target-node   <none>           <none>
kube-system           metrics-server-ff9dbcb6c-vz9w2                   1/1     Running            0                76m   10.42.0.4     cluster2-ha1-perf-target-node   <none>           <none>
submariner-operator   submariner-operator-789c775b8-xj5tv              1/1     Running            1 (67m ago)      67m   10.42.2.2     cluster2-worker1-node           <none>           <none>
submariner-operator   submariner-lighthouse-agent-b5f5c9d8d-ntvnz      1/1     Running            0                67m   10.42.1.2     cluster2-ha2-node               <none>           <none>
submariner-operator   submariner-lighthouse-coredns-5b56499d77-9n2g2   1/1     Running            0                67m   10.42.1.3     cluster2-ha2-node               <none>           <none>
submariner-operator   submariner-lighthouse-coredns-5b56499d77-bqq9t   1/1     Running            0                67m   10.42.2.4     cluster2-worker1-node           <none>           <none>
submariner-operator   submariner-routeagent-brbcr                      1/1     Running            0                67m   10.10.0.125   cluster2-worker1-node           <none>           <none>
submariner-operator   submariner-gateway-jnjhj                         1/1     Running            0                67m   10.10.0.18    cluster2-ha2-node               <none>           <none>
submariner-operator   submariner-gateway-l9ng8                         1/1     Running            0                67m   10.10.0.50    cluster2-ha1-perf-target-node   <none>           <none>
submariner-operator   submariner-routeagent-bjlxv                      1/1     Running            0                67m   10.10.0.18    cluster2-ha2-node               <none>           <none>
submariner-operator   submariner-routeagent-wb9sb                      1/1     Running            0                67m   10.10.0.50    cluster2-ha1-perf-target-node   <none>           <none>
default               netserver-0                                      1/1     Running            0                65m   10.42.1.4     cluster2-ha2-node               <none>           <none>
submariner-operator   submariner-globalnet-849467f9bb-985vk            0/1     CrashLoopBackOff   17 (2m36s ago)   67m   10.42.2.3     cluster2-worker1-node           <none>           <none>


[fedora@cluster2-ha1-perf-target-node ~]$ k describe pod submariner-globalnet-849467f9bb-985vk -n submariner-operator
Name:         submariner-globalnet-849467f9bb-985vk
Namespace:    submariner-operator
Priority:     0
Node:         cluster2-worker1-node/10.10.0.125
Start Time:   Thu, 21 Apr 2022 04:34:18 +0000
Labels:       app=submariner-globalnet
              component=globalnet
              pod-template-hash=849467f9bb
Annotations:  <none>
Status:       Running
IP:           10.42.2.3
IPs:
  IP:           10.42.2.3
Controlled By:  ReplicaSet/submariner-globalnet-849467f9bb
Containers:
  submariner-globalnet:
    Container ID:   containerd://2695851334254d12d72733cc91bd2437e9a91cb313c0a42cd650a8e77139bb8f
    Image:          quay.io/networkstatic/submariner-globalnet:devel
    Image ID:       quay.io/networkstatic/submariner-globalnet@sha256:24c4c8d869cce8fb2545776ca4dedd92acff368e725aefc2d5b53d2810aee5bb
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Thu, 21 Apr 2022 05:38:52 +0000
      Finished:     Thu, 21 Apr 2022 05:38:52 +0000
    Ready:          False
    Restart Count:  17
    Environment:
      SUBMARINER_NAMESPACE:                  submariner-operator
      SUBMARINER_CLUSTERID:                  cluster2-ha1-perf-target-node
      SUBMARINER_MULTIACTIVEGATEWAYENABLED:  true
      SUBMARINER_EXCLUDENS:                  submariner-operator,kube-system,operators,openshift-monitoring,openshift-dns
      NODE_NAME:                              (v1:spec.nodeName)
    Mounts:
      /run/xtables.lock from host-run-xtables-lock (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ssj4t (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  host-run-xtables-lock:
    Type:          HostPath (bare host directory volume)
    Path:          /run/xtables.lock
    HostPathType:
  kube-api-access-ssj4t:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 op=Exists
Events:
  Type     Reason   Age                   From     Message
  ----     ------   ----                  ----     -------
  Warning  BackOff  2m5s (x290 over 65m)  kubelet  Back-off restarting failed container
  
  
erf-target-node"
I0421 05:33:47.646197       1 node_controller.go:99] Processing created Node "cluster2-ha2-node"
I0421 05:33:47.746400       1 cluster_egressip_controller.go:112] Processing created ClusterGlobalEgressIP "cluster2-ha2-node-cluster-egress.submariner.io", Spec.NumberOfIPs: 8, Status: v1.GlobalEgressIPStatus{Conditions:[]v1.Condition{v1.Condition{Type:"Allocated", Status:"False", ObservedGeneration:0, LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:63786112468, loc:(*time.Location)(0x1f9cac0)}}, Reason:"NoDatapathRules", Message:"Allocated 8 global IP(s)"}, v1.Condition{Type:"Allocated", Status:"True", ObservedGeneration:0, LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:63786112469, loc:(*time.Location)(0x1f9cac0)}}, Reason:"DatapathRuleWriteSuccess", Message:"Allocated 8 global IP(s)"}}, AllocatedIPs:[]string{"242.1.0.1", "242.1.0.2", "242.1.0.3", "242.1.0.4", "242.1.0.5", "242.1.0.6", "242.1.0.7", "242.1.0.8"}}
I0421 05:33:47.746464       1 cluster_egressip_controller.go:174] Update called for "cluster2-ha2-node-cluster-egress.submariner.io", but numberOfIPs 8 are already allocated
I0421 05:33:47.746556       1 cluster_egressip_controller.go:112] Processing created ClusterGlobalEgressIP "cluster2-ha1-perf-target-node-cluster-egress.submariner.io", Spec.NumberOfIPs: 8, Status: v1.GlobalEgressIPStatus{Conditions:[]v1.Condition{v1.Condition{Type:"Allocated", Status:"False", ObservedGeneration:0, LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:63786112471, loc:(*time.Location)(0x1f9cac0)}}, Reason:"NoDatapathRules", Message:"Allocated 8 global IP(s)"}, v1.Condition{Type:"Allocated", Status:"True", ObservedGeneration:0, LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:63786112471, loc:(*time.Location)(0x1f9cac0)}}, Reason:"DatapathRuleWriteSuccess", Message:"Allocated 8 global IP(s)"}}, AllocatedIPs:[]string{"242.1.0.9", "242.1.0.10", "242.1.0.11", "242.1.0.12", "242.1.0.13", "242.1.0.14", "242.1.0.15", "242.1.0.16"}}
I0421 05:33:47.746589       1 cluster_egressip_controller.go:174] Update called for "cluster2-ha1-perf-target-node-cluster-egress.submariner.io", but numberOfIPs 8 are already allocated
I0421 05:33:47.947615       1 global_ingressip_controller.go:112] Processing created default/pod-netserver-0, TargetRef: "HeadlessServicePod", "netserver-0", Status: v1.GlobalIngressIPStatus{Conditions:[]v1.Condition(nil), AllocatedIP:""}
I0421 05:33:47.947663       1 global_ingressip_controller.go:151] Allocated global IP ["242.1.255.252"] for "default/pod-netserver-0"
E0421 05:33:47.947762       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 328 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x13cbd40, 0x1f7e800)
	k8s.io/[email protected]/pkg/util/runtime/runtime.go:74 +0xa6
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	k8s.io/[email protected]/pkg/util/runtime/runtime.go:48 +0x86
panic(0x13cbd40, 0x1f7e800)
	runtime/panic.go:965 +0x1b9
github.com/submariner-io/submariner/pkg/globalnet/controllers.(*globalIngressIPController).onCreate(0xc000651180, 0xc000fa3340, 0x15787d9)
	github.com/submariner-io/submariner/pkg/globalnet/controllers/global_ingressip_controller.go:216 +0x6da
github.com/submariner-io/submariner/pkg/globalnet/controllers.(*globalIngressIPController).process(0xc000651180, 0x16ea408, 0xc000fa3340, 0x1, 0x0, 0x13848e0, 0xc000efba20, 0x1)
	github.com/submariner-io/submariner/pkg/globalnet/controllers/global_ingressip_controller.go:118 +0x394
github.com/submariner-io/admiral/pkg/syncer.(*resourceSyncer).transform(0xc0003a8000, 0xc0003e65b8, 0xc000ed5710, 0x17, 0x0, 0xf, 0x0, 0x0, 0x681314)
	github.com/submariner-io/[email protected]/pkg/syncer/resource_syncer.go:533 +0x136
github.com/submariner-io/admiral/pkg/syncer.(*resourceSyncer).processNextWorkItem(0xc0003a8000, 0xc000ed5710, 0x17, 0xc000ed5718, 0xf, 0xc000ed5710, 0x7, 0x2, 0x10, 0x7f6413af35b8)
	github.com/submariner-io/[email protected]/pkg/syncer/resource_syncer.go:417 +0x338
github.com/submariner-io/admiral/pkg/workqueue.(*queueType).processNextWorkItem.func1(0xc000ed5710, 0x17, 0xc000efb480, 0x0, 0x0, 0x7400000000000000)
	github.com/submariner-io/[email protected]/pkg/workqueue/queue.go:103 +0x93
github.com/submariner-io/admiral/pkg/workqueue.(*queueType).processNextWorkItem(0xc0005ce140, 0xc000efb480, 0x203000)
	github.com/submariner-io/[email protected]/pkg/workqueue/queue.go:104 +0x129
github.com/submariner-io/admiral/pkg/workqueue.(*queueType).Run.func1()
	github.com/submariner-io/[email protected]/pkg/workqueue/queue.go:79 +0x49
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00062cd08)
	k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00062cd08, 0x16e4580, 0xc000ef5ad0, 0xc00043c001, 0xc0005e0960)
	k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0x9b
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00062cd08, 0x3b9aca00, 0x0, 0x15e7901, 0xc0005e0960)
	k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc00062cd08, 0x3b9aca00, 0xc0005e0960)
	k8s.io/[email protected]/pkg/util/wait/wait.go:90 +0x4d
created by github.com/submariner-io/admiral/pkg/workqueue.(*queueType).Run
	github.com/submariner-io/[email protected]/pkg/workqueue/queue.go:78 +0x8b
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x12b84fa]

goroutine 328 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	k8s.io/[email protected]/pkg/util/runtime/runtime.go:55 +0x109
panic(0x13cbd40, 0x1f7e800)
	runtime/panic.go:965 +0x1b9
github.com/submariner-io/submariner/pkg/globalnet/controllers.(*globalIngressIPController).onCreate(0xc000651180, 0xc000fa3340, 0x15787d9)
	github.com/submariner-io/submariner/pkg/globalnet/controllers/global_ingressip_controller.go:216 +0x6da
github.com/submariner-io/submariner/pkg/globalnet/controllers.(*globalIngressIPController).process(0xc000651180, 0x16ea408, 0xc000fa3340, 0x1, 0x0, 0x13848e0, 0xc000efba20, 0x1)
	github.com/submariner-io/submariner/pkg/globalnet/controllers/global_ingressip_controller.go:118 +0x394
github.com/submariner-io/admiral/pkg/syncer.(*resourceSyncer).transform(0xc0003a8000, 0xc0003e65b8, 0xc000ed5710, 0x17, 0x0, 0xf, 0x0, 0x0, 0x681314)
	github.com/submariner-io/[email protected]/pkg/syncer/resource_syncer.go:533 +0x136
github.com/submariner-io/admiral/pkg/syncer.(*resourceSyncer).processNextWorkItem(0xc0003a8000, 0xc000ed5710, 0x17, 0xc000ed5718, 0xf, 0xc000ed5710, 0x7, 0x2, 0x10, 0x7f6413af35b8)
	github.com/submariner-io/[email protected]/pkg/syncer/resource_syncer.go:417 +0x338
github.com/submariner-io/admiral/pkg/workqueue.(*queueType).processNextWorkItem.func1(0xc000ed5710, 0x17, 0xc000efb480, 0x0, 0x0, 0x7400000000000000)
	github.com/submariner-io/[email protected]/pkg/workqueue/queue.go:103 +0x93
github.com/submariner-io/admiral/pkg/workqueue.(*queueType).processNextWorkItem(0xc0005ce140, 0xc000efb480, 0x203000)
	github.com/submariner-io/[email protected]/pkg/workqueue/queue.go:104 +0x129
github.com/submariner-io/admiral/pkg/workqueue.(*queueType).Run.func1()
	github.com/submariner-io/[email protected]/pkg/workqueue/queue.go:79 +0x49
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00062cd08)
	k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00062cd08, 0x16e4580, 0xc000ef5ad0, 0xc00043c001, 0xc0005e0960)
	k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0x9b
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00062cd08, 0x3b9aca00, 0x0, 0x15e7901, 0xc0005e0960)
	k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc00062cd08, 0x3b9aca00, 0xc0005e0960)
	k8s.io/[email protected]/pkg/util/wait/wait.go:90 +0x4d
created by github.com/submariner-io/admiral/pkg/workqueue.(*queueType).Run
	github.com/submariner-io/[email protected]/pkg/workqueue/queue.go:78 +0x8b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment