- Node Addresses:
[brokerNode]
35.171.157.41 ansible_user=fedora ansible_connection=ssh k8s_master=10.10.0.80 hostname=cluster1-ha1-node
[secondaryGatewayNode]
18.234.103.8 ansible_user=fedora ansible_connection=ssh k8s_master=10.10.0.80 hostname=cluster1-ha2-node
- Gateway labels and K8s worker/master
[fedora@cluster1-ha1-node ~]$ kubectl get nodes --show-labels
NAME STATUS ROLES AGE VERSION LABELS
cluster1-ha2-node Ready <none> 40m v1.22.7+k3s1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=cluster1-ha2-node,kubernetes.io/os=linux,node.kubernetes.io/instance-type=k3s,submariner.io/gateway=true
cluster1-ha1-node Ready control-plane,master 43m v1.22.7+k3s1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=cluster1-ha1-node,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=true,node-role.kubernetes.io/master=true,node.kubernetes.io/instance-type=k3s,submariner.io/gateway=true
- subctl binary and Vars
submariner_subctl_binary_url : https://perf-testing-submariner-s3.s3.amazonaws.com/subctl
submariner_broker_repository: quay.io/networkstatic # custom image repo broker deploys
submariner_image_override: quay.io/networkstatic # custom image repo for submariner deploys
- Deploy Broker on HA1
subctl --kubeconfig /home/{{ ansible_user }}/config deploy-broker \
--components service-discovery,connectivity \
--globalnet \
--repository {{ submariner_broker_repository }}
- Broker node join from HA1
subctl --kubeconfig /home/{{ ansible_user }}/config \
join /home/{{ ansible_user }}/broker-info.subm \
--clusterid {{ hostname }} \
--clustercidr={{ submariner_cluster_cidr }} \
--nattport 4500 \
--cable-driver {{ submariner_cable_driver }} \
--natt=false \
--multi-active-gateway=true \
--image-override=submariner-operator={{ submariner_image_override }}/submariner-operator:devel \
--image-override=submariner-globalnet={{ submariner_image_override }}/submariner-globalnet:devel \
--image-override=submariner-gateway={{ submariner_image_override }}/submariner-gateway:devel \
--image-override=submariner-route-agent={{ submariner_image_override }}/submariner-route-agent:devel \
--image-override=submariner-networkplugin-syncer={{ submariner_image_override }}/submariner-networkplugin-syncer:devel
Worker HA Secondary Join from HA2
# Uncomment for active/active HA experimental branch
- name: Join the submariner node to the broker as a second active/active gateway
shell: |
subctl --kubeconfig /home/{{ ansible_user }}/config \
join /home/{{ ansible_user }}/broker-info.subm \
--clusterid {{ hostname }} \
--cable-driver {{ submariner_cable_driver }} \
--clustercidr={{ submariner_cluster_cidr }} \
--nattport 4500 \
--natt=false \
--multi-active-gateway=true \
--image-override=submariner-operator={{ submariner_image_override }}/submariner-operator:devel \
--image-override=submariner-globalnet={{ submariner_image_override }}/submariner-globalnet:devel \
--image-override=submariner-gateway={{ submariner_image_override }}/submariner-gateway:devel \
--image-override=submariner-route-agent={{ submariner_image_override }}/submariner-route-agent:devel \
--image-override=submariner-networkplugin-syncer={{ submariner_image_override }}/submariner-networkplugin-syncer:devel
[fedora@cluster1-ha1-node ~]$ subctl show all
Cluster "default"
✓ Detecting broker(s)
NAMESPACE NAME COMPONENTS
submariner-k8s-broker submariner-broker service-discovery, connectivity
✗ Showing Connections
✗ No connections found
✓ Showing Endpoints
CLUSTER ID ENDPOINT IP PUBLIC IP CABLE DRIVER TYPE
cluster1-ha2-node 10.10.0.80 35.171.157.41 vxlan local
cluster1-ha2-node 10.10.0.179 18.234.103.8 vxlan local
✓ Showing Gateways
NODE HA STATUS SUMMARY
cluster1-ha1-node active There are no connections
cluster1-ha2-node active There are no connections
Discovered network details via Submariner:
✓ Showing Network details
Network plugin: generic
Service CIDRs: [10.43.0.0/16]
Cluster CIDRs: [10.42.0.0/16]
Global CIDR: 242.1.0.0/16
✓ Showing versions
COMPONENT REPOSITORY VERSION
submariner quay.io/submariner devel
submariner-operator quay.io/networkstatic devel
service-discovery quay.io/submariner devel
+ exec submariner-gateway -v=2 -alsologtostderr
I0418 21:44:02.812089 1 main.go:94] Starting the submariner gateway engine
W0418 21:44:02.813741 1 client_config.go:608] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0418 21:44:02.815482 1 main.go:118] Creating the cable engine
I0418 21:44:02.987515 1 main.go:135] Creating the datastore syncer
I0418 21:44:03.170138 1 natdiscovery.go:118] NAT discovery server starting on port 4490
I0418 21:44:03.215349 1 syncer.go:90] CableEngine syncer started
I0418 21:44:03.216273 1 main.go:349] Cluster is configured to use multiple active gateways
I0418 21:44:03.216441 1 main.go:368] Using namespace "submariner-operator" for the leader election lock
I0418 21:44:03.216461 1 main.go:380] Multiple Active Gateways Enabled, creating unique reourcelock, name "cluster1-ha1-node-submariner-gateway-lock" identity "cluster1-ha1-node-cluster1-ha1-node-submariner-gateway"
I0418 21:44:03.216475 1 leaderelection.go:243] attempting to acquire leader lease submariner-operator/cluster1-ha1-node-submariner-gateway-lock...
I0418 21:44:03.231534 1 leaderelection.go:253] successfully acquired lease submariner-operator/cluster1-ha1-node-submariner-gateway-lock
I0418 21:44:03.231741 1 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"submariner-operator", Name:"cluster1-ha1-node-submariner-gateway-lock", UID:"b58d3ddf-2121-4b2a-98c3-bdfd5ab75ff3", APIVersion:"v1", ResourceVersion:"1699", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' cluster1-ha1-node-cluster1-ha1-node-submariner-gateway became leader
I0418 21:44:03.238150 1 vxlan.go:154] Successfully configured rp_filter to loose mode(2) on vxlan-tunnel
I0418 21:44:03.241749 1 cableengine.go:105] CableEngine controller started, driver: "vxlan"
I0418 21:44:03.241780 1 gateway_monitor.go:117] Starting GatewayMonitor to monitor the active Gateway node in the cluster.
I0418 21:44:03.241959 1 tunnel.go:38] Starting the tunnel controller
I0418 21:44:03.242178 1 datastoresyncer.go:72] Starting the datastore syncer
I0418 21:44:03.346476 1 healthchecker.go:108] CableEngine HealthChecker started with PingInterval: 1, MaxPacketLossCount: 5
I0418 21:44:03.350136 1 gateway_monitor.go:163] In processNextEndpoint, endpoint info: {"metadata":{"name":"cluster1-ha2-node-submariner-cable-cluster1-ha2-node-10-10-0-179","namespace":"submariner-operator","uid":"8ba30470-729d-427e-b738-022cebf77096","resourceVersion":"1673","generation":1,"creationTimestamp":"2022-04-18T21:44:02Z","managedFields":[{"manager":"submariner-gateway","operation":"Update","apiVersion":"submariner.io/v1","time":"2022-04-18T21:44:02Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{".":{},"f:backend":{},"f:backend_config":{".":{},"f:natt-discovery-port":{},"f:preferred-server":{},"f:udp-port":{}},"f:cable_name":{},"f:cluster_id":{},"f:hostname":{},"f:nat_enabled":{},"f:private_ip":{},"f:public_ip":{},"f:subnets":{}}}}]},"spec":{"cluster_id":"cluster1-ha2-node","cable_name":"submariner-cable-cluster1-ha2-node-10-10-0-179","hostname":"cluster1-ha2-node","subnets":["242.1.0.0/16"],"private_ip":"10.10.0.179","public_ip":"18.234.103.8","nat_enabled":false,"backend":"vxlan","backend_config":{"natt-discovery-port":"4490","preferred-server":"false","udp-port":"4500"}}}
I0418 21:44:03.360369 1 gateway_monitor.go:320] Install/ensure SUBMARINER-GN-MARK chain exists
I0418 21:44:03.363316 1 gateway_monitor.go:132] On GW node starting all globalnet Datapath controllers
I0418 21:44:03.363398 1 gateway_monitor.go:230] On Gateway node - starting globalnet controllers
I0418 21:44:03.363406 1 gateway_monitor.go:332] Install/ensure SUBMARINER-GN-INGRESS chain exists
I0418 21:44:03.388267 1 gateway_monitor.go:343] Install/ensure SUBMARINER-GN-EGRESS chain exists
I0418 21:44:03.404241 1 gateway_monitor.go:349] Install/ensure SUBMARINER-POSTROUTING chain exists
I0418 21:44:03.423284 1 gateway_monitor.go:365] Install/ensure SM-GN-EGRESS-PODS chain exists
I0418 21:44:03.427458 1 gateway_monitor.go:371] Install/ensure SM-GN-EGRESS-HDLS-PODS chain exists
I0418 21:44:03.430861 1 gateway_monitor.go:377] Install/ensure SM-GN-EGRESS-NS chain exists
I0418 21:44:03.435075 1 gateway_monitor.go:383] Install/ensure SM-GN-EGRESS-CLUSTER chain exists
I0418 21:44:03.464277 1 node_controller.go:42] Creating Globalnet Node datapath controller for node cluster1-ha1-node
I0418 21:44:03.531777 1 node_controller.go:100] cniIfaceIP annotation on node "cluster1-ha1-node" is currently missing
I0418 21:44:03.545914 1 cluster_egressip_controller.go:42] Creating ClusterGlobalEgressIP dataplane controller
I0418 21:44:03.623987 1 datastoresyncer.go:339] Creating local submariner Cluster: types.SubmarinerCluster{ID:"cluster1-ha2-node", Spec:v1.ClusterSpec{ClusterID:"cluster1-ha2-node", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"10.43.0.0/16"}, ClusterCIDR:[]string{"10.42.0.0/16"}, GlobalCIDR:[]string{"242.1.0.0/16"}}}
I0418 21:44:03.633792 1 datastoresyncer.go:352] Creating local submariner Endpoint: types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1-ha2-node", CableName:"submariner-cable-cluster1-ha2-node-10-10-0-80", HealthCheckIP:"", Hostname:"cluster1-ha1-node", Subnets:[]string{"242.1.0.0/16"}, PrivateIP:"10.10.0.80", PublicIP:"35.171.157.41", NATEnabled:false, Backend:"vxlan", BackendConfig:map[string]string{"natt-discovery-port":"4490", "preferred-server":"false", "udp-port":"4500"}, AllocatedIPs:[]string(nil)}}
I0418 21:44:03.651408 1 node_controller.go:110] Processing local created Node "cluster1-ha1-node"
I0418 21:44:03.651440 1 node_controller.go:122] No cniIfaceIP allocated for node "cluster1-ha1-node", writing no rules for globalnet
I0418 21:44:03.654791 1 gateway_monitor.go:163] In processNextEndpoint, endpoint info: {"metadata":{"name":"cluster1-ha2-node-submariner-cable-cluster1-ha2-node-10-10-0-80","namespace":"submariner-operator","uid":"910a5b43-0a9b-4eb8-943f-6a0235cfc2a8","resourceVersion":"1712","generation":1,"creationTimestamp":"2022-04-18T21:44:03Z","managedFields":[{"manager":"submariner-gateway","operation":"Update","apiVersion":"submariner.io/v1","time":"2022-04-18T21:44:03Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{".":{},"f:backend":{},"f:backend_config":{".":{},"f:natt-discovery-port":{},"f:preferred-server":{},"f:udp-port":{}},"f:cable_name":{},"f:cluster_id":{},"f:hostname":{},"f:nat_enabled":{},"f:private_ip":{},"f:public_ip":{},"f:subnets":{}}}}]},"spec":{"cluster_id":"cluster1-ha2-node","cable_name":"submariner-cable-cluster1-ha2-node-10-10-0-80","hostname":"cluster1-ha1-node","subnets":["242.1.0.0/16"],"private_ip":"10.10.0.80","public_ip":"35.171.157.41","nat_enabled":false,"backend":"vxlan","backend_config":{"natt-discovery-port":"4490","preferred-server":"false","udp-port":"4500"}}}
I0418 21:44:03.749324 1 gateway_monitor.go:303] Successfully started the controllers
I0418 21:44:04.085948 1 datastoresyncer.go:112] Datastore syncer started
I0418 21:44:04.360882 1 clusterglobalegressip_handler.go:34] Update called for globalegressIP cluster1-ha2-node-cluster-egress.submariner.io datastoreSyncer's Hostname cluster1-ha1-node
I0418 21:44:04.861042 1 clusterglobalegressip_handler.go:34] Update called for globalegressIP cluster1-ha2-node-cluster-egress.submariner.io datastoreSyncer's Hostname cluster1-ha1-node
I0418 21:44:04.878126 1 gateway_monitor.go:163] In processNextEndpoint, endpoint info: {"metadata":{"name":"cluster1-ha2-node-submariner-cable-cluster1-ha2-node-10-10-0-179","namespace":"submariner-operator","uid":"8ba30470-729d-427e-b738-022cebf77096","resourceVersion":"1738","generation":2,"creationTimestamp":"2022-04-18T21:44:02Z","managedFields":[{"manager":"submariner-gateway","operation":"Update","apiVersion":"submariner.io/v1","time":"2022-04-18T21:44:04Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{".":{},"f:allocated_ips":{},"f:backend":{},"f:backend_config":{".":{},"f:natt-discovery-port":{},"f:preferred-server":{},"f:udp-port":{}},"f:cable_name":{},"f:cluster_id":{},"f:hostname":{},"f:nat_enabled":{},"f:private_ip":{},"f:public_ip":{},"f:subnets":{}}}}]},"spec":{"cluster_id":"cluster1-ha2-node","cable_name":"submariner-cable-cluster1-ha2-node-10-10-0-179","hostname":"cluster1-ha2-node","subnets":["242.1.0.0/16"],"private_ip":"10.10.0.179","public_ip":"18.234.103.8","nat_enabled":false,"backend":"vxlan","backend_config":{"natt-discovery-port":"4490","preferred-server":"false","udp-port":"4500"},"allocated_ips":["242.1.0.1","242.1.0.2","242.1.0.3","242.1.0.4","242.1.0.5","242.1.0.6","242.1.0.7","242.1.0.8"]}}
I0418 21:44:04.946615 1 clusterglobalegressip_handler.go:34] Update called for globalegressIP cluster1-ha2-node-cluster-egress.submariner.io datastoreSyncer's Hostname cluster1-ha1-node
I0418 21:44:05.314383 1 node_handler.go:66] areNodesEquivalent called for "cluster1-ha1-node", existingGlobalIP "242.0.255.254", newGlobalIP "242.0.255.254"
I0418 21:44:05.328978 1 node_controller.go:110] Processing local updated Node "cluster1-ha1-node"
I0418 21:44:05.329000 1 node_controller.go:132] Adding ingress rules for node "cluster1-ha1-node" with global IP 242.0.255.254, CNI IP 10.42.0.0
I0418 21:44:05.329029 1 iptables.go:319] Installing iptable ingress rules for Node: -p icmp -d 242.0.255.254 -j DNAT --to 10.42.0.0
I0418 21:44:06.457273 1 cluster_egressip_controller.go:88] Processing created ClusterGlobalEgressIP "cluster1-ha1-node-cluster-egress.submariner.io", Spec.NumberOfIPs: 8, Status: v1.GlobalEgressIPStatus{Conditions:[]v1.Condition(nil), AllocatedIPs:[]string(nil)}
I0418 21:44:06.457316 1 cluster_egressip_controller.go:111] Event received for "cluster1-ha1-node-cluster-egress.submariner.io", but numberOfIPs 8 are not yet allocated by globalnet controller
I0418 21:44:06.457447 1 clusterglobalegressip_handler.go:34] Update called for globalegressIP cluster1-ha1-node-cluster-egress.submariner.io datastoreSyncer's Hostname cluster1-ha1-node
I0418 21:44:07.067480 1 clusterglobalegressip_handler.go:34] Update called for globalegressIP cluster1-ha1-node-cluster-egress.submariner.io datastoreSyncer's Hostname cluster1-ha1-node
I0418 21:44:07.067517 1 clusterglobalegressip_handler.go:44] Updating the endpoint AllocatedIPs to [242.1.0.9 242.1.0.10 242.1.0.11 242.1.0.12 242.1.0.13 242.1.0.14 242.1.0.15 242.1.0.16]
I0418 21:44:07.067542 1 datastoresyncer.go:352] Creating local submariner Endpoint: types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1-ha2-node", CableName:"submariner-cable-cluster1-ha2-node-10-10-0-80", HealthCheckIP:"", Hostname:"cluster1-ha1-node", Subnets:[]string{"242.1.0.0/16"}, PrivateIP:"10.10.0.80", PublicIP:"35.171.157.41", NATEnabled:false, Backend:"vxlan", BackendConfig:map[string]string{"natt-discovery-port":"4490", "preferred-server":"false", "udp-port":"4500"}, AllocatedIPs:[]string{"242.1.0.9", "242.1.0.10", "242.1.0.11", "242.1.0.12", "242.1.0.13", "242.1.0.14", "242.1.0.15", "242.1.0.16"}}}
I0418 21:44:07.068601 1 cluster_egressip_controller.go:88] Processing updated ClusterGlobalEgressIP "cluster1-ha1-node-cluster-egress.submariner.io", Spec.NumberOfIPs: 8, Status: v1.GlobalEgressIPStatus{Conditions:[]v1.Condition{v1.Condition{Type:"Allocated", Status:"False", ObservedGeneration:0, LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:63785915046, loc:(*time.Location)(0x236ee60)}}, Reason:"NoDatapathRules", Message:"Allocated 8 global IP(s)"}}, AllocatedIPs:[]string{"242.1.0.9", "242.1.0.10", "242.1.0.11", "242.1.0.12", "242.1.0.13", "242.1.0.14", "242.1.0.15", "242.1.0.16"}}
I0418 21:44:07.068643 1 base_controllers.go:72] Flushing Iptables rules for previously allocated IPs [242.1.0.9 242.1.0.10 242.1.0.11 242.1.0.12 242.1.0.13 242.1.0.14 242.1.0.15 242.1.0.16] for "cluster1-ha1-node-cluster-egress.submariner.io"
I0418 21:44:07.068674 1 iptables.go:228] Deleting iptable egress rules for Cluster: -p all -s 10.42.0.0/16 -m mark --mark 0xC0000/0xC0000 -j SNAT --to 242.1.0.9-242.1.0.16
I0418 21:44:07.073078 1 iptables.go:228] Deleting iptable egress rules for Cluster: -p all -s 10.43.0.0/16 -m mark --mark 0xC0000/0xC0000 -j SNAT --to 242.1.0.9-242.1.0.16
I0418 21:44:07.075867 1 iptables.go:217] Installing iptable egress rules for Cluster: -p all -s 10.42.0.0/16 -m mark --mark 0xC0000/0xC0000 -j SNAT --to 242.1.0.9-242.1.0.16
I0418 21:44:07.082158 1 gateway_monitor.go:163] In processNextEndpoint, endpoint info: {"metadata":{"name":"cluster1-ha2-node-submariner-cable-cluster1-ha2-node-10-10-0-80","namespace":"submariner-operator","uid":"910a5b43-0a9b-4eb8-943f-6a0235cfc2a8","resourceVersion":"1788","generation":2,"creationTimestamp":"2022-04-18T21:44:03Z","managedFields":[{"manager":"submariner-gateway","operation":"Update","apiVersion":"submariner.io/v1","time":"2022-04-18T21:44:07Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{".":{},"f:allocated_ips":{},"f:backend":{},"f:backend_config":{".":{},"f:natt-discovery-port":{},"f:preferred-server":{},"f:udp-port":{}},"f:cable_name":{},"f:cluster_id":{},"f:hostname":{},"f:nat_enabled":{},"f:private_ip":{},"f:public_ip":{},"f:subnets":{}}}}]},"spec":{"cluster_id":"cluster1-ha2-node","cable_name":"submariner-cable-cluster1-ha2-node-10-10-0-80","hostname":"cluster1-ha1-node","subnets":["242.1.0.0/16"],"private_ip":"10.10.0.80","public_ip":"35.171.157.41","nat_enabled":false,"backend":"vxlan","backend_config":{"natt-discovery-port":"4490","preferred-server":"false","udp-port":"4500"},"allocated_ips":["242.1.0.9","242.1.0.10","242.1.0.11","242.1.0.12","242.1.0.13","242.1.0.14","242.1.0.15","242.1.0.16"]}}
I0418 21:44:07.097690 1 iptables.go:217] Installing iptable egress rules for Cluster: -p all -s 10.43.0.0/16 -m mark --mark 0xC0000/0xC0000 -j SNAT --to 242.1.0.9-242.1.0.16
I0418 21:44:07.111866 1 base_controllers.go:109] Updated: &v1.GlobalEgressIPStatus{Conditions:[]v1.Condition{v1.Condition{Type:"Allocated", Status:"False", ObservedGeneration:0, LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:63785915046, loc:(*time.Location)(0x236ee60)}}, Reason:"NoDatapathRules", Message:"Allocated 8 global IP(s)"}, v1.Condition{Type:"Allocated", Status:"True", ObservedGeneration:0, LastTransitionTime:v1.Time{Time:time.Time{wall:0xc08f94c9c6aabd82, ext:4310451494, loc:(*time.Location)(0x236ee60)}}, Reason:"DatapathRuleWriteSuccess", Message:"Allocated 8 global IP(s)"}}, AllocatedIPs:[]string{"242.1.0.9", "242.1.0.10", "242.1.0.11", "242.1.0.12", "242.1.0.13", "242.1.0.14", "242.1.0.15", "242.1.0.16"}}
I0418 21:44:07.157341 1 clusterglobalegressip_handler.go:34] Update called for globalegressIP cluster1-ha1-node-cluster-egress.submariner.io datastoreSyncer's Hostname cluster1-ha1-node
I0418 21:44:07.162983 1 cluster_egressip_controller.go:88] Processing updated ClusterGlobalEgressIP "cluster1-ha1-node-cluster-egress.submariner.io", Spec.NumberOfIPs: 8, Status: v1.GlobalEgressIPStatus{Conditions:[]v1.Condition{v1.Condition{Type:"Allocated", Status:"False", ObservedGeneration:0, LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:63785915046, loc:(*time.Location)(0x236ee60)}}, Reason:"NoDatapathRules", Message:"Allocated 8 global IP(s)"}, v1.Condition{Type:"Allocated", Status:"True", ObservedGeneration:0, LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:63785915047, loc:(*time.Location)(0x236ee60)}}, Reason:"DatapathRuleWriteSuccess", Message:"Allocated 8 global IP(s)"}}, AllocatedIPs:[]string{"242.1.0.9", "242.1.0.10", "242.1.0.11", "242.1.0.12", "242.1.0.13", "242.1.0.14", "242.1.0.15", "242.1.0.16"}}
I0418 21:44:07.163038 1 base_controllers.go:72] Flushing Iptables rules for previously allocated IPs [242.1.0.9 242.1.0.10 242.1.0.11 242.1.0.12 242.1.0.13 242.1.0.14 242.1.0.15 242.1.0.16] for "cluster1-ha1-node-cluster-egress.submariner.io"
I0418 21:44:07.163057 1 iptables.go:228] Deleting iptable egress rules for Cluster: -p all -s 10.42.0.0/16 -m mark --mark 0xC0000/0xC0000 -j SNAT --to 242.1.0.9-242.1.0.16
I0418 21:44:07.165922 1 iptables.go:228] Deleting iptable egress rules for Cluster: -p all -s 10.43.0.0/16 -m mark --mark 0xC0000/0xC0000 -j SNAT --to 242.1.0.9-242.1.0.16
I0418 21:44:07.168037 1 iptables.go:217] Installing iptable egress rules for Cluster: -p all -s 10.42.0.0/16 -m mark --mark 0xC0000/0xC0000 -j SNAT --to 242.1.0.9-242.1.0.16
I0418 21:44:07.175117 1 iptables.go:217] Installing iptable egress rules for Cluster: -p all -s 10.43.0.0/16 -m mark --mark 0xC0000/0xC0000 -j SNAT --to 242.1.0.9-242.1.0.16
I0418 21:44:08.275444 1 gateway_monitor.go:163] In processNextEndpoint, endpoint info: {"metadata":{"name":"cluster1-ha2-node-submariner-cable-cluster1-ha2-node-10-10-0-179","namespace":"submariner-operator","uid":"8ba30470-729d-427e-b738-022cebf77096","resourceVersion":"1808","generation":3,"creationTimestamp":"2022-04-18T21:44:02Z","managedFields":[{"manager":"submariner-gateway","operation":"Update","apiVersion":"submariner.io/v1","time":"2022-04-18T21:44:08Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{".":{},"f:allocated_ips":{},"f:backend":{},"f:backend_config":{".":{},"f:natt-discovery-port":{},"f:preferred-server":{},"f:udp-port":{}},"f:cable_name":{},"f:cluster_id":{},"f:healthCheckIP":{},"f:hostname":{},"f:nat_enabled":{},"f:private_ip":{},"f:public_ip":{},"f:subnets":{}}}}]},"spec":{"cluster_id":"cluster1-ha2-node","cable_name":"submariner-cable-cluster1-ha2-node-10-10-0-179","healthCheckIP":"242.1.255.254","hostname":"cluster1-ha2-node","subnets":["242.1.0.0/16"],"private_ip":"10.10.0.179","public_ip":"18.234.103.8","nat_enabled":false,"backend":"vxlan","backend_config":{"natt-discovery-port":"4490","preferred-server":"false","udp-port":"4500"},"allocated_ips":["242.1.0.1","242.1.0.2","242.1.0.3","242.1.0.4","242.1.0.5","242.1.0.6","242.1.0.7","242.1.0.8"]}}
I0418 21:44:11.054403 1 node_handler.go:66] areNodesEquivalent called for "cluster1-ha1-node", existingGlobalIP "242.0.255.254", newGlobalIP ""
I0418 21:44:11.061875 1 node_controller.go:110] Processing local updated Node "cluster1-ha1-node"
I0418 21:44:11.061902 1 node_controller.go:128] No global IP allocated for node "cluster1-ha1-node", writing no rules for globalnet
I0418 21:44:12.054123 1 node_handler.go:66] areNodesEquivalent called for "cluster1-ha1-node", existingGlobalIP "", newGlobalIP "242.1.255.253"
I0418 21:44:12.059052 1 node_controller.go:110] Processing local updated Node "cluster1-ha1-node"
I0418 21:44:12.059073 1 node_controller.go:132] Adding ingress rules for node "cluster1-ha1-node" with global IP 242.1.255.253, CNI IP 10.42.0.0
I0418 21:44:12.059082 1 iptables.go:319] Installing iptable ingress rules for Node: -p icmp -d 242.1.255.253 -j DNAT --to 10.42.0.0
I0418 21:44:12.059442 1 node_handler.go:74] Updating the endpoint HealthCheckIP to globalIP "242.1.255.253"
I0418 21:44:12.059456 1 datastoresyncer.go:352] Creating local submariner Endpoint: types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1-ha2-node", CableName:"submariner-cable-cluster1-ha2-node-10-10-0-80", HealthCheckIP:"242.1.255.253", Hostname:"cluster1-ha1-node", Subnets:[]string{"242.1.0.0/16"}, PrivateIP:"10.10.0.80", PublicIP:"35.171.157.41", NATEnabled:false, Backend:"vxlan", BackendConfig:map[string]string{"natt-discovery-port":"4490", "preferred-server":"false", "udp-port":"4500"}, AllocatedIPs:[]string{"242.1.0.9", "242.1.0.10", "242.1.0.11", "242.1.0.12", "242.1.0.13", "242.1.0.14", "242.1.0.15", "242.1.0.16"}}}
I0418 21:44:12.075723 1 gateway_monitor.go:163] In processNextEndpoint, endpoint info: {"metadata":{"name":"cluster1-ha2-node-submariner-cable-cluster1-ha2-node-10-10-0-80","namespace":"submariner-operator","uid":"910a5b43-0a9b-4eb8-943f-6a0235cfc2a8","resourceVersion":"1819","generation":3,"creationTimestamp":"2022-04-18T21:44:03Z","managedFields":[{"manager":"submariner-gateway","operation":"Update","apiVersion":"submariner.io/v1","time":"2022-04-18T21:44:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{".":{},"f:allocated_ips":{},"f:backend":{},"f:backend_config":{".":{},"f:natt-discovery-port":{},"f:preferred-server":{},"f:udp-port":{}},"f:cable_name":{},"f:cluster_id":{},"f:healthCheckIP":{},"f:hostname":{},"f:nat_enabled":{},"f:private_ip":{},"f:public_ip":{},"f:subnets":{}}}}]},"spec":{"cluster_id":"cluster1-ha2-node","cable_name":"submariner-cable-cluster1-ha2-node-10-10-0-80","healthCheckIP":"242.1.255.253","hostname":"cluster1-ha1-node","subnets":["242.1.0.0/16"],"private_ip":"10.10.0.80","public_ip":"35.171.157.41","nat_enabled":false,"backend":"vxlan","backend_config":{"natt-discovery-port":"4490","preferred-server":"false","udp-port":"4500"},"allocated_ips":["242.1.0.9","242.1.0.10","242.1.0.11","242.1.0.12","242.1.0.13","242.1.0.14","242.1.0.15","242.1.0.16"]}}
I0418 21:44:13.054432 1 node_handler.go:66] areNodesEquivalent called for "cluster1-ha1-node", existingGlobalIP "242.1.255.253", newGlobalIP "242.1.255.254"
I0418 21:44:13.059948 1 node_handler.go:74] Updating the endpoint HealthCheckIP to globalIP "242.1.255.254"
I0418 21:44:13.059956 1 node_controller.go:110] Processing local updated Node "cluster1-ha1-node"
I0418 21:44:13.059974 1 node_controller.go:132] Adding ingress rules for node "cluster1-ha1-node" with global IP 242.1.255.254, CNI IP 10.42.0.0
I0418 21:44:13.059982 1 iptables.go:319] Installing iptable ingress rules for Node: -p icmp -d 242.1.255.254 -j DNAT --to 10.42.0.0
I0418 21:44:13.059967 1 datastoresyncer.go:352] Creating local submariner Endpoint: types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1-ha2-node", CableName:"submariner-cable-cluster1-ha2-node-10-10-0-80", HealthCheckIP:"242.1.255.254", Hostname:"cluster1-ha1-node", Subnets:[]string{"242.1.0.0/16"}, PrivateIP:"10.10.0.80", PublicIP:"35.171.157.41", NATEnabled:false, Backend:"vxlan", BackendConfig:map[string]string{"natt-discovery-port":"4490", "preferred-server":"false", "udp-port":"4500"}, AllocatedIPs:[]string{"242.1.0.9", "242.1.0.10", "242.1.0.11", "242.1.0.12", "242.1.0.13", "242.1.0.14", "242.1.0.15", "242.1.0.16"}}}
I0418 21:44:13.077066 1 gateway_monitor.go:163] In processNextEndpoint, endpoint info: {"metadata":{"name":"cluster1-ha2-node-submariner-cable-cluster1-ha2-node-10-10-0-80","namespace":"submariner-operator","uid":"910a5b43-0a9b-4eb8-943f-6a0235cfc2a8","resourceVersion":"1824","generation":4,"creationTimestamp":"2022-04-18T21:44:03Z","managedFields":[{"manager":"submariner-gateway","operation":"Update","apiVersion":"submariner.io/v1","time":"2022-04-18T21:44:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{".":{},"f:allocated_ips":{},"f:backend":{},"f:backend_config":{".":{},"f:natt-discovery-port":{},"f:preferred-server":{},"f:udp-port":{}},"f:cable_name":{},"f:cluster_id":{},"f:healthCheckIP":{},"f:hostname":{},"f:nat_enabled":{},"f:private_ip":{},"f:public_ip":{},"f:subnets":{}}}}]},"spec":{"cluster_id":"cluster1-ha2-node","cable_name":"submariner-cable-cluster1-ha2-node-10-10-0-80","healthCheckIP":"242.1.255.254","hostname":"cluster1-ha1-node","subnets":["242.1.0.0/16"],"private_ip":"10.10.0.80","public_ip":"35.171.157.41","nat_enabled":false,"backend":"vxlan","backend_config":{"natt-discovery-port":"4490","preferred-server":"false","udp-port":"4500"},"allocated_ips":["242.1.0.9","242.1.0.10","242.1.0.11","242.1.0.12","242.1.0.13","242.1.0.14","242.1.0.15","242.1.0.16"]}}
I0418 21:44:15.455366 1 node_handler.go:66] areNodesEquivalent called for "cluster1-ha1-node", existingGlobalIP "242.1.255.254", newGlobalIP ""
I0418 21:44:15.459026 1 node_controller.go:110] Processing local updated Node "cluster1-ha1-node"
I0418 21:44:15.459045 1 node_controller.go:128] No global IP allocated for node "cluster1-ha1-node", writing no rules for globalnet
I0418 21:44:16.675288 1 node_handler.go:66] areNodesEquivalent called for "cluster1-ha1-node", existingGlobalIP "", newGlobalIP "242.1.255.254"
I0418 21:44:16.683053 1 node_controller.go:110] Processing local updated Node "cluster1-ha1-node"
I0418 21:44:16.683121 1 node_controller.go:132] Adding ingress rules for node "cluster1-ha1-node" with global IP 242.1.255.254, CNI IP 10.42.0.0
I0418 21:44:16.683150 1 iptables.go:319] Installing iptable ingress rules for Node: -p icmp -d 242.1.255.254 -j DNAT --to 10.42.0.0
W0418 21:44:23.265857 1 syncer.go:169] stale gateway: cluster1-ha2-node, didn't report for 15s
W0418 21:44:23.270327 1 syncer.go:176] Deleted stale gateway: cluster1-ha2-node, didn't report for 15s
I0418 21:44:25.044522 1 node_handler.go:66] areNodesEquivalent called for "cluster1-ha1-node", existingGlobalIP "242.1.255.254", newGlobalIP "242.1.255.254"
W0418 21:44:43.310809 1 syncer.go:169] stale gateway: cluster1-ha2-node, didn't report for 15s
W0418 21:44:43.316415 1 syncer.go:176] Deleted stale gateway: cluster1-ha2-node, didn't report for 15s
W0418 21:45:03.356278 1 syncer.go:169] stale gateway: cluster1-ha2-node, didn't report for 15s
W0418 21:45:03.362979 1 syncer.go:176] Deleted stale gateway: cluster1-ha2-node, didn't report for 15s
W0418 21:45:23.400850 1 syncer.go:169] stale gateway: cluster1-ha2-node, didn't report for 15s
W0418 21:45:23.406137 1 syncer.go:176] Deleted stale gateway: cluster1-ha2-node, didn't report for 15s
W0418 21:45:43.480383 1 syncer.go:169] stale gateway: cluster1-ha2-node, didn't report for 15s
W0418 21:45:43.485628 1 syncer.go:176] Deleted stale gateway: cluster1-ha2-node, didn't report for 15s
W0418 21:46:03.523459 1 syncer.go:169] stale gateway: cluster1-ha2-node, didn't report for 15s
W0418 21:46:03.528619 1 syncer.go:176] Deleted stale gateway: cluster1-ha2-node, didn't report for 15s
W0418 21:46:23.564140 1 syncer.go:169] stale gateway: cluster1-ha2-node, didn't report for 15s
W0418 21:46:23.569008 1 syncer.go:176] Deleted stale gateway: cluster1-ha2-node, didn't report for 15s
W0418 21:46:43.603372 1 syncer.go:169] stale gateway: cluster1-ha2-node, didn't report for 15s
W0418 21:46:43.608329 1 syncer.go:176] Deleted stale gateway: cluster1-ha2-node, didn't report for 15s
W0418 21:47:03.642608 1 syncer.go:169] stale gateway: cluster1-ha2-node, didn't report for 15s
W0418 21:47:03.647621 1 syncer.go:176] Deleted stale gateway: cluster1-ha2-node, didn't report for 15s
W0418 21:47:23.681579 1 syncer.go:169] stale gateway: cluster1-ha2-node, didn't report for 15s
W0418 21:47:23.686229 1 syncer.go:176] Deleted stale gateway: cluster1-ha2-node, didn't report for 15s
W0418 21:47:43.719227 1 syncer.go:169] stale gateway: cluster1-ha2-node, didn't report for 15s
W0418 21:47:43.724248 1 syncer.go:176] Deleted stale gateway: cluster1-ha2-node, didn't report for 15s
W0418 21:48:03.759206 1 syncer.go:169] stale gateway: cluster1-ha2-node, didn't report for 15s
W0418 21:48:03.764040 1 syncer.go:176] Deleted stale gateway: cluster1-ha2-node, didn't report for 15s
W0418 21:48:23.799692 1 syncer.go:169] stale gateway: cluster1-ha2-node, didn't report for 15s
W0418 21:48:23.804001 1 syncer.go:176] Deleted stale gateway: cluster1-ha2-node, didn't report for 15s
W0418 21:48:43.841314 1 syncer.go:169] stale gateway: cluster1-ha2-node, didn't report for 15s
W0418 21:48:43.851164 1 syncer.go:176] Deleted stale gateway: cluster1-ha2-node, didn't report for 15s
W0418 21:49:03.895382 1 syncer.go:169] stale gateway: cluster1-ha2-node, didn't report for 15s
W0418 21:49:03.901737 1 syncer.go:176] Deleted stale gateway: cluster1-ha2-node, didn't report for 15s