Skip to content

Instantly share code, notes, and snippets.

@digitalronin

digitalronin/notes.md

Created November 4, 2020 12:31
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save digitalronin/140f390f02188698aaf760ae34387f8d to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save digitalronin/140f390f02188698aaf760ae34387f8d to your computer and use it in GitHub Desktop.
Download ZIP
Connection timeout errors investigation
Raw
notes.md

Theory: LB sends pods traffic after they've moved off a node

This would explain a few errors when the node recycler replaces a node which happens to have an ingress controller pod running on it.

Test

In a test cluster (3 worker nodes)

  • deploy a "hello world" ruby app

  • ensure fewer ingress controller pods than worker nodes (to ensure that, if we move a pod from a node, it will be missing if any web traffic arrives for it. i.e. we need traffic to hit a node which used to have an ingress controller pod on it, but which doesn't have one anymore)

kubectl -n ingress-controllers edit deployment nginx-ingress-acme-controller

Change replicas from 6 to 2

  • take a baseline - 10,000 requests to a helloworld app, concurrency 10
ab -n 10000 -c 10 https://helloworld-rubyapp.apps.cp-2810-1431.cloud-platform.service.justice.gov.uk/

Result:

0 failed requests
  • repeat the test, & immediately kill one of the ingress-controller pods
kubectl -n ingress-controllers get pods -o wide # to see which node it's on
kubectl -n ingress-controllers delete pod nginx-ingress-acme-controller-c9c8c7dd6-lqx6q
kubectl -n ingress-controllers get pods -o wide # to see if replacement is on a different node

If the replacement comes up on a different pod, then we should be simulating what happens when the node recycler replaces a node with an ingress controller pod running on it.

Example results:

apr_pollset_poll: The timeout specified has expired (70007)
Total of 79 requests completed

Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
apr_pollset_poll: The timeout specified has expired (70007)
Total of 3782 requests completed

This suggests the theory is correct, but we could just be seeing errors because one ingress controller pod is insufficient, so while the other pod is being replaced, we get errors. To eliminate this, repeat the test with only one ingress controller pod running, but editing the deployment and reducing replicas to 1.

I got a failure when I ran the ab test immediately after changing the replica count to 1, which also suggests the theory is correct - the LB was still trying to send traffic to the terminated ingress controller pod.

Wait for a while after making the change, to ensure that the terminated pod has been flagged as "unhealthy" by the load-balancer)

Result:

0 failed requests

Conclusion

I think this test validates the theory.

Even though there is always at least one ingress controller pod running, which is enough to handle all the requests which arrive during our test, we get timeout errors if we delete one of the pods and let kubernetes replace it. That means traffic is still arriving for the pod which isn't there anymore.

If we don't replace a pod during the test (or shortly before it starts), we don't get timeout errors, even if we only have one ingress controller pod running.

Next step

I suggest we do 2 things:

  1. Pause the node-recycler pipeline for a couple of days, and ask Miguel if this changes the number of these errors which occur.

  2. Depending on the outcome of 1, move ingress controllers to a separate instancegroup whose nodes don't get recycled.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment