Skip to content

Instantly share code, notes, and snippets.

@elsonrodriguez
Last active April 5, 2016 01:28
Show Gist options
  • Save elsonrodriguez/726e78754306a567885d7a00a08dd196 to your computer and use it in GitHub Desktop.
Save elsonrodriguez/726e78754306a567885d7a00a08dd196 to your computer and use it in GitHub Desktop.

Intro

This is the formula for terminating k8s nodes that have a bad kubelet. This effectively fences nodes that are unable to unmount/detach their own storage.

Caveats

Early testing shows that this may not work if a node is rebooted in response to a kubelet/system issue. Therefore the action taken in response to a problem on a kubernetes node MUST BE to terminate it.

Overview

Make LB like this:

            "HealthCheck": {
                "Timeout": 5, 
                "Target": "HTTP:10248/healthz", 
                "UnhealthyThreshold": 6, 
                "HealthyThreshold": 2, 
                "Interval": 15
            }, 

With a name like this:

            "LoadBalancerName": "kubelet-healthcheck", 

Then set this LB in your autoscaling group.

            "LoadBalancerNames": [
                "kubelet-healthcheck"
            ], 

And set this field

            "HealthCheckType": "ELB", 
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment