Work in progress to get "hack/local-up-cluster.sh" working with a Dockerized "kubelet" on Docker for Mac.

local-up-cluster.sh.diff

diff --git a/hack/local-up-cluster.sh b/hack/local-up-cluster.sh
index 92b13eab89..9a85c463c5 100755
--- a/hack/local-up-cluster.sh
+++ b/hack/local-up-cluster.sh
@@ -22,12 +22,14 @@ KUBE_ROOT=$(dirname "${BASH_SOURCE}")/..
 DOCKER_OPTS=${DOCKER_OPTS:-""}
 DOCKER=(docker ${DOCKER_OPTS})
 DOCKERIZE_KUBELET=${DOCKERIZE_KUBELET:-""}
+DOCKERIZED_KUBELET_IMAGE=${DOCKERIZED_KUBELET_IMAGE:-"gcr.io/google_containers/kubelet:local-up-cluster"}
 ALLOW_PRIVILEGED=${ALLOW_PRIVILEGED:-""}
 ALLOW_SECURITY_CONTEXT=${ALLOW_SECURITY_CONTEXT:-""}
 PSP_ADMISSION=${PSP_ADMISSION:-""}
 RUNTIME_CONFIG=${RUNTIME_CONFIG:-""}
 KUBELET_AUTHORIZATION_WEBHOOK=${KUBELET_AUTHORIZATION_WEBHOOK:-""}
 KUBELET_AUTHENTICATION_WEBHOOK=${KUBELET_AUTHENTICATION_WEBHOOK:-""}
+KUBELET_ROOT=${KUBELET_ROOT:-"/var/lib/kubelet"}
 # Name of the network plugin, eg: "kubenet"
 NET_PLUGIN=${NET_PLUGIN:-""}
 # Place the binaries required by NET_PLUGIN in this directory, eg: "/home/kubernetes/bin".
@@ -272,7 +274,10 @@ cleanup_dockerized_kubelet()
 {
   if [[ -e $KUBELET_CIDFILE ]]; then
     docker kill $(<$KUBELET_CIDFILE) > /dev/null
+    docker rm $(<$KUBELET_CIDFILE) > /dev/null
     rm -f $KUBELET_CIDFILE
+    docker rmi -f ${DOCKERIZED_KUBELET_IMAGE}
+    rm -fR ${DOCKERIZED_KUBELET_CONTEXT_DIR}
   fi
 }
 
@@ -471,7 +476,14 @@ function start_kubelet {
       priv_arg="--allow-privileged "
     fi
 
-    mkdir -p /var/lib/kubelet
+    # Create the kubelet root
+    mkdir -p ${KUBELET_ROOT}
+
+    # On macOS/OSX, we always run the kubelet in Docker because it cannot run natively.
+    if [[ "$(uname)" == "Darwin" ]]; then
+        DOCKERIZE_KUBELET="true"
+    fi
+
     if [[ -z "${DOCKERIZE_KUBELET}" ]]; then
       # Enable dns
       if [[ "${ENABLE_CLUSTER_DNS}" = true ]]; then
@@ -542,11 +554,11 @@ function start_kubelet {
         --port="$KUBELET_PORT" >"${KUBELET_LOG}" 2>&1 &
       KUBELET_PID=$!
       # Quick check that kubelet is running.
-      if ps -p $KUBELET_PID > /dev/null ; then 
+      if ps -p $KUBELET_PID > /dev/null ; then
 	echo "kubelet ( $KUBELET_PID ) is running."
       else
 	cat ${KUBELET_LOG} ; exit 1
-      fi	
+      fi
     else
       # Docker won't run a container with a cidfile (container id file)
       # unless that file does not already exist; clean up an existing
@@ -568,20 +580,84 @@ function start_kubelet {
           cred_bind="--volume=${cloud_cred}:${cloud_cred}:ro"
       fi
 
+      export DOCKERIZED_KUBELET_CONTEXT_DIR=${TMP:-"/tmp"}/dockerized_kublet
+      local dockerized_kubelet=${KUBE_ROOT}/_output/dockerized/bin/linux/amd64/kubelet
+
+      # To run the kubelet in container, we first have to build the container.
+      # This means we need to setup a Docker context directory that allows us
+      # to build a functioning container.  To facilitate this, we need to
+      # ensure that the Dockerized kubelet has been built first.
+      if [ ! -f ${dockerized_kubelet} ]; then
+          echo "Please run 'make quick-release' and/or make sure ${dockerized_kubelet} is built"
+          exit 1
+      fi
+
+      # TODO: Do this stuff conditionally
+
+      # Create the Docker context directory
+      mkdir -p ${DOCKERIZED_KUBELET_CONTEXT_DIR}/kubernetes
+      # Copy the Dockerized kubelet
+      cp ${dockerized_kubelet} ${DOCKERIZED_KUBELET_CONTEXT_DIR}
+      # Copy over the Kubernetes configuration files
+      sudo cp -R /var/run/kubernetes/* ${DOCKERIZED_KUBELET_CONTEXT_DIR}/kubernetes/
+      # Change ownership of the Kubernetes configuration files to allow Docker to copy them
+      sudo chown -R ${USER} ${DOCKERIZED_KUBELET_CONTEXT_DIR}/kubernetes/
+      # Create the Dockerfile
+      cat <<'EOF' > ${DOCKERIZED_KUBELET_CONTEXT_DIR}/Dockerfile
+FROM centos
+
+ADD kubelet /kubelet
+ADD kubernetes/* /var/run/kubernetes/
+
+VOLUME /var/lib/docker
+VOLUME /var/run/kubernetes
+
+RUN chmod a+rx /kubelet
+
+CMD [ "/kubelet" ]
+EOF
+      # Change to the Docker context directory
+      pushd ${DOCKERIZED_KUBELET_CONTEXT_DIR} > /dev/null
+      # Build kubelet Docker image
+      docker build -t ${DOCKERIZED_KUBELET_IMAGE} . &> $KUBELET_LOG
+
+      # Make sure we successfully built the Docker image
+      if [ $? -ne 0 ]; then
+          echo "Unable to build the kubelet Docker image"
+          exit 1
+      fi
+
+      # Change back to the original directory
+      popd > /dev/null
+
       docker run \
         --volume=/:/rootfs:ro \
-        --volume=/var/run:/var/run:rw \
         --volume=/sys:/sys:ro \
-        --volume=/var/lib/docker/:/var/lib/docker:ro \
-        --volume=/var/lib/kubelet/:/var/lib/kubelet:rw \
-        --volume=/dev:/dev \
+        --volume=/var/lib/docker:/var/lib/docker:rw \
+        --volume=${KUBELET_ROOT}:${KUBELET_ROOT}:rw \
+        --volume=/var/run:/var/run:rw \
         ${cred_bind} \
-        --net=host \
-        --privileged=true \
-        -i \
+        --cap-add=SYS_ADMIN \
+        --interactive \
+        --network=host \
+        --privileged \
         --cidfile=$KUBELET_CIDFILE \
-        gcr.io/google_containers/kubelet \
-        /kubelet --v=${LOG_LEVEL} --containerized ${priv_arg}--chaos-chance="${CHAOS_CHANCE}" --hostname-override="${HOSTNAME_OVERRIDE}" --cloud-provider="${CLOUD_PROVIDER}" --cloud-config="${CLOUD_CONFIG}" \ --address="127.0.0.1" --require-kubeconfig --kubeconfig "$CERT_DIR"/kubelet.kubeconfig --api-servers="https://${API_HOST}:${API_SECURE_PORT}" --port="$KUBELET_PORT"  --enable-controller-attach-detach="${ENABLE_CONTROLLER_ATTACH_DETACH}" &> $KUBELET_LOG &
+        ${DOCKERIZED_KUBELET_IMAGE} \
+        /kubelet \
+          --address="127.0.0.1" \
+          --api-servers="https://localhost:${API_SECURE_PORT}" \
+          --cert-dir="${CERT_DIR}" \
+          --chaos-chance="${CHAOS_CHANCE}" \
+          --cloud-config="${CLOUD_CONFIG}" \
+          --cloud-provider="${CLOUD_PROVIDER}" \
+          --containerized ${priv_arg} \
+          --enable-controller-attach-detach="${ENABLE_CONTROLLER_ATTACH_DETACH}" \
+          --hostname-override="${HOSTNAME_OVERRIDE}" \
+          --kubeconfig "$CERT_DIR"/kubelet.kubeconfig \
+          --port="$KUBELET_PORT" \
+          --require-kubeconfig \
+          --root-dir=${KUBELET_ROOT} \
+          --v=${LOG_LEVEL} &> $KUBELET_LOG &
     fi
 }
 
@@ -621,7 +697,7 @@ function start_kubedns {
           sed -i -e "/{{ pillar\['federations_domain_map'\] }}/d" kubedns-deployment.yaml
         fi
         sed -e "s/{{ pillar\['dns_server'\] }}/${DNS_SERVER_IP}/g" "${KUBE_ROOT}/cluster/addons/dns/kubedns-svc.yaml.in" >| kubedns-svc.yaml
-        
+
         # TODO update to dns role once we have one.
         ${KUBECTL} --kubeconfig="${CERT_DIR}/admin.kubeconfig" create clusterrolebinding system:kube-dns --clusterrole=cluster-admin --serviceaccount=kube-system:default
         # use kubectl to create kubedns rc and service

Prior to these changes, hack/local-up-cluster.sh would attempt to look for a local Docker image named gcr.io/google_containers/kubelet:latest but this image was never built (ever) by any make target or hack/* script. So whenever you run hack/local-up-cluster.sh, things appear to work until you try to query the nodes or do much else because you'll get a "No resources found." These changes update hack/local-up-cluster.sh to do the following:

1. Sets DOCKERIZE_KUBELET to true when on macOS/OS X (reason being is kubelet does not run natively on macOS/OS X)
2. Builds the kubelet Docker image when DOCKERIZE_KUBELET is set
3. Minor changes to the docker run command for recent CLI changes

With these changes the kubelet Docker image appears to work because cluster/kubectl.sh get ... now returns actual results instead of the aforementioned "No resources found." But whenever you attempt deploy a pod, you'll see that the pod never deploys because when attempting to mount the service account token, it fails with something like this:

I0110 04:10:17.907225       1 reconciler.go:307] MountVolume operation started for volume "kubernetes.io/secret/ad3693ce-d6ea-11e6-99ba-9801a7a764ff-default-token-dmz26" (spec.Name: "default-token-dmz26") to pod "ad3693ce-d6ea-11e6-99ba-9801a7a764ff" (UID: "ad3693ce-d6ea-11e6-99ba-9801a7a764ff").
I0110 04:10:17.908896       1 secret.go:180] Setting up volume default-token-dmz26 for pod ad3693ce-d6ea-11e6-99ba-9801a7a764ff at /var/lib/kubelet/pods/ad3693ce-d6ea-11e6-99ba-9801a7a764ff/volumes/kubernetes.io~secret/default-token-dmz26
I0110 04:10:17.915586       1 nsenter_mount.go:188] Failed findmnt command for path /var/lib/kubelet/pods/ad3693ce-d6ea-11e6-99ba-9801a7a764ff/volumes/kubernetes.io~secret/default-token-dmz26: exit status 1

I also ran into a number of these as well:

E0110 04:48:10.477655       1 kubelet.go:1638] Failed to check if disk space is available on the root partition: failed to get fs info for "root": error trying to get filesystem Device for dir /var/lib/kubelet: err: could not find device with major: 0, minor: 39 in cached partitions map

I did a ton of digging and tried many different variations of Docker volume syntaxes and locations but no matter what, the secret never mounts into the container. I think this is related to Docker for Mac based on the digging I did I'm not 100% sure as of writing this. I did look into the nsenter requirements, and all have been verified as met other than possibly #5.

Any help would be appreciated and any progress I make will be shared as soon as I have it.

Update: Might not be specific to Docker for Mac as it's reported here for Linux: kubernetes/kubernetes#38337