Local Kubernetes setup on macOS with minikube on VirtualBox and local Docker registry

1_kubernetes_on_macOS.md

Requirements

Minikube requires that VT-x/AMD-v virtualization is enabled in BIOS. To check that this is enabled on OSX / macOS run:

sysctl -a | grep machdep.cpu.features | grep VMX

If there's output, you're good!

Prerequisites

• kubectl
• docker (for Mac)
• minikube
• virtualbox

brew update && brew install kubectl && brew install --cask docker virtualbox && brew install minikube

Verify

docker --version                # Docker version 17.09.0-ce, build afdb6d4
docker-compose --version        # docker-compose version 1.16.1, build 6d1ac21
docker-machine --version        # docker-machine version 0.12.2, build 9371605
minikube version                # minikube version: v0.22.3
kubectl version --client        # Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"f38e43b221d08850172a9a4ea785a86a3ffa3b3a", GitTreeState:"clean", BuildDate:"2017-10-12T00:45:05Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"darwin/amd64"}      

Start

minikube start

This can take a while, expected output:

Starting local Kubernetes cluster...
Kubectl is now configured to use the cluster.

Great! You now have a running Kubernetes cluster locally. Minikube started a virtual machine for you, and a Kubernetes cluster is now running in that VM.

Check k8s

kubectl get nodes

Should output something like:

NAME       STATUS    ROLES     AGE       VERSION
minikube   Ready     <none>    40s       v1.7.5

Use minikube's built-in docker daemon:

eval $(minikube docker-env)

Add this line to .bash_profile or .zshrc or ... if you want to use minikube's daemon by default (or if you do not want to set this every time you open a new terminal).

You can revert back to the host docker daemon by running:

eval $(docker-machine env -u)

If you now run docker ps, it should now output something like:

CONTAINER ID        IMAGE                                         COMMAND                 CREATED             STATUS              PORTS               NAMES
e97128790bf9        gcr.io/google-containers/kube-addon-manager   "/opt/kube-addons.sh"   22 seconds ago      Up 22 seconds                           k8s_kube-addon-manager_kube-addon-manager-minikube_kube-system_c654b2f084cf26941c334a2c3d6db53d_0
69707e54d1d0        gcr.io/google_containers/pause-amd64:3.0      "/pause"                33 seconds ago      Up 33 seconds                           k8s_POD_kube-addon-manager-minikube_kube-system_c654b2f084cf26941c334a2c3d6db53d_0

Build, deploy and run an image on your local k8s setup

First setup a local registry, so Kubernetes can pull the image(s) from there:

docker run -d -p 5000:5000 --restart=always --name registry registry:2

Build

First of, store all files (Dockerfile, my-app.yml, index.html) in this gist locally in some new (empty) directory.

You can build the Dockerfile below locally if you want to follow this guide to the letter. Store the Dockerfile locally, preferably in an empty directory and run:

docker build . --tag my-app

You should now have an image named 'my-app' locally, check by using docker images (or your own image of course). You can then publish it to your local docker registry:

docker tag my-app localhost:5000/my-app:0.1.0

Running docker images should now output the following:

REPOSITORY                                             TAG                 IMAGE ID            CREATED             SIZE
my-app                                                 latest              cc949ad8c8d3        44 seconds ago      89.3MB
localhost:5000/my-app                                  0.1.0               cc949ad8c8d3        44 seconds ago      89.3MB
httpd                                                  2.4-alpine          fe26194c0b94        7 days ago          89.3MB

Deploy and run

Store the file below my-app.yml on your system and run the following:

kubectl create -f my-app.yml

You should now see your pod and your service:

kubectl get all

The configuration exposes my-app outside of the cluster, you can get the address to access it by running:

minikube service my-app --url

This should give an output like http://192.168.99.100:30304 (the port will most likely differ). Go there with your favorite browser, you should see "Hello world!". You just accessed your application from outside of your local Kubernetes cluster!

Kubernetes GUI

minikube dashboard

Delete deployment of my-app

kubectl delete deploy my-app
kubectl delete service my-app

You're now good to go and deploy other images!

Reset everything

minikube stop;
minikube delete;
rm -rf ~/.minikube ~/.kube;
brew uninstall kubectl;
brew cask uninstall docker virtualbox minikube;

Version

Last tested on 2017 October 20th macOS Sierra 10.12.6

Dockerfile

# Just for demo purposes obviously
FROM httpd:2.4-alpine

COPY ./index.html /usr/local/apache2/htdocs/

index.html

Hello world!

my-app.yml

# APP DEPLOYMENT

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: my-app
  name: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      run: my-app-exposed
  template:
    metadata:
      labels:
        run: my-app-exposed
    spec:
      containers:
      - image: localhost:5000/my-app:0.1.0
        name: my-app
        ports:
        - containerPort: 80
          protocol: TCP

---

# APP SERVICE

apiVersion: v1
kind: Service
metadata:
  labels:
    run: my-app
  name: my-app
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: my-app-exposed
  type: NodePort

Don't forget to run:
eval "$(docker-machine env -u)"
to undo https://gist.github.com/kevin-smets/b91a34cea662d0c523968472a81788f7#use-minikubes-built-in-docker-daemon and
return to your default Docker Daemon

Good job! thank you very much

Good one thanks

If you're using this and minikube hangs at Starting cluster components..., you might be running into kubernetes/minikube#2765 (see also: kubernetes/minikube#2791). A workaround is to use the localkube bootstrapper by starting with minikube start --bootstrapper=localkube.

@kevin-smets Such a nice gist! Thanks a lot.

@michaelahlers I use the updatedest minikube , when use proxy,no longer hang with Starting cluster components.
no_proxy=192.168.99.100 https_proxy='http://:8080' minikube start --docker-env 'HTTP_PROXY=http://:8123' --docker-env 'HTTPS_PROXY=http://:8123'
First no any proxy,hang,then only https_proxy='http://:8080' minikube start,hang, then as above ,it work.

Runeval "$(docker-machine env -u)"to go back to your host docker daemon

Yeah - i spent a while without realising the eval $(minikube docker-env) needs to be run in any shell you're planning on using to interact with docker.

I was publishing to Docker Hub from the wrong console window and obviously my registry was running in minikube under different export values.

@17twenty good tip, I added a line about making this permanent if you want to do prevent from running into this.

You should also add docker-compose to the install command because this is not installed automatically with docker.

You are my hero

According to this SO answer you may not necessarily need to start a local registry with newer versions of the software...

typo. In the "Reset Everything" section, it should be:

rm -rf ~/.minikube ~/.kube;

Nice catch @johntellsall - fixed it!

Hi,

Below are my configuration version:

docker-compose version 1.22.0, build f46880f
docker-machine version 0.15.0, build b48dc28d
docker-machine version 0.15.0, build b48dc28d
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"clean", BuildDate:"2018-12-04T07:48:45Z", GoVersion:"go1.11.2", Compiler:"gc", `Platform:"darwin/amd64"}

I got following error when I start minikube.

E1212 08:21:57.797601 24610 start.go:281] Error restarting cluster: running cmd:
sudo kubeadm alpha phase certs all --config /var/lib/kubeadm.yaml &&
sudo /usr/bin/kubeadm alpha phase kubeconfig all --config /var/lib/kubeadm.yaml &&
sudo /usr/bin/kubeadm alpha phase controlplane all --config /var/lib/kubeadm.yaml &&
sudo /usr/bin/kubeadm alpha phase etcd local --config /var/lib/kubeadm.yaml
: Process exited with status 1

minikube stop;
minikube delete;
rm -rf ~/.minikube ~/.kube;

In some cases, with VBox manager installed, doing this can leave you with a VM in VirtualBox with no associated disk.
Attempting to restart minikube after will result in a vm failure.

[Workspace] $ minikube start
Starting local Kubernetes v1.12.4 cluster...
Starting VM...
E0111 09:54:45.302821   74151 start.go:187] Error starting host: Error starting stopped host: Unable to start the VM: /usr/local/bin/VBoxManage startvm minikube --type headless failed:
VBoxManage: error: UUID {4f939cca-1d1f-467a-9ad9-8c13b3847c47} of the medium '.../.minikube/machines/minikube/disk.vmdk' does not match the value {d4075459-35be-4715-a606-2c2fdc9d38ec} stored in the media registry ('/Users/.../Library/VirtualBox/VirtualBox.xml')
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component MediumWrap, interface IMedium

Details: 00:00:00.052189 Power up failed (vrc=VINF_SUCCESS, rc=NS_ERROR_FAILURE (0X80004005)).

You can remove the orphans from the the Vbox Manager GUI - or use VBoxManage unregistervm <vmname or guid> to clean these up if you have already run the stop/rm commands.

Make sure all your components are up to date. Running minikube delete should cleanly remove all components.

Isn't it missing a docker push after docker tag ?

Maybe I'm mistaking, but after tagging the image as localhost:5000, I need to push it to the localhost:5000 registry for it to be available in minikube :)

Thanks a lot for your gist ! Full of useful info. Great job.

Thanks so much.

Thank you, nice manual.
Works also on macOS High Sierra 10.13.6

Hi!
I just realise that you don't actually push the image to the registry. It works because you are still in the docker environment inside minikube.

If you do the docker push: docker push localhost:5000/my-app:0.1.0 and after that you do: curl http://localhost:5000/v2/_catalog you should see the image:
{"repositories":["my-app"]}

Or if you execute curl http://localhost:5000/v2/my-app/tags/list should show:
{"name":"my-app","tags":["0.1.0"]}

This is just a clarification. Thanks again for the gist! Very useful.

Nice guide!
Thank you!

Isn't it missing a docker push after docker tag ?

Maybe I'm mistaking, but after tagging the image as localhost:5000, I need to push it to the localhost:5000 registry for it to be available in minikube :)

@aesteve This is what I'm finding too. I have to push with docker push localhost:5000/my-app:0.1.0 or I will still get ErrImagePull.

@MihaiLupoiu just to make sure that I understand your comment: does it work for you without pushing?

I got errors ala invalid object doesn't have additional properties when applying kubectl create -f my-app.yml. I got it resolved it by following this answer: https://stackoverflow.com/questions/55417410/kubernetes-create-deployment-unexpected-schemaerror/55737973#55737973

Thanks for this gist.

Wow, you rock! You maybe saved me a very massive amount of time! There are sooooo many bad tutorials out there! :(

Thank you, You have made my day! 🙏

Hi,

Few small changes required in .yml file for latest vesion of kubernetes (1.16)

apiVersion: apps/v1
kind: Deployment

Update: Minikube is not delivered with cask right now. Change command brew cask install minikube to brew install minikube

Done, thanks for the heads up!

Another update: brew cask has been deprecated. so the cmd should be something similar to:

brew update && brew install kubectl && brew install docker virtualbox && brew install minikube

or with docker-compose and machine if it's also needed:

brew update && brew install kubectl && brew install docker docker-compose docker-machine virtualbox && brew install minikube

Thanks for this