Skip to content

Instantly share code, notes, and snippets.

@John-Lin

John-Lin/Kubeflow.md

Last active November 10, 2022 19:31
Show Gist options
  • Save John-Lin/c8083b4db75e97cf12943c545301acd8 to your computer and use it in GitHub Desktop.
Save John-Lin/c8083b4db75e97cf12943c545301acd8 to your computer and use it in GitHub Desktop.
Download ZIP
Kubeflow on Minikube
Raw
Kubeflow.md

Deploy Kubeflow on Minikube

Install minikube

Please see How to Minikube

Running a minikube with 4CPUs and 8GB memory

$ minikube start --cpus 4 --memory 8192

If you want to use the vm driver with hyperkit

$ minikube start --vm-driver=hyperkit --cpus 4 --memory 8192

Deploy Kubeflow

$ git clone [email protected]:google/kubeflow.git
$ cd kubeflow
$ kubectl apply -f components/ -R
# Wait until all pod are getting running state. (6-7 mins)

Get the URL for the notebook.

$ minikube service tf-hub-lb --url

or access JupyterHub on http://localhost:8000 by

$ kubectl port-forward tf-hub-0 8000:8000

User Login

input any username and password will login

Basic notebook with scipy in JupyterHub

Spawner options

Image : jupyter/scipy-notebook (jupyterhub version MUST be at least 0.8.1)

CPU: 200m (could be more)

Memory: 256Mi (could be more)

Extra Resource Limits: (leave it empty)

than press the Spawn button

##Tensorflow wit notebook in JupyterHub

Spawner options

Image : gcr.io/kubeflow/tensorflow-notebook-cpu

CPU: 300m (almost run out of quota in 4 CPU minikube)

Memory: 1.5Gi (could be more)

Extra Resource Limits: (leave it empty)

than press the Spawn button

Running a tensorflow example

See Neural Network Example

You may get this message, python3.5 still works. but we are in 3.6 wait until google fixed this issue. We could just ignore.

/home/raju/anaconda3/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6

See tensorflow/tensorflow#14182

CPU/Mem resource monitoring on K8S

$ kubectl describe node
@sfl0r3nz05
Copy link

sfl0r3nz05 commented Nov 10, 2022 •
edited
Loading

I am having the same issue that @leeadh. Is it known if it has been fixed?

Note: I am not able to clone the repo from [email protected]:google/kubeflow.git, so I have used https://github.com/kubeflow/kubeflow.git.

@phgSW61HS
Copy link

phgSW61HS commented Nov 10, 2022 •
edited
Loading

I am having the same issue that @leeadh. Is it known if it has been fixed?

Note: I am not able to clone the repo from [email protected]:google/kubeflow.git, so I have used https://github.com/kubeflow/kubeflow.git.

Hi guys. FYI I remember that I switched to a different approach which turned successful based on MicroK8s and multipass prior to installing KubeFlow. Here are a few links I used: https://itnext.io/microk8s-on-macos-98f1de3aa63e , https://microk8s.io/docs/install-multipass https://www.youtube.com/watch?v=KPEGKKNB63Q

Once you have installed ubuntu server VM's, it's pretty easy and the youtube video I added above is crystal clear. Just a small remark : running KF is very heavy in terms of resources (deployed around 40 pods just to run KF) As far as I remember when I deployed it locally on my macbook pro (4core 16Gb Ram) my laptop was freezing so much that the only thing I managed to do was browsing the KF web UI and printing the k8s objects in my terminal :D If you want to achieve advanced stuff with KF like ML pipelines... I would advise you to go to the cloud and provision either VMs (and of course manually install k8s through kubeadm or even microk8s) or AKS/GKE (unless you have machines at home you can use to build your own cluster... if so drop me a mail as I have already done it and I can share some docs tips and tricks)

@phgSW61HS
Copy link

phgSW61HS commented Nov 10, 2022 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment