Google AI Platform Notebook tips

AI Notebooks vs Colab

Google Colab: https://colab.research.google.com/

Free compute! and interesting, fixed cost, 10$/month Google Colab Pro version.
Notebooks are very easy to share, with ability to control permissions (integration in google drive/doc ecosystem).
Instant availability. No requirement other than a regular google account.
Idle time limit : 1h in free version.
Storage is not persistent! (often requires mounting google drive directories)
Unconfigurable and randomly set GPU variants: T4, P4, V100, K80.

Google AI Platform Notebooks: https://cloud.google.com/ai-platform-notebooks

Integrates with GCloud ecosystem (VM instance can be managed within a GCE project).
Instances are fully configurable (instance type, images, sizes, GPUs, etc.).
'classical' Jupyter lab interface.
Persistent storage!
No instant access, requires connections to CGE console and waiting up to 2 minutes for the instance to start.
Notebooks and Jupyter instance can not be easily shared (though it may be possible ).
Billed by seconds, required manual shutdown. Do not forget to delete/stop the instance when you are finished!

Pricing

Billing by seconds, following the GCE compute, bandwidth and storage pricing (no extra cost).

2020-03-25 : Google does not bill compute (VM instance) when the notebook is stopped (the VM is terminated). This makes the AI notebook very interesting for occasional experiment requiring GPUs. However permanent disk storage is always billed.

2020-04-2020 : After a few weeks of usage, I was billed less than 1$/h bills for a medium VM (n1-standard-4, 15GB ram, 4 vCPU) with Tesla T4 GPU (including storage and bandwidth).

2021-02-2020 : The service now requires 2 disks (minimum 100GB each) for boot and data partition. This increase the upkeep cost to about ~8$/month (even if the VM is terminated).

SSH access

Instance ssh access seems to be enabled by default (no fixed public IP though).

Using the gloud client is straightforward:

gcloud compute ssh --project [project] --zone [zone] [instance name]

VS Code Remote development.

First auto-configure the .ssh/config:

gcloud compute config-ssh

You should be then able to connect in VS code using the SSH remote extension.

Troubleshooting:

You may first have to connect to the instance once with the gcloud ssh ... command, then retry gcloud compute config-ssh.
Try gcloud compute instances add-metadata [instance name] --metadata enable-guest-attributes=TRUE first, then connect to the VM with gcloud ssh ..., finally retry gcloud compute config-ssh.

Share conda installation between the main ssh (vscode) user and 'jupyter' user

Jupyter lab run under the jupyter user wheras ssh sessions/vscode will run under the account owner user name.

Follow instructions from the annaconda docs to share your installation between multiple users:

CONDA_INSTALLATION_DIR=/opt/conda  # set this to your conda installation path
SSH_USERNAME=$(whoami) # Set this to the ssh 

sudo groupadd conda-users
sudo adduser jupyter conda-users
sudo adduser $SSH_USERNAME conda-users

sudo chgrp -R conda-users $CONDA_INSTALLATION_DIR
sudo chmod 770 -R $CONDA_INSTALLATION_DIR

First time init for ssh user:

CONDA_INSTALLATION_DIR=/opt/conda
source $CONDA_INSTALLATION_DIR/bin/activate
conda init

important: In order to refresh the group owneship in vscode, kill the remote vscode server with the command: Ctrl+P > Remote-SSH: Kill VS Code Server on Host.

Root access

Depending on the image used sudo/root access may not be available from the jupyter lab terminal.

The base 'Deep learning VM' Image should allow sudo from jupyter lab (annoucement from google).

It is always possible to get sudo privilege using ssh (also works from vscode): gcloud compute ssh --zone [zone] [instance name]

AI notebooks command line

Enable API:

gcloud services enable notebooks.googleapis.com

List instances:

gcloud beta notebooks instances list --location [zone]

Get jupyter lab url:

gcloud beta notebooks instances describe --location [zone] --format="value(proxyUri)" [instance name]

Stop instance (untested):

gcloud beta notebooks instances stop --location [zone] [instance name]

Other useful commands: stop, reset, start, delete, create

Things to test

Auto-stop script (stop VM when idle for a long time)
Use a AI notebook instance within Google Colab ('to get the best of both world')

martinsotir/google_ai_notebooks_tips.md