Connect to remote Jupyter Notebook server from VScode Jupyter Notebook IDE

1. Download Anaconda .sh file locally for linux server from https://www.anaconda.com/products/individual and move it to remote server using the following command

ssh -r <path>/Anaconda-latest-Linux-x86_64.sh [email protected]:/root/test

2. Enter to remote server using ssh tunnelling. Make sure that port 8000 is open and not being blocked by copying PID using the following command and subsequently using it to free port. Note: VSCode will not throw any error, not even jupyter notebook server but we won't be able to connect. Very annoying ...

sudo lsof -i :8000

sudo kill -9 14135

3. Connect to the server using ssh tunneling (enter password when asked). here keep all ports as it is as. replace remote username and ip address accordingly. NOTE: DO NOT EXIT/STOP TUNNEL OTHERWISE THE WHOLE CONNECTION WILL BREAK. IT IS NOT LIKE TIMUX. WE NEED TO ALWAYS REMAIN CONNECTED TO THE SERVER VIA TUNNEL.

ssh -L 8000:localhost:8888 [email protected]

4. Now install Anaconda using the following command. Keep hitting enter.

bash /root/test/Anaconda-latest-Linux-x86_64.sh

5. NOTE: This step of making tmux session might not be needed as we need to keep ssh tunneling active all the time. Above command will install Anaconda somewhere. Copy and use that path. I have used mine for now. But before that create a new tmux environment for anaconda environment. 

tmux new-session -s jupyter-server

6. Create new environment for tensorflow work. Below I have used the path where my anaconda is installed. Keep hitting enter.

/root/anaconda3/bin/conda create -n tensorflow-env

Note: In order to create virtual env with 2.7 version of Python, use the following command

/root/anaconda3/bin/conda create -n tensorflow-env python=2.7

or for python 3.6

/root/anaconda3/bin/conda create -n tensorflow-env python=3.6

7. Activate the environment now while being inside tmux. Note: to deactivate just type "conda deactivate" without full conda path. 

source /root/anaconda3/bin/activate /root/anaconda3/envs/tensorflow-env/

8. Now install all required pip3 libraries, inclusing jupyter and tensorflow

pip3 install runipy ipython jupyter tensorflow scikit-image xgboost

apt-get install libgeos-dev

pip3 install https://github.com/matplotlib/basemap/archive/master.zip
OR
/root/anaconda3/bin/conda install basemap

9. Start jupyter notebook server while being inside tmux env and anaconda env. NOTE: make sure that port 8888 is alrady not in use. Otherwise free it up using point 2 commands.

jupyter notebook --allow-root --no-browser

10. The last command above will give a uri like this. Copy it and close the tmux session securely. NOTE: exiting tmux does not mean we can end the tunnel as well. We always need to remain connected to the server via tunnel.

http://localhost:8888/?token=ad64c16832d3c297596e18e2fcce4207b9924d032d6911b3

11. Now, since we have used 8000 port for ssh tunnerling to connect it to port 8888 on remote server. Use the following url to browse remote .ipynb files in your browser. All cells etc will be executed on the server. NOTE: We do not have to adjust any ports now.

http://localhost:8000

12. In order to connect to this remote jupyter notebook via VSCode open vscode and hit command+shift+P or ⌘+⇧+P. Select "Jupyter: Specify local or remote Jupyter server for connections". This will open "Pick how to connect to Jupyter". By default we use "Default". This time select "Existing" and enter the following uri.

http://localhost:8000/?token=ad64c16832d3c297596e18e2fcce4207b9924d032d6911b3

13. Note above that we have used 8000 port to connect to remote jupyter the way we did for browser, but additionally we also need to provide token key which we copied when we started jupyter notebook for the first time in remote server.

14. Enter above uri will ask you to restart VSCode. Once done everything should start working after few seconds of establishing connection (hopefully you don't have to select kernel as there should be only one kernel there in the server. otherwise select the correct one).

15. IMPORTANT: Always make sure that port 8000 is open and clean. If some connection is just not getting established or Port 403 error is being thrown by anaconda env, make sure port 8000 is fine. Also, in order to run big scripts and dataset the best is to transfer .ipynb/.py file to remote server and run it there using anaconda env (more on how to connect and set it there will come later.)

16. IMPORTANT: In case of wrong Username and Passphrase error check for VSCode Maven Kernal thing. Search for "@id:ms-toolsai.jupyter" under extension tab and disable and enable it. You might also need to do this - Try opening Keychain Access.app. Right-click on the login keychain and try locking it and then unlocking it again. In case imshow renderer fails you might need to disable and then enable "Jupyter Notebook Renderers" extension.