How to prevent a remote Jupyter kernel from being interrupted when connection from a local machine to it is closed?

how_to_run_jupyter_kernels_without_connection_to_them.md

The Problem

Suppose that Jupyter notebooks are hosted at a remote server and you access them from your local machine via web interface. By default, if connection is closed (e.g., you close a browser tab with remotely hosted notebook), corresponding to this connection running kernels die. Hence, time-consuming processes can not be finished without maintaining their connections open. Sometimes it is unfeasible — for instance, assume that you must shut down your local machine in the evening and you can turn it on only in the next morning.

The Solution

Initial Once-Only Preparations

Open local .bashrc file in edit mode:

nano .bashrc

Append this to the end of the .bashrc file:

alias jupyter_ssh="ssh -L 8889:localhost:8889 your_user@hostname_of_your_server"

The quoted command forwards port 8889 from your local host to port 8889 of your remote server.

Then connect to your remote server and create a tmux session there (alternatively, screen can be used instead of tmux):

ssh your_user@hostname_of_your_server
tmux new -s jupyter_server

Start Jupyter server from tmux window:

jupyter notebook --port=8889 --no-browser

There will be a link with authentication token, copy it and save it in a place where you can find it.

Connecting

Every time you want to connect to remote Jupyter server from scratch, i.e. without previous authentications in the current browser session, do the following.

Create SSH connection and forward port 8889 to that of your remote server:

jupyter_ssh

Then copy the link with authentication token. If you forget where you saved the link, you can find it manually. Run this from a terminal of the remote server:

tmux attach -t jupyter_server
Ctrl+b
[
# Now you can scroll window; find the link in logs and copy it with Ctrl+Shift+C. 

Paste the link with authentication token in address bar of a local browser and press Enter. That is it, you can edit and run notebooks like you usually do it.

Limitations

If you close connection while below cell is running, new files still will be created:

from time import sleep

n_tasks = 1000000
sleep_time = 5
for i in range(n_tasks):
    with open('tmp/tmp_{}'.format(i), 'w') as out_file:
        out_file.write(str(i))
    sleep(sleep_time)

However, if you close connection while below cell is running, you will see nothing new when you connect to your notebook again.

from time import sleep

n_tasks = 1000000
sleep_time = 5
for i in range(n_tasks):
    print(i)
    sleep(sleep_time)

To sum it up, processes can run offline, but notebooks can not be updated without connection, so you need to store all necessary results in external files.