Last active
October 16, 2024 03:23
-
-
Save bikcrum/f9d4cfb0c40dff23329b89cd664d4964 to your computer and use it in GitHub Desktop.
This is the way how can you connect google colab as well as google drive (mounted) using SSH. This is useful when you want to download data directly to your google drive specially for machine learning purpose. It can be easy to mount google drive and use files into for your code.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
{ | |
"nbformat": 4, | |
"nbformat_minor": 0, | |
"metadata": { | |
"colab": { | |
"name": "Connect Google Colab+Drive with SSH.ipynb", | |
"provenance": [], | |
"collapsed_sections": [ | |
"DDsNUSe-lWMk", | |
"-framHcYlZZl", | |
"hnP5d6SBmPOi", | |
"azVypUY2nQiT", | |
"woSIgj7TnImq", | |
"syBlFACknx-D", | |
"WV2KqCkQoVWn" | |
] | |
}, | |
"kernelspec": { | |
"name": "python3", | |
"display_name": "Python 3" | |
} | |
}, | |
"cells": [ | |
{ | |
"cell_type": "markdown", | |
"metadata": { | |
"id": "WCXdafPUkW62", | |
"colab_type": "text" | |
}, | |
"source": [ | |
"### Configure NGROK" | |
] | |
}, | |
{ | |
"cell_type": "markdown", | |
"metadata": { | |
"id": "DDsNUSe-lWMk", | |
"colab_type": "text" | |
}, | |
"source": [ | |
"#### Installation" | |
] | |
}, | |
{ | |
"cell_type": "code", | |
"metadata": { | |
"id": "z5At1w0bk1O5", | |
"colab_type": "code", | |
"colab": {} | |
}, | |
"source": [ | |
"!wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip -O ngrok-stable-linux-amd64.zip\n", | |
"!unzip -u ngrok-stable-linux-amd64.zip" | |
], | |
"execution_count": 0, | |
"outputs": [] | |
}, | |
{ | |
"cell_type": "markdown", | |
"metadata": { | |
"id": "-framHcYlZZl", | |
"colab_type": "text" | |
}, | |
"source": [ | |
"#### Copy Ngrok to root and change permission" | |
] | |
}, | |
{ | |
"cell_type": "code", | |
"metadata": { | |
"id": "fTOJQv7mlqDY", | |
"colab_type": "code", | |
"colab": {} | |
}, | |
"source": [ | |
"!cp /content/ngrok /ngrok\n", | |
"!chmod +x /ngrok" | |
], | |
"execution_count": 0, | |
"outputs": [] | |
}, | |
{ | |
"cell_type": "markdown", | |
"metadata": { | |
"id": "zx8FRSbglwhm", | |
"colab_type": "text" | |
}, | |
"source": [ | |
"#### Authorize ngrok\n", | |
"\n", | |
"- Sign up for a free ngrok account: https://ngrok.com\n", | |
"- Copy your authtoken from https://dashboard.ngrok.com/auth and paste it into the following cell\n", | |
"- Run the cell" | |
] | |
}, | |
{ | |
"cell_type": "code", | |
"metadata": { | |
"id": "sv8y5ofImKwX", | |
"colab_type": "code", | |
"colab": { | |
"base_uri": "https://localhost:8080/", | |
"height": 34 | |
}, | |
"outputId": "8961de9c-d2ff-4e20-821c-1ed5690f9991" | |
}, | |
"source": [ | |
"!/ngrok authtoken YOUR_AUTH_TOKEN_HERE" | |
], | |
"execution_count": 9, | |
"outputs": [ | |
{ | |
"output_type": "stream", | |
"text": [ | |
"Authtoken saved to configuration file: /root/.ngrok2/ngrok.yml\n" | |
], | |
"name": "stdout" | |
} | |
] | |
}, | |
{ | |
"cell_type": "markdown", | |
"metadata": { | |
"id": "hnP5d6SBmPOi", | |
"colab_type": "text" | |
}, | |
"source": [ | |
"#### Make config file for ngrok" | |
] | |
}, | |
{ | |
"cell_type": "code", | |
"metadata": { | |
"id": "C2s9ifkWmt4X", | |
"colab_type": "code", | |
"colab": { | |
"base_uri": "https://localhost:8080/", | |
"height": 34 | |
}, | |
"outputId": "dbfbca0a-13f5-4dd5-eefa-83b130ed9b4d" | |
}, | |
"source": [ | |
"%%writefile /content/ssh.yml\n", | |
"tunnels:\n", | |
" ssh:\n", | |
" proto: tcp\n", | |
" addr: 22" | |
], | |
"execution_count": 10, | |
"outputs": [ | |
{ | |
"output_type": "stream", | |
"text": [ | |
"Writing /content/ssh.yml\n" | |
], | |
"name": "stdout" | |
} | |
] | |
}, | |
{ | |
"cell_type": "code", | |
"metadata": { | |
"id": "MXJ5FhF3m3A7", | |
"colab_type": "code", | |
"colab": { | |
"base_uri": "https://localhost:8080/", | |
"height": 34 | |
}, | |
"outputId": "f116118a-e0ac-4292-9d45-a2851cd37082" | |
}, | |
"source": [ | |
"%%writefile /content/run_ngrok.sh\n", | |
"#!/bin/sh\n", | |
"set -x\n", | |
"/ngrok start --config ~/.ngrok2/ngrok.yml --config /content/ssh.yml ssh" | |
], | |
"execution_count": 11, | |
"outputs": [ | |
{ | |
"output_type": "stream", | |
"text": [ | |
"Writing /content/run_ngrok.sh\n" | |
], | |
"name": "stdout" | |
} | |
] | |
}, | |
{ | |
"cell_type": "markdown", | |
"metadata": { | |
"id": "T3VC0sfCm8nR", | |
"colab_type": "text" | |
}, | |
"source": [ | |
"### Configura SSH" | |
] | |
}, | |
{ | |
"cell_type": "markdown", | |
"metadata": { | |
"id": "azVypUY2nQiT", | |
"colab_type": "text" | |
}, | |
"source": [ | |
"#### Install ssh" | |
] | |
}, | |
{ | |
"cell_type": "code", | |
"metadata": { | |
"id": "uq2RkeOwnCa4", | |
"colab_type": "code", | |
"colab": {} | |
}, | |
"source": [ | |
"!apt-get install ssh" | |
], | |
"execution_count": 0, | |
"outputs": [] | |
}, | |
{ | |
"cell_type": "markdown", | |
"metadata": { | |
"id": "woSIgj7TnImq", | |
"colab_type": "text" | |
}, | |
"source": [ | |
"#### Overwrite sshd_config file " | |
] | |
}, | |
{ | |
"cell_type": "code", | |
"metadata": { | |
"id": "D-DMY1iEnXie", | |
"colab_type": "code", | |
"colab": { | |
"base_uri": "https://localhost:8080/", | |
"height": 34 | |
}, | |
"outputId": "7dc6c2b4-9d49-4966-efbf-ebb3b066e21a" | |
}, | |
"source": [ | |
"%%writefile /etc/ssh/sshd_config\n", | |
"#\t$OpenBSD: sshd_config,v 1.101 2017/03/14 07:19:07 djm Exp $\n", | |
"\n", | |
"# This is the sshd server system-wide configuration file. See\n", | |
"# sshd_config(5) for more information.\n", | |
"\n", | |
"# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin\n", | |
"\n", | |
"# The strategy used for options in the default sshd_config shipped with\n", | |
"# OpenSSH is to specify options with their default value where\n", | |
"# possible, but leave them commented. Uncommented options override the\n", | |
"# default value.\n", | |
"\n", | |
"#Port 22\n", | |
"#AddressFamily any\n", | |
"#ListenAddress 0.0.0.0\n", | |
"#ListenAddress ::\n", | |
"\n", | |
"#HostKey /etc/ssh/ssh_host_rsa_key\n", | |
"#HostKey /etc/ssh/ssh_host_ecdsa_key\n", | |
"#HostKey /etc/ssh/ssh_host_ed25519_key\n", | |
"\n", | |
"# Ciphers and keying\n", | |
"#RekeyLimit default none\n", | |
"\n", | |
"# Logging\n", | |
"#SyslogFacility AUTH\n", | |
"#LogLevel INFO\n", | |
"\n", | |
"# Authentication:\n", | |
"\n", | |
"#LoginGraceTime 2m\n", | |
"#PermitRootLogin prohibit-password\n", | |
"#StrictModes yes\n", | |
"#MaxAuthTries 6\n", | |
"#MaxSessions 10\n", | |
"\n", | |
"#PubkeyAuthentication yes\n", | |
"\n", | |
"# Expect .ssh/authorized_keys2 to be disregarded by default in future.\n", | |
"#AuthorizedKeysFile\t.ssh/authorized_keys .ssh/authorized_keys2\n", | |
"\n", | |
"#AuthorizedPrincipalsFile none\n", | |
"\n", | |
"#AuthorizedKeysCommand none\n", | |
"#AuthorizedKeysCommandUser nobody\n", | |
"\n", | |
"# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts\n", | |
"#HostbasedAuthentication no\n", | |
"# Change to yes if you don't trust ~/.ssh/known_hosts for\n", | |
"# HostbasedAuthentication\n", | |
"#IgnoreUserKnownHosts no\n", | |
"# Don't read the user's ~/.rhosts and ~/.shosts files\n", | |
"#IgnoreRhosts yes\n", | |
"\n", | |
"# To disable tunneled clear text passwords, change to no here!\n", | |
"#PasswordAuthentication yes\n", | |
"#PermitEmptyPasswords no\n", | |
"\n", | |
"# Change to yes to enable challenge-response passwords (beware issues with\n", | |
"# some PAM modules and threads)\n", | |
"ChallengeResponseAuthentication no\n", | |
"\n", | |
"# Kerberos options\n", | |
"#KerberosAuthentication no\n", | |
"#KerberosOrLocalPasswd yes\n", | |
"#KerberosTicketCleanup yes\n", | |
"#KerberosGetAFSToken no\n", | |
"\n", | |
"# GSSAPI options\n", | |
"#GSSAPIAuthentication no\n", | |
"#GSSAPICleanupCredentials yes\n", | |
"#GSSAPIStrictAcceptorCheck yes\n", | |
"#GSSAPIKeyExchange no\n", | |
"\n", | |
"# Set this to 'yes' to enable PAM authentication, account processing,\n", | |
"# and session processing. If this is enabled, PAM authentication will\n", | |
"# be allowed through the ChallengeResponseAuthentication and\n", | |
"# PasswordAuthentication. Depending on your PAM configuration,\n", | |
"# PAM authentication via ChallengeResponseAuthentication may bypass\n", | |
"# the setting of \"PermitRootLogin without-password\".\n", | |
"# If you just want the PAM account and session checks to run without\n", | |
"# PAM authentication, then enable this but set PasswordAuthentication\n", | |
"# and ChallengeResponseAuthentication to 'no'.\n", | |
"UsePAM yes\n", | |
"\n", | |
"#AllowAgentForwarding yes\n", | |
"AllowTcpForwarding yes\n", | |
"#GatewayPorts no\n", | |
"X11Forwarding yes\n", | |
"#X11DisplayOffset 10\n", | |
"#X11UseLocalhost yes\n", | |
"#PermitTTY yes\n", | |
"PrintMotd no\n", | |
"#PrintLastLog yes\n", | |
"#TCPKeepAlive yes\n", | |
"#UseLogin no\n", | |
"#PermitUserEnvironment no\n", | |
"#Compression delayed\n", | |
"#ClientAliveInterval 0\n", | |
"#ClientAliveCountMax 3\n", | |
"#UseDNS no\n", | |
"#PidFile /var/run/sshd.pid\n", | |
"#MaxStartups 10:30:100\n", | |
"#PermitTunnel no\n", | |
"#ChrootDirectory none\n", | |
"#VersionAddendum none\n", | |
"\n", | |
"# no default banner path\n", | |
"#Banner none\n", | |
"\n", | |
"# Allow client to pass locale environment variables\n", | |
"AcceptEnv LANG LC_*\n", | |
"\n", | |
"# override default of no subsystems\n", | |
"Subsystem\tsftp\t/usr/lib/openssh/sftp-server\n", | |
"\n", | |
"# Example of overriding settings on a per-user basis\n", | |
"#Match User anoncvs\n", | |
"#\tX11Forwarding no\n", | |
"#\tAllowTcpForwarding no\n", | |
"#\tPermitTTY no\n", | |
"#\tForceCommand cvs server" | |
], | |
"execution_count": 13, | |
"outputs": [ | |
{ | |
"output_type": "stream", | |
"text": [ | |
"Overwriting /etc/ssh/sshd_config\n" | |
], | |
"name": "stdout" | |
} | |
] | |
}, | |
{ | |
"cell_type": "markdown", | |
"metadata": { | |
"id": "syBlFACknx-D", | |
"colab_type": "text" | |
}, | |
"source": [ | |
"#### Make .ssh directory in root" | |
] | |
}, | |
{ | |
"cell_type": "code", | |
"metadata": { | |
"id": "WQaNRQ_wn41F", | |
"colab_type": "code", | |
"colab": {} | |
}, | |
"source": [ | |
"!mkdir -p ~/.ssh" | |
], | |
"execution_count": 0, | |
"outputs": [] | |
}, | |
{ | |
"cell_type": "markdown", | |
"metadata": { | |
"id": "zbQdx_Pgn52L", | |
"colab_type": "text" | |
}, | |
"source": [ | |
"#### Copy content of id_rsa.pub from your system and paste below" | |
] | |
}, | |
{ | |
"cell_type": "markdown", | |
"metadata": { | |
"id": "mi9NAas-y7eq", | |
"colab_type": "text" | |
}, | |
"source": [ | |
"- Goto `~/.ssh` on your system\n", | |
"- Find id_rsa.pub then open it\n", | |
"- In case if you don't have it create one using `ssh-keygen -t rsa`. Enter nothing in file name and password.\n", | |
"- Copy content from id_rsa.pub and paste in the following cell" | |
] | |
}, | |
{ | |
"cell_type": "code", | |
"metadata": { | |
"id": "GNJw9O1yoJ57", | |
"colab_type": "code", | |
"colab": { | |
"base_uri": "https://localhost:8080/", | |
"height": 34 | |
}, | |
"outputId": "6a11f781-e063-40b5-8127-843a11ad04ac" | |
}, | |
"source": [ | |
"%%writefile ~/.ssh/authorized_keys\n", | |
"CONTENT_OF_ID_RSA_PUB_FILE" | |
], | |
"execution_count": 15, | |
"outputs": [ | |
{ | |
"output_type": "stream", | |
"text": [ | |
"Writing /root/.ssh/authorized_keys\n" | |
], | |
"name": "stdout" | |
} | |
] | |
}, | |
{ | |
"cell_type": "markdown", | |
"metadata": { | |
"id": "WV2KqCkQoVWn", | |
"colab_type": "text" | |
}, | |
"source": [ | |
"#### Restart ssh service" | |
] | |
}, | |
{ | |
"cell_type": "code", | |
"metadata": { | |
"id": "XidUAxZ6ocp9", | |
"colab_type": "code", | |
"colab": { | |
"base_uri": "https://localhost:8080/", | |
"height": 51 | |
}, | |
"outputId": "35aacbde-b16e-49e6-cd7e-68ad755609f5" | |
}, | |
"source": [ | |
"!service ssh restart" | |
], | |
"execution_count": 16, | |
"outputs": [ | |
{ | |
"output_type": "stream", | |
"text": [ | |
" * Restarting OpenBSD Secure Shell server sshd\n", | |
" ...done.\n" | |
], | |
"name": "stdout" | |
} | |
] | |
}, | |
{ | |
"cell_type": "markdown", | |
"metadata": { | |
"id": "Bqpvztxnodj9", | |
"colab_type": "text" | |
}, | |
"source": [ | |
"\n", | |
"### START SSH TUNNELING" | |
] | |
}, | |
{ | |
"cell_type": "markdown", | |
"metadata": { | |
"id": "jA6uBL8lrmOy", | |
"colab_type": "text" | |
}, | |
"source": [ | |
"#### Run .sh file created above" | |
] | |
}, | |
{ | |
"cell_type": "code", | |
"metadata": { | |
"id": "LeQKIcQlonGJ", | |
"colab_type": "code", | |
"colab": {} | |
}, | |
"source": [ | |
"get_ipython().system_raw('bash /content/run_ngrok.sh ssh &')" | |
], | |
"execution_count": 0, | |
"outputs": [] | |
}, | |
{ | |
"cell_type": "markdown", | |
"metadata": { | |
"id": "o2jHPggnrvEb", | |
"colab_type": "text" | |
}, | |
"source": [ | |
"#### Show ssh connect command" | |
] | |
}, | |
{ | |
"cell_type": "code", | |
"metadata": { | |
"id": "MIfMw7xHr0qU", | |
"colab_type": "code", | |
"colab": {} | |
}, | |
"source": [ | |
"import requests\n", | |
"import urllib.parse\n", | |
"\n", | |
"def get_ngrok_info():\n", | |
" return requests.get('http://localhost:4040/api/tunnels').json()\n", | |
"\n", | |
"def get_ngrok_tunnels():\n", | |
" for tunnel in get_ngrok_info()['tunnels']:\n", | |
" name = tunnel['name']\n", | |
" yield name, tunnel\n", | |
"\n", | |
"def get_ngrok_tunnel(name):\n", | |
" for name1, tunnel in get_ngrok_tunnels():\n", | |
" if name == name1:\n", | |
" return tunnel\n", | |
"\n", | |
"def get_ngrok_url(name, local=False):\n", | |
" if local:\n", | |
" return get_ngrok_tunnel(name)['config']['addr']\n", | |
" else:\n", | |
" return get_ngrok_tunnel(name)['public_url']\n", | |
"\n", | |
"\n", | |
"public_url, public_port = urllib.parse.urlparse(get_ngrok_url('ssh')).netloc.split(':')\n", | |
"print('To SSH into this colab instance, run the following command on your local machine:')\n", | |
"print('ssh root@{} -p {}'.format(public_url, public_port))" | |
], | |
"execution_count": 0, | |
"outputs": [] | |
}, | |
{ | |
"cell_type": "markdown", | |
"metadata": { | |
"id": "tN9F_W45spkF", | |
"colab_type": "text" | |
}, | |
"source": [ | |
"#### Mount Google Drive to access it's file using SSH" | |
] | |
}, | |
{ | |
"cell_type": "code", | |
"metadata": { | |
"id": "ZqAf7BrBswXz", | |
"colab_type": "code", | |
"colab": {} | |
}, | |
"source": [ | |
"from google.colab import drive\n", | |
"drive.mount('/content/drive', force_remount=True)\n", | |
"\n", | |
"# To unmount using following\n", | |
"# drive.flush_and_unmount()" | |
], | |
"execution_count": 0, | |
"outputs": [] | |
} | |
] | |
} |
Can someone guide me what will be the password for ssh?
@Swastik2000 you shouldn't be prompted for a password if you put a correct rsa pub key in:
%%writefile ~/.ssh/authorized_keys
CONTENT_OF_ID_RSA_PUB_FILE
@bikcrum Thank you so much for this helpful post! It really saves me!
http://localhost:4040/api/tunnels
is not available. Hence, giving a
ConnectionError: HTTPConnectionPool(host='localhost', port=4040): Max retries exceeded with url: /api/tunnels (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7c1f67e215a0>: Failed to establish a new connection: [Errno 111] Connection refused'))
I'd likely at least amend this to include a direct reference to NGrok, if that's what you want to use for tunneling. Blindly downloading binaries from other sites generally isn't recommended.
I'd also recommend not completely overwriting the sshd_config
file, but using a local config for the user, instead (read: there's likely a better way to do this tunnel).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
There has got to be an easier way to do this...