Thomas Mukei

Downloading Shared Files on Google Drive Using Curl

When the shared files on Google Drive is downloaded, it is necessary to change the download method by the file size. The boundary of file size when the method is changed is about 40MB.

File size < 40MB

CURL

filename="### filename ###"
fileid="### file ID ###"
curl -L -o ${filename} "https://drive.google.com/uc?export=download&amp;id=${fileid}"

filter result based on images names containing "base" in their names

index = [index for index, im_similar in enumerate(im_similar) if "base" in im_similar]

im_similar = [im_similar[i] for i in index]
distances = [distances[0][i] for i in index]

	#source https://www.kaggle.com/general/64962

	!pip install kaggle

	#Authenticating with Kaggle API
	from googleapiclient.discovery import build
	import io, os
	from googleapiclient.http import MediaIoBaseDownload
	from google.colab import auth
	auth.authenticateuser() driveservice = build('drive', 'v3')

	# To access kaggle datasets
	!pip install kaggle

	# Import data from Kaggle
	# Colab's file access feature
	from google.colab import files

	# Retrieve uploaded file
	uploaded = files.upload()

	#One workaround is to create clone environment, and then remove original one:
	#(remember about deactivating current environment with deactivate on Windows and source deactivate on macOS/Linux)

	conda create --name new_name --clone old_name --offline #use --offline flag to disable the redownload of all your packages
	conda remove --name old_name --all # or its alias: `conda env remove --name old_name`

	#There are several drawbacks of this method:
	# time consumed on copying environment's files,
	# temporary double disk usage.

	#
	# Original solution via StackOverflow:
	# http://stackoverflow.com/questions/35802939/install-only-available-packages-using-conda-install-yes-file-requirements-t
	#

	#
	# Install via `conda` directly.
	# This will fail to install all
	# dependencies. If one fails,
	# all dependencies will fail to install.

	# List unique values in a DataFrame column
	# h/t @makmanalp for the updated syntax!
	df['Column Name'].unique()

	# Convert Series datatype to numeric (will error if column has non-numeric values)
	# h/t @makmanalp
	pd.to_numeric(df['Column Name'])

	# Convert Series datatype to numeric, changing non-numeric values to NaN
	# h/t @makmanalp for the updated syntax!

Thomas Mukei

Downloading Shared Files on Google Drive Using Curl

File size < 40MB

CURL

filter result based on images names containing "base" in their names

Table of Contents

Installation

Linux

	#!
	# Using aws s3 cp will require the --recursive parameter to copy multiple files.
	aws s3 cp s3://myBucket/dir localdir --recursive

	#The aws s3 sync command will, by default, copy a whole directory. It will only copy new/modified files.
	aws s3 sync s3://mybucket/dir localdir
	#Just experiment to get the result you want.

	#You can combine --exclude and --include options to copy only objects that match a pattern, excluding all others.
	aws s3 cp s3://mybucket/logs/ s3://mybucket2/logs/ --recursive --exclude "" --include ".log"