Verdi March verdimrc

NOTE: # means comment. The rest are a mixture of languages: Python, line magic, cell magic, etc.

Mixture

# Set environment variable that're valid throughout a notebook.
# Use /home/ec2-user instead of ~, otherwise even !ls -al $VAR_NAME fails.
var_name = 1234
%env VAR_NAME=$var_name
%set_env VAR_NAME=$var_name

How to setup Python runtime that's compatible with AWS Lambda

Will use python-3.8 as the example throughout this document. In a nutshell:

on EC2 or container, pull the runtime binary from https://lambci.s3.amazonaws.com/fs/python3.8.tgz or aws cp s3://lambci/fs/python3.8.tgz <DST>.
Extract the files on /, which will put a few binaries on /var/lang/bin.
[Optional] Install additional packages using /var/lang/bin/pip ... and zip these as a layer.

And that's all it takes!

General

This is a diary on setting up a workable Python environment on a Windows machine with internet going through a proxy server.

General tips:

Use “classical” cmd, and the conda commands should just work.
- Powershell needs proper setup of permissions; unfortunately I’m completely not familiar with PS. YMMV.
  - One symptom of improper configuration of PS: conda install ... throws SSL error.
Tips to identify proxy servers

	# Authenticate to ECR
	dkrlogin=$(aws ecr get-login --no-include-email)
	eval $dkrlogin

	# Pull image
	account=123456789012
	image=image-name
	docker pull ${account}.dkr.ecr.us-east-2.amazonaws.com/${image}:latest

	# Tag/register as local image

	# Get video info
	ffmpeg -i video.avi

	# Convert video to images spaced at 1-hour interval (i.e., 1 frame every 3,600 seconds)
	# -vsync 0, otherwise for some media format, gonna get excess frames.
	ffmpeg -i video.mkv -vf fps=1/3600 -vsync 0 images-ffmpeg/image-%04d.jpg

	# Read from s3
	aws s3 cp s3://bucket/dir/video.mkv - \| ffmpeg -i - -vf fps=1/3600 -vsync 0 images-ffmpeg/image-%04d.jpg

	# Run tests
	pytest -v --tb=short --s3path s3://vm-hello-world/s3fs-compat test/s3fscompat

	# Alternatively, run tests with increased verbosity with -rA
	pytest -v -rA --tb=short --s3path s3://vm-hello-world/s3fs-compat test/s3fscompat

	"""Debugging tips, esp. under vscode, but might also applicable to pdb/ipdb."""

	# try block to prevent isort shifting the import statements.
	# See also: https://github.com/timothycrosley/isort/issues/295#issuecomment-570898035
	try:
	import sys

	if sys.gettrace() or (set(sys.modules) & {"pdb", "ipdb"}): # Am I running under a debugger?
	import multiprocessing

	from typing import Generic, TypeVar, get_type_hints

	T = TypeVar("T")


	class MyObject(Generic[T]):
	def __getitem__(self, k: T):
	type_hints = get_type_hints(self.__getitem__)
	print(self.__getitem__.__annotations__)
	print(type_hints)

	# Install 7zip which at times is more robust in handling 'corrupted' zip files.
	sudo yum-config-manager --enable epel
	7za --help

	# Need `pyenv activate my_env` to set the VIRTUAL_ENV environment variable.

	# Remove old config which may come from different virtual envs.
	rm ~/.pipupgrade/{config.ini,db.db}

	# Dry-run
	/usr/local/bin/pipupgrade --pip-path $VIRTUAL_ENV/bin/pip3 --check

	# Update (NOTE: still have chance to enter choice to cont or cancel).
	/usr/local/bin/pipupgrade --pip-path $VIRTUAL_ENV/bin/pip3