changjonathanc’s gists

changjonathanc / mc4-timestamp-analysis.ipynb

Created September 17, 2021 15:22

mc4-timestamp-analysis.ipynb

Sorry, something went wrong. Reload?

Sorry, we cannot display this file.

Sorry, this file is invalid so it cannot be displayed.

changjonathanc / LengthedGeneratorWrapper.py

Created July 10, 2021 13:33

	from tqdm.auto import tqdm
	import time

	class LengthedGeneratorWrapper:
	"""wraps an infinite generator with length for tqdm"""
	def __init__(self, infinite_generator, len):
	self.generator = infinite_generator
	self.len = len
	def __len__(self):
	return self.len

changjonathanc / train_time.py

Created July 9, 2021 09:01

	import sys

	# imports utils and imports
	from src import *

	from dataclasses import dataclass
	from typing import Optional
	from omegaconf import OmegaConf

changjonathanc / csv.py

Created February 8, 2021 11:58

python mmap to concatenate csv files

	❯ rm out.csv
	❯ cat 1.py
	from glob import glob
	import mmap

	files = glob("data/*")

	files.sort(key=lambda x: int(x.split("/")[-1].split(".")[0]))
	write_f = open("out.csv", "w+b")

changjonathanc / shelve-test.py

Last active January 27, 2021 06:46

simple benchmark/test for python shelve

	import shelve
	import time
	# initialize
	with shelve.open('shelvedb') as db:
	for i in range(10000):
	db[f'{i}'] = '{i*2}'

	# no caching
	with shelve.open('shelvedb') as db:
	tic = time.time()

changjonathanc / bug_report_model.py

Last active December 7, 2020 02:32

pytorch-lightning ddp BatchEncoding

	# Copyright The PyTorch Lightning team.
	#
	# Licensed under the Apache License, Version 2.0 (the "License");
	# you may not use this file except in compliance with the License.
	# You may obtain a copy of the License at
	#
	# http://www.apache.org/licenses/LICENSE-2.0
	#
	# Unless required by applicable law or agreed to in writing, software
	# distributed under the License is distributed on an "AS IS" BASIS,

changjonathanc / install julia.md

Last active November 9, 2020 15:00

Install Julia and use it in jupyter notebook

conda install -c conda-forge julia
conda activate julia
conda install notebook
julia # use julia REPL to install IJulia (necessary)

> using Pkg
> Pkg.add("IJulia")

changjonathanc / utils.py

Created July 22, 2020 11:44

python utils

	import json
	def read_json(path, encoding = 'utf-8'):
	with open(path, encoding = encoding) as f:
	return json.load(f)
	def write_json(data, path, encoding = 'utf-8'):
	with open(path, 'w', encoding = encoding) as f:
	return json.dump(data, f)

changjonathanc / gist:e217dfa9a39ff837b948b03bc3afc1da

Created May 30, 2020 12:27

access images on server over ssh, without downloading, using one-line python http server

	local_account@local_hostname$ ssh -L 12345:0.0.0.0:12345 remote_account@remote_hostname
	remote_account@remote_hostname$ python -m http.server 12345
	# open http://localhost:12345/ on local computer browser

changjonathanc / gist:b113a2a562c15e6487c3daf898f5193a

Last active May 30, 2020 03:15

Crack password from shadow file: Install john the ripper on ubuntu with gpu support

	Usage (Official Repo):
	https://github.com/magnumripper/JohnTheRipper

	Ubuntu Install Guide:
	https://github.com/magnumripper/JohnTheRipper/blob/bleeding-jumbo/doc/INSTALL-UBUNTU

	Other install guides can be found in:
	https://github.com/magnumripper/JohnTheRipper/blob/bleeding-jumbo/doc/
	https://github.com/magnumripper/JohnTheRipper/blob/bleeding-jumbo/doc/INSTALL
	https://github.com/magnumripper/JohnTheRipper/blob/bleeding-jumbo/doc/INSTALL-FEDORA

Jonathan Chang changjonathanc