gabefair / atwarrior.md

Created April 27, 2023 21:03 — forked from catleeball/atwarrior.md

Tutorial on running an Archive Team Warrior container with Docker Compose

You can run an Archive Team container on your computer! It'll help crawl the web and putting stuff into archive.org, based on the current projects.

gabefair / git.migrate

Created May 2, 2020 00:49 — forked from niksumeiko/git.migrate

Moving git repository and all its branches, tags to a new remote repository keeping commits history

	#!/bin/bash
	# Sometimes you need to move your existing git repository
	# to a new remote repository (/new remote origin).
	# Here are a simple and quick steps that does exactly this.
	#
	# Let's assume we call "old repo" the repository you wish
	# to move, and "new repo" the one you wish to move to.
	#
	### Step 1. Make sure you have a local copy of all "old repo"
	### branches and tags.

gabefair / progress.py

Created September 2, 2018 16:19

How I monitor progress in python

	import sys, os, datetime
	from time import time

	def update_progress(total_imported, skipped_files, current_location):
	text = str(total_imported) + "/" + str(total_imported+skipped_files) + " imported. Looking in folder: " + current_location
	sys.stdout.write('\r' + text)
	sys.stdout.flush()

	def main():
	start_time = datetime.datetime.fromtimestamp(time())

gabefair / split_json-lines_block.py

Created August 30, 2018 22:00

Splits a giant jsonL file into multipble jsonL files.

	# Gabriel Fair
	# Please suggest any improvements: http://keybase.io/gabefair
	import mmap
	import re
	import argparse
	import sys
	import progressbar
	from time import sleep
	import os
	from time import clock

gabefair / to_dataframe.py

Last active November 2, 2020 02:49

Mongodb collection to pandas dataframe

	import pandas as pd
	from pymongo import MongoClient

	client = MongoClient("mongodb://localhost:27017")
	db = client.database_Name

	collection_conn = db['collection_name']
	collection_cursor = collection_conn.find()
	collection_pandas_df = pd.DataFrame(list(collection_cursor))

gabefair / cool_name.sh

Created August 18, 2018 17:30

Cool terminal text

gabefair / setup.py

Created August 9, 2018 19:54

merge rows of two csv files based on a column-key and place one of them ahead infront of the other file's rows after the merge

	import sys, os

	try:
	import pandas as pd
	except ImportError:
	print("Please run `pip install pandas` from a command line")
	exit()


	def get_file_head(file_name):

gabefair / restore_packages.R

Created August 5, 2018 02:31 — forked from arne-cl/restore_packages.R

save/load/install a list of your currently installed R packages

	# restore_packages.R
	#
	# installs each package from the stored list of packages
	# source: http://hlplab.wordpress.com/2012/06/01/transferring-installed-packages-between-different-installations-of-r/

	load("~/installed_packages.rda")

	for (count in 1:length(installedpackages)) {
	install.packages(installedpackages[count])
	}

gabefair / v2_import.py

Last active September 25, 2018 05:34

Json to Mongo file importer

	# pylint: disable=unsubscriptable-object
	import sys, os, json, datetime, platform, argparse, traceback, mechanize
	from time import time

	try:
	import mechanize
	except ImportError:
	print("Please run `pip install mechanize` from a command line")
	exit()

gabefair / Tilestache-guide.md

Created December 15, 2017 14:48

Installing Tilestache to serve custom ArcGIS layers using Postgres 10 on Ubuntu 16 (Xenial Xerus)

Here I share my work notes to get custom layers that were once in ArcGIS to serve on a Ubuntu web host.

Gabriel Fair gabefair

Installing Tilestache to serve custom ArcGIS layers using Postgres 10 on Ubuntu 16 (Xenial Xerus)