Skip to content

Instantly share code, notes, and snippets.

@dogrdon
dogrdon / convert_date_to_epoch.py
Created April 30, 2018 03:55
Convert a date time string with a specified format and timezone to Unix Epoch time stamp
# timezone solution from: https://stackoverflow.com/questions/12165691/python-datetime-with-timezone-to-epoch/17257177#17257177
'''
Take a timezone in the form of `MM/DD/YYYY HH:MM:SS {AM/PM}`
and convert to unix epoch in python3
'''
from datetime import datetime
import pytz
@dogrdon
dogrdon / scrambled_csv.py
Created March 7, 2017 17:45
python script to scramble some csvs
'''SOMETIMES YOU WANT SOME SCRAMBLED CSVs: python scrambled_csv.py <path_to_csv_file>'''
import csv, random
import sys
def scrambler(head, cols):
scrambled = []
@dogrdon
dogrdon / get_official_socrata_data.py
Last active November 7, 2016 16:21
Getting only the official datasets in a socrata portal (or anything that follows the data.json standard - https://project-open-data.cio.gov/v1.1/api/)
import csv, requests
with open('./canonical_datasets_nycopen_07NOV2016.csv', 'w') as w:
writer = csv.writer(w)
writer.writerow(['access', 'issued', 'modified', 'publisher', 'title', 'location', 'identifier', 'id', 'theme', 'description'])
data_location = 'https://nycopendata.socrata.com/data.json'
res = requests.get(data_location)
data = res.json()
datasets = data['dataset']
for d in datasets:
@dogrdon
dogrdon / collapse_by_column.py
Created October 10, 2016 20:59
collapse csv rows by a column value
'''you have a csv file with something like:
col1, col2
x, 1
x, 2
y, 1
z, 4
z, 8
and you want to output something like:
col1, col2
x, 1;2
def leftJoinCSV(file1, file2, *args):
'''Really should only be used to left join two csv files. Left file is first arg, right file is second arg.
Column(s) title (string in the head) are the third and fourth args'''
if len(args) > 1:
'''can either enter one column for both or each one '''
column1 = args[0]
column2 = args[1]
elif len(args) == 1:
column1 = args[0]
column2 = args[0]
@dogrdon
dogrdon / thumbnailgen.py
Last active April 3, 2016 14:11
Scrappy mockup of a script to generate a thumbnail preview for a video found online...no need to download. Run me like: `python thumbnailgen.py -i http://example.com/video.f4v`
#!/usr/bin/env python
from subprocess import PIPE, Popen
import argparse
import os
'''
Give me a url to a video file and I will return a thumbnail preview
Run me like: `python thumbnailgen.py -i https://catalog.archives.gov/OpaAPI/media/13729/content/arcmedia/mopix/107/107-1130.wmv?download=true`
@dogrdon
dogrdon / thumbnail.sh
Created March 9, 2016 16:59
simple command for generating thumbnail preview from videos with a url
ffmpeg -i "put_url_here" -vf fps=1/15 img_%03d.jpg && montage -geometry +4+4 img_* output.png
@dogrdon
dogrdon / addregex.py
Last active February 25, 2016 16:46
Handy commandline tool for providing the essential business analytics necessary to determine the outcome of using regular expressions as a potential solution.
import sys
import time
if len(sys.argv) < 2:
print("You need to tell me how many problems you have for which you are trying to apply regex as a solution")
print("run script as `python addregex.py <no. of problems>`")
sys.exit()
problems = int(sys.argv[1])
@dogrdon
dogrdon / map.js
Created October 22, 2012 02:05
map.js file (leaflet js logic for web map).
var map;
var pointsLayer;
$(document).ready(function () {
map = new L.Map('mapContainer');
var url = 'http://{s}.tiles.mapbox.com/v3/mapbox.mapbox-streets/{z}/{x}/{y}.png';
var copyright = 'Map data &copy; 2011 OpenStreetMap contributors, Imagery &copy; 2011 CloudMade';
var tileLayer = new L.TileLayer(url, {
attribution: copyright
});
var startPosition = new L.LatLng(41.883333, - 87.633333);
@dogrdon
dogrdon / irc2.js
Created May 22, 2012 16:41
irc.js file 2 (forked from wikistream, working to modify).
//imports
var fs = require('fs'),
path = require('path'),
irc = require('irc-js'),
nodeio = require('node.io'),
//redis = require('redis').createClient();
mongo = require('mongodb'),
db = new mongo.Db('wikis', new mongo.Server('localhost', 27017, {}), {});