Skip to content

Instantly share code, notes, and snippets.

View larsyencken's full-sized avatar

Lars Yencken larsyencken

View GitHub Profile
@larsyencken
larsyencken / gist:3375334
Created August 17, 2012 02:11
colorific: example in Python
>>> from colorific import extract_colors, rgb_to_hex
>>> palette = extract_colors('some_image.png')
>>> hex_colors = []
>>> for color in palette.colors:
... hex_colors.append(rgb_to_hex(color.value))
>>> hex_colors
['#0047f1', '#e61b31', '#3e7bf4', '#00b018', '#f9a600']
@larsyencken
larsyencken / sqlpy.py
Created August 22, 2012 06:10
SQL via Python
#!/usr/bin/env python
# -*- coding: utf-8 -*-
#
# sqlpy.py
#
"""
Query SQL and dump as CSV.
"""
@larsyencken
larsyencken / stack_trace.log
Created September 5, 2012 02:13
colorific: PIL stack trace
Process: Python [64795]
Path: /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
Identifier: Python
Version: 2.7.3 (2.7.3)
Code Type: X86-64 (Native)
Parent Process: bash [64400]
User ID: 501
Date/Time: 2012-09-05 12:11:53.915 +1000
OS Version: Mac OS X 10.8.1 (12B19)
@larsyencken
larsyencken / resilient.sql
Created September 17, 2012 04:57
Resilient Hive queries
SET mapred.skip.mode.enabled = true;
SET mapred.map.max.attempts = 100;
SET mapred.reduce.max.attempts = 100;
SET mapred.skip.map.max.skip.records = 30000;
SET mapred.skip.attempts.to.start.skipping = 1;
@larsyencken
larsyencken / fetch_and_combine.py
Created November 15, 2012 03:09
Aggregating CloudFront logs
#!/usr/bin/env python
# -*- coding: utf-8 -*-
#
# fetch_and_combine.py
#
"""
Scans CloudFront logs in an S3 bucket for any that are new. Combines log files
into a single local file per hour. If logs for multiple CloudFront
distributions are present, combines them all.
@larsyencken
larsyencken / rel_python.patch
Last active December 11, 2015 00:09
Fix Python relative ldflags issue in graph-tool 2.2.1's configure script
diff --git a/configure b/configure
index 359d74c..3a4f825 100755
--- a/configure
+++ b/configure
@@ -16543,6 +16543,9 @@ $as_echo_n "checking python extra linking flags... " >&6; }
PYTHON_EXTRA_LDFLAGS=`$PYTHON -c "import distutils.sysconfig; \
conf = distutils.sysconfig.get_config_var; \
print conf('LINKFORSHARED')"`
+ # fix a relative import issue
+ framework_dir=$($PYTHON-config --ldflags | sed 's/^-L\([^ ]*Python.framework\).*$/\1/g' | sed 's/\//\\\//g')
@larsyencken
larsyencken / kml_to_csv.py
Last active November 22, 2022 13:01
Convert a Google Latitude KML dump to a CSV of timestamp, latitude and longitude
#!/usr/bin/env python
# -*- coding: utf-8 -*-
#
# kml_to_csv.py
#
"""
Convert a KML file to a CSV set of timestamps and locations.
"""
@larsyencken
larsyencken / workflow.d
Last active December 12, 2015 02:08
drake: minimal example of content-based dependencies
;
; workflow.d
;
; Minimal content-based dependency example
;
;
; recalculate md5 every time, overwite the old one only on change
; XXX we have to use an implicit input name
;
@larsyencken
larsyencken / monitor-jobflows.py
Last active December 14, 2015 11:29
Monitor Elastic MapReduce jobflows for state changes, and announce them using OS X's say command.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
#
# monitor-jobflows.py
#
"""
Monitors for changes in any jobflow's status and announces them.
"""
@larsyencken
larsyencken / README.md
Last active December 15, 2015 09:09
Google Latitude export script

Google latitude export

Export your entire Google Latitude history to a local folder of CSV files, one per day.

1. Get an API key

Head over to Google's API Console and create a new project, with access to Google Latitude. Pick "installed application" as the project type, and once you've created it, download the client secrets file (client_secrets.json) and put it in the current directory.

2. Run the first export