Aaron Lidman aaronlidman
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import fiona | |
| schema = {'geometry': 'Polygon', | |
| 'properties': {'province': 'str', | |
| 'city': 'str', | |
| 'name': 'str', | |
| 'country': 'str', | |
| 'POP_CNTRY': 'float'}} | |
| with fiona.open( |
aaronlidman
/ gist:5370295
Created
April 12, 2013 07:49
python set vs frozenset, parsing a los-angeles.osm.pbf extract, testing the main node lookup (~3 million items)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| time | |
| memory (mb) | |
| set | |
| 0:00:41.377259 | |
| 736.04 | |
| frozenset | |
| 0:00:37.533754 | |
| 735.94 |
aaronlidman
/ join.txt
Created
April 13, 2013 18:20
Figuring out why my parser is so slow.
3.3M items
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Time, RAM (mb) | |
| Count only | |
| 00:22.57 | |
| 365.82 | |
| '|'.join(map(str, (lon, lat))) | |
| 00:27.99 | |
| 365.78 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Time, Memory | |
| quick.py | |
| CPython: | |
| 00:28.36 | |
| 697.21 | |
| Pypy: | |
| 01:09.85 | |
| 645.42 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 1049930250 function calls (1049916672 primitive calls) in 3737.590 seconds | |
| Ordered by: internal time | |
| List reduced from 304 to 30 due to restriction <30> | |
| ncalls tottime percall cumtime percall filename:lineno(function) | |
| 14829194 1436.229 0.000 1557.391 0.000 /Users/aaron/centerline/ENV/lib/python2.7/site-packages/shapely/predicates.py:8(__call__) | |
| 41129739 200.483 0.000 218.108 0.000 {map} | |
| 55989715 158.995 0.000 171.340 0.000 /Users/aaron/centerline/ENV/lib/python2.7/site-packages/shapely/coords.py:56(_update) | |
| 4847561 134.502 0.000 649.796 0.000 /Users/aaron/centerline/ENV/lib/python2.7/site-packages/shapely/geometry/linestring.py:154(geos_linestring_from_py) |
aaronlidman
/ simplegeo
Created
June 6, 2013 21:14
Simplegeo data dump - I keep not being able to find the link, here it is.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| http://s3.amazonaws.com/simplegeo-public/places_dump_20110628.zip |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| polygon | |
| 1 | |
| -7.711977E+01 3.893428E+01 | |
| -7.711900E+01 3.893381E+01 | |
| -7.711876E+01 3.893367E+01 | |
| -7.711859E+01 3.893349E+01 | |
| -7.711847E+01 3.893328E+01 | |
| -7.711823E+01 3.893298E+01 | |
| -7.711783E+01 3.893251E+01 | |
| -7.711747E+01 3.893181E+01 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| {'profile': True, 'group_size': 20, 'source': 'us-west-latest.osm.pbf', 'require_name': False, 'overwrite': True, 'out': 'output.geojson'} | |
| parsing ways and passing through nodes | |
| parsing coordinates | |
| processing... | |
| 858745 | |
| saved as: output.geojson | |
| 1.54 | |
| 183217409 function calls in 996.461 seconds | |
| Ordered by: internal time |
aaronlidman
/ TIGER-parks
Created
July 16, 2013 21:26
Get all the parks, from TIGER 2012, convert them to a single sqlite.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/bin/bash | |
| # downloading, list created via https://gist.github.com/3745763 | |
| wget -i rawr.out -q | |
| # unzipping | |
| for county in *arealm.zip | |
| do | |
| unzip -q $county | |
| rm -rf $county |
aaronlidman
/ gist:6084694
Created
July 25, 2013 23:19
Quick python profile, keep having to look this up
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| python -m cProfile -s tottime (then your regular python script) |