Aaron Steele eightysteele
eightysteele
/ gist:1144211
Created
August 13, 2011 20:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| cells = dict((k, simplejson.dumps(v)) for k,v in cells.iteritems()) |
eightysteele
/ gist:1144340
Created
August 13, 2011 22:55
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [dict(cellkey=k, doc=simplejson.dumps(cells[k])) for k in cells.keys()] |
eightysteele
/ gist:1144364
Created
August 13, 2011 23:25
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| simplejson.dumps(dict(a=1,b=2)) | |
| '{"a": 1, "b": 2}' |
eightysteele
/ bulkloader_error.txt
Created
August 14, 2011 23:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [ERROR ] [WorkerThread-0] WorkerThread: | |
| Traceback (most recent call last): | |
| File "/home/eighty/Dropbox/projects/lib/python/google_appengine/google/appengine/tools/adaptive_thread_pool.py", line 176, in WorkOnItems | |
| status, instruction = item.PerformWork(self.__thread_pool) | |
| File "/home/eighty/Dropbox/projects/lib/python/google_appengine/google/appengine/tools/bulkloader.py", line 764, in PerformWork | |
| transfer_time = self._TransferItem(thread_pool) | |
| File "/home/eighty/Dropbox/projects/lib/python/google_appengine/google/appengine/tools/bulkloader.py", line 933, in _TransferItem | |
| self.content = self.request_manager.EncodeContent(self.rows) | |
| File "/home/eighty/Dropbox/projects/lib/python/google_appengine/google/appengine/tools/bulkloader.py", line 1392, in EncodeContent | |
| entity = loader.create_entity(values, key_name=key, parent=parent) |
eightysteele
/ gist:1145479
Created
August 15, 2011 00:01
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| from ndb import model | |
| class LayerPolygon(model.Model): | |
| polygonid = model.StringProperty('p', required=True) | |
| areaid = model.StringProperty('a', required=True) | |
| specieslatin = model.StringProperty('s', required=True) | |
| source = model.StringProperty('r', required=True) | |
| class LayerIndex(model.Expando): | |
| polygons = model.StructuredProperty(LayerPolygon, repeated=True) |
eightysteele
/ gist:1145809
Created
August 15, 2011 06:40
Bulkload post_import_function that shims in a repeated StructuredProperty.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| def add_polygons(input_dict, instance, bulkload_state_copy): | |
| """Bulkload post_import_function that shims in a repeated StructuredProperty. | |
| This function allows us to bulkload a StructuredProperty model using the App | |
| Engine bulkloader by adding the following lines to the bulkload.yaml | |
| configuration file: | |
| - kind: LayerIndex | |
| connector: csv | |
| connector_options: |
eightysteele
/ gist:1148636
Created
August 16, 2011 08:00
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| python_preamble: | |
| - import: base64 | |
| - import: re | |
| - import: google.appengine.ext.bulkload.transform | |
| - import: google.appengine.ext.bulkload.bulkloader_wizard | |
| - import: google.appengine.ext.db | |
| - import: google.appengine.api.datastore | |
| - import: bulkload_helper | |
| transformers: |
eightysteele
/ gist:1154976
Created
August 18, 2011 19:43
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| json = simplejson.dumps(dict((k, v) for k,v in simplejson.loads(recjson) if k is not None)) |
eightysteele
/ gist:1172685
Created
August 26, 2011 04:16
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| INFO:root:Performing a dry run... | |
| INFO:root:Processing source directories: ['iucn', 'jetz'] | |
| INFO:root:Collections in iucn: ['mammals', 'amphibians'] | |
| INFO:root:Processing 1 layers in the mammals collection | |
| INFO:root:Extracting DBF fields from abditomys_latidens.shp | |
| INFO:root:All collection metadata saved to collection.csv.txt | |
| INFO:root:Dry run complete! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| id | name | age | |
|---|---|---|---|
| 0 | aaron | 34 | |
| 1 | tina | 36 | |
| 2 | noah | © |