Skip to content

Instantly share code, notes, and snippets.

@eightysteele
Created August 15, 2011 06:40
Show Gist options
  • Select an option

  • Save eightysteele/1145809 to your computer and use it in GitHub Desktop.

Select an option

Save eightysteele/1145809 to your computer and use it in GitHub Desktop.
Bulkload post_import_function that shims in a repeated StructuredProperty.
def add_polygons(input_dict, instance, bulkload_state_copy):
"""Bulkload post_import_function that shims in a repeated StructuredProperty.
This function allows us to bulkload a StructuredProperty model using the App
Engine bulkloader by adding the following lines to the bulkload.yaml
configuration file:
- kind: LayerIndex
connector: csv
connector_options:
encoding: utf_8
property_map:
- property: __key__
external_name: id
post_import_function: our_bulkloader_transforms.add_polygons
Then, given our LayerPolygon model:
class LayerPolygon(model.Model):
polygonid = model.StringProperty(required=True)
areaid = model.StringProperty(required=True)
specieslatin = model.StringProperty(required=True)
source = model.StringProperty(required=True)
We create a StructuredProperty in our LayerIndex model:
class LayerIndex(model.Expando):
polygons = model.StructuredProperty(LayerPolygon, repeated=True)
After the bulkload we can filter for the StructuredProperty field values as
described in the Datastore Plus Tutorial, Part One (http://goo.gl/ky5sa). For
example:
qry = LayerIndex.query(
LayerIndex.polygons == LayerPolygon(
specieslatin='puma concolor', polygonid=2))
"""
# Model the LayerPolygon properties as parallel arrays
polygonid = []
areaid = []
specieslatin = []
source = []
# Populate the parallel arrays from values in the input_dict (a CSV row)
polygons = simplejson.loads(input_dict['polygons'])
for p in polygons:
polygonid.append(p['polygonid'])
areaid.append(p['areaid'])
specieslatin.append(p['specieslatin'])
source.append(p['source'])
# Update entity instance with new property names that match the StructuredProperty
instance['polygons.polygonid'] = polygonid
instance['polygons.areaid'] = areaid
instance['polygons.specieslatin'] = specieslatin
instance['polygons.source'] = source
return instance
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment