Improve GeoStore.rasterize performance

geostore-rasterize-performance.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Improve GeoStore.rasterize performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example #1 (the same CRSs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Input\n",
    "\n",
    "<div class=\"alert alert-danger\">\n",
    "<ul>\n",
    "    <li><strong>Time: </strong>5.47 s</li>\n",
    "    <li><strong>Data CRS: </strong>EPSG:4326</li>\n",
    "    <li><strong>Target CRS: </strong>EPSG:4326</li>\n",
    "    <li><strong>Command: </strong><code>db.rasterize('building_polygon', pixel_resolution=0.00001, roi=roi, crs=constants.WGS84_CRS)</code></li>\n",
    "</ul>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bottlenecks "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Checking for non-emptiness (`GeoVector.is_empty` → `GeoVector.get_shape` → checking for CRS equality)\n",
    "\n",
    "* **Before: **`shapes = [feature.geometry.get_shape(crs) for feature in polygonized if not feature.is_empty]`\n",
    "* **After: **`shapes = [feature.geometry.get_shape(crs) for feature in self]`\n",
    "* **Where: ** [telluric/collections.py#L282](https://github.com/satellogic/telluric/blob/9cabe2a974637297134b711cda8af61628026ec1/telluric/collections.py#L282)\n",
    "* **Overall speedup: **17%\n",
    "* **Overall time: **4.51 s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Checking for CRS equality for each geometry (within `GeoVector.get_shape` method)\n",
    "\n",
    "* **Before: **`shapes = [feature.geometry.get_shape(crs) for feature in polygonized if not feature.is_empty]`\n",
    "* **After: **`shapes = [feature.geometry._shape for feature in self]`\n",
    "* **Where: ** [telluric/collections.py#L282](https://github.com/satellogic/telluric/blob/9cabe2a974637297134b711cda8af61628026ec1/telluric/collections.py#L282)\n",
    "* **Overall speedup: **33%\n",
    "* **Overall time: **3.65 s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Checking for CRS equality for each geometry (within `GeoVector.reproject` method)\n",
    "\n",
    "* **Before: **`result = dataset.reproject(crs).rasterize()`\n",
    "* **After: **`result = dataset.rasterize()`\n",
    "* **Where: ** [geostore/geostore.py#L747](https://publicgitlab.satellogic.com/telluric/geostore/blob/master/geostore/geostore.py#L747)\n",
    "* **Overall speedup: **48%\n",
    "* **Overall time: **2.82 s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. Checking for CRS equality for each geometry (within `GeoVector.reproject` method)\n",
    "\n",
    "* **Question: ** Catalog CRS and data CRS?\n",
    "* **Before: **`features_list = [feat.reproject(source_crs) for feat in self._generate_features_list(results, geo_column, crs)]`\n",
    "* **After: **`features_list = [feat for feat in self._generate_features_list(results, geo_column, crs)]`\n",
    "* **Where: ** [geostore/geostore.py#L256](https://publicgitlab.satellogic.com/telluric/geostore/blob/master/geostore/geostore.py#L256)\n",
    "* **Overall speedup: **67%\n",
    "* **Overall time: **1.82 s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5. Checking for CRS validity for each geometry (within `GeoVector.__init__` method)\n",
    "\n",
    "* **Before: **`assert self._crs.is_valid`\n",
    "* **After: **`#assert self._crs.is_valid`\n",
    "* **Where: ** [telluric/vectors.py#L281](https://github.com/satellogic/telluric/blob/9cabe2a974637297134b711cda8af61628026ec1/telluric/vectors.py#L281)\n",
    "* **Overall speedup: **75%\n",
    "* **Overall time: **1.34 s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results\n",
    "\n",
    "<div class=\"alert alert-success\">\n",
    "<strong>Reference time: </strong>5.47 s<br>\n",
    "<strong>Overall time: </strong>1.34 s<br>\n",
    "<strong>Overall speedup: </strong>75%\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example #2 (different CRSs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Input\n",
    "\n",
    "<div class=\"alert alert-danger\">\n",
    "<ul>\n",
    "    <li><strong>Time: </strong>11.4 s</li>\n",
    "    <li><strong>Data CRS: </strong>EPSG:4326</li>\n",
    "    <li><strong>Target CRS: </strong>EPSG:3857</li>\n",
    "    <li><strong>Command: </strong><code>db.rasterize('building_polygon', pixel_resolution=1, roi=roi)</code></li>\n",
    "</ul>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bottlenecks "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Checking for non-emptiness (`GeoVector.is_empty` → `GeoVector.get_shape` → checking for CRS equality)\n",
    "\n",
    "* **Before: **`shapes = [feature.geometry.get_shape(crs) for feature in polygonized if not feature.is_empty]`\n",
    "* **After: **`shapes = [feature.geometry.get_shape(crs) for feature in self]`\n",
    "* **Where: ** [telluric/collections.py#L282](https://github.com/satellogic/telluric/blob/9cabe2a974637297134b711cda8af61628026ec1/telluric/collections.py#L282)\n",
    "* **Overall speedup: **13%\n",
    "* **Overall time: **9.89 s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Checking for CRS equality and reprojecting for each geometry (within `GeoVector.get_shape` method)\n",
    "\n",
    "* **Before: **`shapes = [feature.geometry.get_shape(crs) for feature in polygonized if not feature.is_empty]`\n",
    "* **After: **`shapes = [feature.geometry._shape for feature in self]`\n",
    "* **Where: ** [telluric/collections.py#L282](https://github.com/satellogic/telluric/blob/9cabe2a974637297134b711cda8af61628026ec1/telluric/collections.py#L282)\n",
    "* **Overall speedup: **31%\n",
    "* **Overall time: **7.91 s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Checking for CRS equality for each geometry (within `GeoVector.reproject` method)\n",
    "\n",
    "* **Before: **`result = dataset.reproject(crs).rasterize()`\n",
    "* **After: **`result = dataset.reproject(crs, check_crs=False).rasterize()`\n",
    "* **Note: **add new `check_crs` parameter to `BaseCollection.reproject`\n",
    "* **Where: ** [geostore/geostore.py#L747](https://publicgitlab.satellogic.com/telluric/geostore/blob/master/geostore/geostore.py#L747)\n",
    "* **Overall speedup: **41%\n",
    "* **Overall time: **6.67 s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. Checking for CRS equality for each geometry (within `GeoVector.reproject` method)\n",
    "\n",
    "* **Question: ** Catalog CRS and data CRS?\n",
    "* **Before: **`features_list = [feat.reproject(source_crs) for feat in self._generate_features_list(results, geo_column, crs)]`\n",
    "* **After: **`features_list = [feat for feat in self._generate_features_list(results, geo_column, crs)]`\n",
    "* **Where: ** [geostore/geostore.py#L256](https://publicgitlab.satellogic.com/telluric/geostore/blob/master/geostore/geostore.py#L256)\n",
    "* **Overall speedup: **50%\n",
    "* **Overall time: **5.72 s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5. Checking for CRS validity for each geometry (within `GeoVector.__init__` method)\n",
    "\n",
    "* **Before: **`assert self._crs.is_valid`\n",
    "* **After: **`#assert self._crs.is_valid`\n",
    "* **Where: ** [telluric/vectors.py#L281](https://github.com/satellogic/telluric/blob/9cabe2a974637297134b711cda8af61628026ec1/telluric/vectors.py#L281)\n",
    "* **Overall speedup: **71%\n",
    "* **Overall time: **3.36 s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results\n",
    "\n",
    "<div class=\"alert alert-success\">\n",
    "<strong>Reference time: </strong>11.4 s<br>\n",
    "<strong>Overall time: </strong>3.36 s<br>\n",
    "<strong>Overall speedup: </strong>71%\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}