Skip to content

Instantly share code, notes, and snippets.

@giswqs

giswqs/Working_with_MosaicJSON2.ipynb

Created December 8, 2020 18:38
Show Gist options
  • Save giswqs/e36b96f410858c38cdd2ad06b7b434f3 to your computer and use it in GitHub Desktop.
Save giswqs/e36b96f410858c38cdd2ad06b7b434f3 to your computer and use it in GitHub Desktop.
Download ZIP
Working_with_MosaicJSON2.ipynb
Display the source blob
Display the rendered blob
Raw
Working_with_MosaicJSON2.ipynb
{
"cells": [
{
"metadata": {},
"cell_type": "markdown",
"source": "# Working With MosaicJSON\n\n[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/developmentseed/titiler/master?filepath=docs%2Fexamples%2FWorking_with_MosaicJSON.ipynb)\n\n### MosaicJSON\n\nMosaicJSON is a specification created by DevelopmentSeed which aims to be an open standard for representing metadata about a mosaic of Cloud-Optimized GeoTIFF (COG) files.\n\n\n> MosaicJSON can be seen as a Virtual raster (see GDAL's VRT) enabling spatial and temporal processing for a list of Cloud-Optimized GeoTIFF.\n\nRef:https://github.com/developmentseed/mosaicjson-spec\n\n\n### Data\n\nFor this demo, we are going to use CloudOptimized GeoTIFF from Digitalglobe opendata: https://www.digitalglobe.com/ecosystem/open-data\n\n\n### Endpoint\n\nBy default, TiTiler has a complete `mosaicjson` endpoint. For this demo we are going to use a slightly modifed version hosted by developmentseed at `https://api.cogeo.xyz`\n\nDocs: https://api.cogeo.xyz/docs#/MosaicJSON"
},
{
"metadata": {},
"cell_type": "markdown",
"source": "\n## Requirements\n\nTo be able to run this notebook you'll need the following requirements:\n- rasterio\n- ipyleaflet\n- requests\n- tqdm\n- BeautifullSoup (webpage parsing) \n- rio-tiler (2.0b8) (Optional)\n- cogeo-mosaic (Optional)\n\n`!pip install rasterio ipyleaflet requests tqdm bs4 requests`\n\n`!pip install rio-tiler cogeo-mosaic --pre`"
},
{
"metadata": {},
"cell_type": "markdown",
"source": "## Get the Data"
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "import os\nimport json\nimport rasterio\nimport requests\nimport bs4 as BeautifulSoup\n\nfrom concurrent import futures\nfrom rio_tiler.io import COGReader\nfrom rasterio.features import bounds as featureBounds\n\nimport bs4 as BeautifulSoup\n\nfrom ipyleaflet import Map, basemaps, TileLayer, basemap_to_tiles, GeoJSON",
"execution_count": 1,
"outputs": []
},
{
"metadata": {},
"cell_type": "markdown",
"source": "### 1. Fetch page and find links"
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "url = \"https://www.digitalglobe.com/ecosystem/open-data/california-colorado-fires\"\n\n# Read Page\nr = requests.get(url)\nsoup = BeautifulSoup.BeautifulSoup(r.text)\ns = soup.findAll('textarea')[0]",
"execution_count": 2,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "list_file = [line.lstrip() for line in s.contents[0].splitlines() if line.endswith(\".tif\")]",
"execution_count": 3,
"outputs": []
},
{
"metadata": {
"scrolled": false,
"trusted": true
},
"cell_type": "code",
"source": "list_file",
"execution_count": null,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "",
"execution_count": null,
"outputs": []
},
{
"metadata": {},
"cell_type": "markdown",
"source": "### 2. Find GeoTIFF Urls"
},
{
"metadata": {
"tags": [],
"trusted": true
},
"cell_type": "code",
"source": "files = [\n dict(\n date=l.split(\"/\")[6],\n tags=[l.split(\"/\")[5]],\n path=l,\n sceneid=l.split(\"/\")[7],\n preview=f\"https://api.discover.digitalglobe.com/show?id={l.split('/')[7]}&f=jpeg\",\n event=l.split(\"/\")[4],\n )\n for l in list_file\n]\n\nfiles = sorted(files, key=lambda x:x[\"date\"])\n\nprint(f\"Number of GeoTIFF: {len(list_file)}\")",
"execution_count": 4,
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": "Number of GeoTIFF: 76\n"
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "### 3. Pre/Post event"
},
{
"metadata": {
"tags": [],
"trusted": true
},
"cell_type": "code",
"source": "pre_event = list(filter(lambda x: x[\"tags\"] == [\"pre-event\"], files))\npost_event = list(filter(lambda x: x[\"tags\"] == [\"post-event\"], files))\n\nprint(f\"Number of Pre Event COG: {len(pre_event)}\")\nprint(f\"Number of Post Event COG: {len(post_event)}\")",
"execution_count": 5,
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": "Number of Pre Event COG: 38\nNumber of Post Event COG: 38\n"
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "### 4. Create Features and Viz(Optional)"
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "def worker(meta):\n try:\n with COGReader(meta[\"path\"]) as cog:\n wgs_bounds = cog.bounds\n except:\n return {}\n\n return {\n \"geometry\": {\n \"type\": \"Polygon\",\n \"coordinates\": [\n [\n [wgs_bounds[0], wgs_bounds[3]],\n [wgs_bounds[0], wgs_bounds[1]],\n [wgs_bounds[2], wgs_bounds[1]],\n [wgs_bounds[2], wgs_bounds[3]],\n [wgs_bounds[0], wgs_bounds[3]]\n ]\n ]\n },\n \"properties\": meta,\n \"type\": \"Feature\"\n }\n\n\n \nwith futures.ThreadPoolExecutor(max_workers=5) as executor:\n responses = [r for r in executor.map(worker, post_event) if r]\nprint(responses)",
"execution_count": 15,
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": "[]\n"
}
]
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "geojson = {'type': 'FeatureCollection', 'features': responses}\n\nbounds = featureBounds(geojson)\n\nm = Map(\n basemap=basemaps.OpenStreetMap.Mapnik,\n center=((bounds[1] + bounds[3]) / 2,(bounds[0] + bounds[2]) / 2),\n zoom=6\n)\n\ngeo_json = GeoJSON(\n data=geojson,\n style={\n 'opacity': 1, 'dashArray': '1', 'fillOpacity': 0, 'weight': 1\n },\n)\nm.add_layer(geo_json)\nm",
"execution_count": 7,
"outputs": [
{
"ename": "ValueError",
"evalue": "min() arg is an empty sequence",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mValueError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-7-9cf8c94f695b>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0mgeojson\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m'type'\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'FeatureCollection'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'features'\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mresponses\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mbounds\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfeatureBounds\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgeojson\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 4\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5\u001b[0m m = Map(\n",
"\u001b[0;32m~/.conda/envs/tiler/lib/python3.8/site-packages/rasterio/features.py\u001b[0m in \u001b[0;36mbounds\u001b[0;34m(geometry, north_up, transform)\u001b[0m\n\u001b[1;32m 389\u001b[0m )\n\u001b[1;32m 390\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 391\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0m_bounds\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgeom\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnorth_up\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnorth_up\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtransform\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtransform\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 392\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 393\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32mrasterio/_features.pyx\u001b[0m in \u001b[0;36mrasterio._features._bounds\u001b[0;34m()\u001b[0m\n",
"\u001b[0;31mValueError\u001b[0m: min() arg is an empty sequence"
]
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "### 5. Create Mosaic"
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "titiler_endpoint = \"https://api.cogeo.xyz/\" # Devseed temporary endpoint\nusername = \"anonymous\" # Update this\nlayername = \"dgopendata_CAfire_2020_post3\" # WARNING, you can overwrite Mosaics",
"execution_count": 12,
"outputs": []
},
{
"metadata": {},
"cell_type": "markdown",
"source": "###### 5.1. Create Token\n\nNote: Right now everyone can create a token to upload or create a mosaic in DevSeed infrastructure\n\nDocs: https://api.cogeo.xyz/docs#/Token/create_token_tokens_create_post"
},
{
"metadata": {
"tags": [],
"trusted": true
},
"cell_type": "code",
"source": "r = requests.post(\n f\"{titiler_endpoint}/tokens/create\",\n json={\n \"username\": username,\n \"scope\": [\"mosaic:read\", \"mosaic:create\"]\n }\n).json()\ntoken = r[\"token\"]\nprint(token)",
"execution_count": 13,
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJhbm9ueW1vdXMiLCJzY29wZSI6WyJtb3NhaWM6cmVhZCIsIm1vc2FpYzpjcmVhdGUiXSwiaWF0IjoxNjA3NDUyNjMwLCJleHAiOjE2MDc0NTYyMzB9.PMiGYAK8SFGvmgKIBFWlqpft_c-O-vsAlqhXl-w6fAg\n"
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "###### 5.2. Create Mosaic\n\nDocs: https://api.cogeo.xyz/docs#/MosaicJSON/create_mosaic_mosaicjson_create_post"
},
{
"metadata": {
"tags": [],
"trusted": true
},
"cell_type": "code",
"source": "r = requests.post(\n f\"{titiler_endpoint}/mosaicjson/create\",\n json={\n \"username\": username,\n \"layername\": layername,\n \"files\": [f[\"path\"] for f in post_event]\n },\n params={\n \"access_token\": r[\"token\"]\n }\n).json()\nprint(r)",
"execution_count": 14,
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": "{'username': 'anonymous', 'layername': 'dgopendata_CAfire_2020_post3', 'mosaic': 'anonymous.dgopendata_CAfire_2020_post3'}\n"
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "###### You can also `upload` a mosaic"
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "# from cogeo_mosaic.mosaic import MosaicJSON\n\n# mosaicdata = MosaicJSON.from_urls([f[\"path\"] for f in post_event])\n\n# print(mosaicdata)\n\n# r = requests.post(\n# f\"{titiler_endpoint}/mosaicjson/upload\",\n# json={\n# \"username\": username,\n# \"layername\": layername,\n# \"mosaic\": mosaicdata.dict(exclude_none=True)\n# },\n# params={\n# \"access_token\": token\n# }\n# ).json()",
"execution_count": null,
"outputs": []
},
{
"metadata": {},
"cell_type": "markdown",
"source": "###### 5.3. Display Tiles\n\nDocs: https://api.cogeo.xyz/docs#/MosaicJSON/tilejson_mosaicjson__layer__tilejson_json_get"
},
{
"metadata": {
"tags": [],
"trusted": true
},
"cell_type": "code",
"source": "r = requests.get(\n f\"{titiler_endpoint}/mosaicjson/{username}.{layername}/tilejson.json\",\n).json()\nprint(r)\n\n# m = Map(\n# center=((bounds[1] + bounds[3]) / 2,(bounds[0] + bounds[2]) / 2),\n# zoom=10\n# )\n\nm = Map()\n\ntiles = TileLayer(\n url=r[\"tiles\"][0],\n min_zoom=r[\"minzoom\"],\n max_zoom=r[\"maxzoom\"],\n opacity=1\n)\n\ngeo_json = GeoJSON(\n data=geojson,\n style={\n 'opacity': 1, 'dashArray': '1', 'fillOpacity': 0, 'weight': 1\n },\n)\nm.add_layer(geo_json)\nm.add_layer(tiles)\nm",
"execution_count": null,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "",
"execution_count": null,
"outputs": []
}
],
"metadata": {
"hide_input": false,
"kernelspec": {
"name": "python3",
"display_name": "Python 3",
"language": "python"
},
"language_info": {
"name": "python",
"version": "3.8.5",
"mimetype": "text/x-python",
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"pygments_lexer": "ipython3",
"nbconvert_exporter": "python",
"file_extension": ".py"
},
"toc": {
"nav_menu": {},
"number_sections": true,
"sideBar": true,
"skip_h1_title": false,
"base_numbering": 1,
"title_cell": "Table of Contents",
"title_sidebar": "Contents",
"toc_cell": false,
"toc_position": {},
"toc_section_display": true,
"toc_window_display": false
},
"gist": {
"id": "",
"data": {
"description": "Working_with_MosaicJSON2.ipynb",
"public": true
}
}
},
"nbformat": 4,
"nbformat_minor": 4
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment