Skip to content

Instantly share code, notes, and snippets.

@giswqs

giswqs/Working_with_MosaicJSON.ipynb

Created December 8, 2020 14:07
Show Gist options
  • Save giswqs/1078742fc65ac05e67c4473dbe6be851 to your computer and use it in GitHub Desktop.
Save giswqs/1078742fc65ac05e67c4473dbe6be851 to your computer and use it in GitHub Desktop.
Download ZIP
Working_with_MosaicJSON.ipynb
Display the source blob
Display the rendered blob
Raw
Working_with_MosaicJSON.ipynb
{
"cells": [
{
"metadata": {},
"cell_type": "markdown",
"source": "# Working With MosaicJSON\n\n[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/developmentseed/titiler/master?filepath=docs%2Fexamples%2FWorking_with_MosaicJSON.ipynb)\n\n### MosaicJSON\n\nMosaicJSON is a specification created by DevelopmentSeed which aims to be an open standard for representing metadata about a mosaic of Cloud-Optimized GeoTIFF (COG) files.\n\n\n> MosaicJSON can be seen as a Virtual raster (see GDAL's VRT) enabling spatial and temporal processing for a list of Cloud-Optimized GeoTIFF.\n\nRef:https://github.com/developmentseed/mosaicjson-spec\n\n\n### Data\n\nFor this demo, we are going to use CloudOptimized GeoTIFF from Digitalglobe opendata: https://www.digitalglobe.com/ecosystem/open-data\n\n\n### Endpoint\n\nBy default, TiTiler has a complete `mosaicjson` endpoint. For this demo we are going to use a slightly modifed version hosted by developmentseed at `https://api.cogeo.xyz`\n\nDocs: https://api.cogeo.xyz/docs#/MosaicJSON"
},
{
"metadata": {},
"cell_type": "markdown",
"source": "\n## Requirements\n\nTo be able to run this notebook you'll need the following requirements:\n- rasterio\n- ipyleaflet\n- requests\n- tqdm\n- BeautifullSoup (webpage parsing) \n- rio-tiler (2.0b8) (Optional)\n- cogeo-mosaic (Optional)\n\n`!pip install rasterio ipyleaflet requests tqdm bs4 requests`\n\n`!pip install rio-tiler cogeo-mosaic --pre`"
},
{
"metadata": {},
"cell_type": "markdown",
"source": "## Get the Data"
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "import os\nimport json\nimport rasterio\nimport requests\nimport bs4 as BeautifulSoup\n\nfrom concurrent import futures\n# from rio_tiler.io import COGReader\nfrom rasterio.features import bounds as featureBounds\n\nimport bs4 as BeautifulSoup\n\nfrom ipyleaflet import Map, basemaps, TileLayer, basemap_to_tiles, GeoJSON",
"execution_count": 1,
"outputs": []
},
{
"metadata": {},
"cell_type": "markdown",
"source": "### 1. Fetch page and find links"
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "url = \"https://www.digitalglobe.com/ecosystem/open-data/california-colorado-fires\"\n\n# Read Page\nr = requests.get(url)\nsoup = BeautifulSoup.BeautifulSoup(r.text)\ns = soup.findAll('textarea')",
"execution_count": 2,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "content = s[0].contents[0].strip()\nURLs = [URL.strip() for URL in content.split('\\n')]",
"execution_count": 3,
"outputs": []
},
{
"metadata": {
"scrolled": false,
"trusted": true
},
"cell_type": "code",
"source": "list_file = []\n\nfor URL in URLs:\n if URL.endswith('.tif'):\n list_file.append(URL)",
"execution_count": 4,
"outputs": []
},
{
"metadata": {
"scrolled": false,
"trusted": true
},
"cell_type": "code",
"source": "with open(\"cog.txt\", 'w') as f:\n for URL in list_file:\n f.write(URL + '\\n')",
"execution_count": null,
"outputs": []
},
{
"metadata": {},
"cell_type": "markdown",
"source": "### 2. Find GeoTIFF Urls"
},
{
"metadata": {
"tags": [],
"trusted": true
},
"cell_type": "code",
"source": "# list_file = list(set([l.get('href') for l in s if l.get('href').endswith(\".tif\")]))\n\nfiles = [\n dict(\n date=l.split(\"/\")[6],\n tags=[l.split(\"/\")[5]],\n path=l,\n sceneid=l.split(\"/\")[7],\n preview=f\"https://api.discover.digitalglobe.com/show?id={l.split('/')[7]}&f=jpeg\",\n event=l.split(\"/\")[4],\n )\n for l in list_file\n]\n\nfiles = sorted(files, key=lambda x:x[\"date\"])\n\nprint(f\"Number of GeoTIFF: {len(list_file)}\")",
"execution_count": 5,
"outputs": [
{
"output_type": "stream",
"text": "Number of GeoTIFF: 76\n",
"name": "stdout"
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "### 3. Pre/Post event"
},
{
"metadata": {
"tags": [],
"trusted": true
},
"cell_type": "code",
"source": "pre_event = list(filter(lambda x: x[\"tags\"] == [\"pre-event\"], files))\npost_event = list(filter(lambda x: x[\"tags\"] == [\"post-event\"], files))\n\nprint(f\"Number of Pre Event COG: {len(pre_event)}\")\nprint(f\"Number of Post Event COG: {len(post_event)}\")",
"execution_count": 6,
"outputs": [
{
"output_type": "stream",
"text": "Number of Pre Event COG: 38\nNumber of Post Event COG: 38\n",
"name": "stdout"
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "### 4. Create Features and Viz(Optional)"
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "def worker(meta):\n try:\n with COGReader(meta[\"path\"]) as cog:\n wgs_bounds = cog.bounds\n except:\n return {}\n\n return {\n \"geometry\": {\n \"type\": \"Polygon\",\n \"coordinates\": [\n [\n [wgs_bounds[0], wgs_bounds[3]],\n [wgs_bounds[0], wgs_bounds[1]],\n [wgs_bounds[2], wgs_bounds[1]],\n [wgs_bounds[2], wgs_bounds[3]],\n [wgs_bounds[0], wgs_bounds[3]]\n ]\n ]\n },\n \"properties\": meta,\n \"type\": \"Feature\"\n }\n\n\n \nwith futures.ThreadPoolExecutor(max_workers=5) as executor:\n responses = [r for r in executor.map(worker, post_event) if r]",
"execution_count": null,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "geojson = {'type': 'FeatureCollection', 'features': responses}\n\nbounds = featureBounds(geojson)\n\nm = Map(\n basemap=basemaps.OpenStreetMap.Mapnik,\n center=((bounds[1] + bounds[3]) / 2,(bounds[0] + bounds[2]) / 2),\n zoom=6\n)\n\ngeo_json = GeoJSON(\n data=geojson,\n style={\n 'opacity': 1, 'dashArray': '1', 'fillOpacity': 0, 'weight': 1\n },\n)\nm.add_layer(geo_json)\nm",
"execution_count": null,
"outputs": []
},
{
"metadata": {},
"cell_type": "markdown",
"source": "### 5. Create Mosaic"
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "titiler_endpoint = \"https://api.cogeo.xyz/\" # Devseed temporary endpoint\nusername = \"anonymous\" # Update this\nlayername = \"ca_fire\" # WARNING, you can overwrite Mosaics",
"execution_count": 10,
"outputs": []
},
{
"metadata": {},
"cell_type": "markdown",
"source": "###### 5.1. Create Token\n\nNote: Right now everyone can create a token to upload or create a mosaic in DevSeed infrastructure\n\nDocs: https://api.cogeo.xyz/docs#/Token/create_token_tokens_create_post"
},
{
"metadata": {
"tags": [],
"trusted": true
},
"cell_type": "code",
"source": "r = requests.post(\n f\"{titiler_endpoint}/tokens/create\",\n json={\n \"username\": username,\n \"scope\": [\"mosaic:read\", \"mosaic:create\"]\n }\n).json()\ntoken = r[\"token\"]\nprint(token)",
"execution_count": 11,
"outputs": [
{
"output_type": "stream",
"text": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJhbm9ueW1vdXMiLCJzY29wZSI6WyJtb3NhaWM6cmVhZCIsIm1vc2FpYzpjcmVhdGUiXSwiaWF0IjoxNjA3NDM2MjA1LCJleHAiOjE2MDc0Mzk4MDV9.UMVBgdavzOWcZ7Kc1K642EgTHr6HyOMEZlL2RdDd4Lg\n",
"name": "stdout"
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "###### 5.2. Create Mosaic\n\nDocs: https://api.cogeo.xyz/docs#/MosaicJSON/create_mosaic_mosaicjson_create_post"
},
{
"metadata": {
"tags": [],
"trusted": true
},
"cell_type": "code",
"source": "r = requests.post(\n f\"{titiler_endpoint}/mosaicjson/create\",\n json={\n \"username\": username,\n \"layername\": layername,\n \"files\": [f[\"path\"] for f in post_event]\n },\n params={\n \"access_token\": r[\"token\"]\n }\n).json()\nprint(r)",
"execution_count": 12,
"outputs": [
{
"output_type": "stream",
"text": "{'detail': 'An error occurred (AccessDeniedException) when calling the DescribeTable operation: User: arn:aws:sts::552819999234:assumed-role/titiler-production-titilerproductionlambdaServiceR-AKW530NH9QUW/titiler-production-titilerproductionlambda43A73745-1WHGW5IICYERZ is not authorized to perform: dynamodb:DescribeTable on resource: arn:aws:dynamodb:us-east-1:552819999234:table/ds-mosaics-labs'}\n",
"name": "stdout"
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "###### You can also `upload` a mosaic"
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "# from cogeo_mosaic.mosaic import MosaicJSON\n\n# mosaicdata = MosaicJSON.from_urls([f[\"path\"] for f in post_event])\n\n# print(mosaicdata)\n\n# r = requests.post(\n# f\"{titiler_endpoint}/mosaicjson/upload\",\n# json={\n# \"username\": username,\n# \"layername\": layername,\n# \"mosaic\": mosaicdata.dict(exclude_none=True)\n# },\n# params={\n# \"access_token\": token\n# }\n# ).json()",
"execution_count": null,
"outputs": []
},
{
"metadata": {},
"cell_type": "markdown",
"source": "###### 5.3. Display Tiles\n\nDocs: https://api.cogeo.xyz/docs#/MosaicJSON/tilejson_mosaicjson__layer__tilejson_json_get"
},
{
"metadata": {
"tags": [],
"trusted": true
},
"cell_type": "code",
"source": "r = requests.get(\n f\"{titiler_endpoint}/mosaicjson/{username}.{layername}/tilejson.json\",\n).json()\nprint(r)\n\nm = Map(\n center=((bounds[1] + bounds[3]) / 2,(bounds[0] + bounds[2]) / 2),\n zoom=10\n)\n\ntiles = TileLayer(\n url=r[\"tiles\"][0],\n min_zoom=r[\"minzoom\"],\n max_zoom=r[\"maxzoom\"],\n opacity=1\n)\n\ngeo_json = GeoJSON(\n data=geojson,\n style={\n 'opacity': 1, 'dashArray': '1', 'fillOpacity': 0, 'weight': 1\n },\n)\nm.add_layer(geo_json)\nm.add_layer(tiles)\nm",
"execution_count": null,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "import geemap.eefolium as geemap",
"execution_count": null,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "import geemap",
"execution_count": 1,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "Map = geemap.Map()",
"execution_count": 2,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "url = 'https://opendata.digitalglobe.com/events/california-fire-2020/pre-event/2018-02-16/pine-gulch-fire20/1030010076004E00.tif'",
"execution_count": 3,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "Map.add_tile_layer()",
"execution_count": null,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "Map.add_COG_layer(url, name=\"COG\")",
"execution_count": 4,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "Map",
"execution_count": 5,
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "6a6c737af1f4482a967523bdf004032a",
"version_major": 2,
"version_minor": 0
},
"text/plain": "Map(center=[39.49489764407821, -108.5072786256228], controls=(WidgetControl(options=['position'], widget=HBox(…"
},
"metadata": {},
"output_type": "display_data"
}
]
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "",
"execution_count": null,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "# import folium",
"execution_count": null,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "# layer = folium.TileLayer(tiles='https://api.cogeo.xyz/cog/tiles/WebMercatorQuad/{z}/{x}/{y}@1x?url=https%3A%2F%2Fopendata.digitalglobe.com%2Fevents%2Fcalifornia-fire-2020%2Fpre-event%2F2018-02-16%2Fpine-gulch-fire20%2F1030010076004E00.tif')",
"execution_count": null,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "url = \"https://canada-spot-ortho.s3.amazonaws.com/canada_spot_orthoimages/canada_spot5_orthoimages/S5_2007/S5_11055_6057_20070622/S5_11055_6057_20070622.json\"",
"execution_count": 6,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "Map.add_STAC_layer(url, bands=['B3', 'B2', 'B1'])",
"execution_count": 7,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "Map.addLayerControl()\nMap",
"execution_count": 8,
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "6a6c737af1f4482a967523bdf004032a",
"version_major": 2,
"version_minor": 0
},
"text/plain": "Map(bottom=99996.0, center=[60.95410634999892, -110.90184690000001], controls=(WidgetControl(options=['positio…"
},
"metadata": {},
"output_type": "display_data"
}
]
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "",
"execution_count": null,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "",
"execution_count": null,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "url2 = 'https://opendata.digitalglobe.com/events/california-fire-2020/post-event/2020-08-14/pine-gulch-fire20/10300100AAC8DD00.tif'",
"execution_count": null,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "Map.add_COG_layer(url2, name=\"COG2\")",
"execution_count": null,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "url3 = 'https://opendata.digitalglobe.com/events/california-fire-2020/post-event/2020-10-22/east-troublesome-fire20/10300100AD0CA600.tif'",
"execution_count": null,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "Map.add_COG_layer(url3, name=\"COG3\")",
"execution_count": null,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "",
"execution_count": null,
"outputs": []
}
],
"metadata": {
"hide_input": false,
"kernelspec": {
"name": "python3",
"display_name": "Python 3",
"language": "python"
},
"language_info": {
"name": "python",
"version": "3.8.5",
"mimetype": "text/x-python",
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"pygments_lexer": "ipython3",
"nbconvert_exporter": "python",
"file_extension": ".py"
},
"toc": {
"nav_menu": {},
"number_sections": true,
"sideBar": true,
"skip_h1_title": false,
"base_numbering": 1,
"title_cell": "Table of Contents",
"title_sidebar": "Contents",
"toc_cell": false,
"toc_position": {},
"toc_section_display": true,
"toc_window_display": false
},
"gist": {
"id": "",
"data": {
"description": "Working_with_MosaicJSON.ipynb",
"public": true
}
}
},
"nbformat": 4,
"nbformat_minor": 4
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment