Skip to content

Instantly share code, notes, and snippets.

@nickrsan
Last active December 17, 2024 01:05
Show Gist options
  • Save nickrsan/537b55aef233a9e4b77fa7611612fe07 to your computer and use it in GitHub Desktop.
Save nickrsan/537b55aef233a9e4b77fa7611612fe07 to your computer and use it in GitHub Desktop.
Sample code to use in retrieving tiles from Hexagon's WMTS with Oauth authentication to retrieve and manage a token easily so you can download tiles
"""
Based on the docs from Hexagon at https://docs.hxgncontent.com/hxgn-content-program-resources/Latest-version/batch-processing
- they are sparse docs, and then you need to authenticate a specific way, so this file includes an example sparse implementation
of how to use their authentication within Python.
The HexagonManager class takes your OAuth Client ID and OAuth Secret as the parameters client_id and client_secret.
After providing those when instantiating the class, you can just call the get_tile method on the class, and it will
handle retrieving, managing, and refreshing the authentication token for you for the requests to the service.
The implementation is sparse in that it needs to better handle errors from the server, doesn't provide logging,
and doesn't help you find the right tiles at each scale. That's up to you, but this handles the bare tile retrieval for you.
You must get an OAuth Client ID and OAuth Secret from your organization's hexagon administrator, who can provide that information
from their user account manager by creating a specific account for you. If they do not know how, they can reach out to CA Dept. of
Technology.
Example Usage after copying the code into your own file (or add your own code to the bottom):
```
hexagon = HexagonManager(client_id="MY_CLIENT_ID_FROM_YOUR_ADMIN", client_secret="MY_CLIENT_SECRET_FROM_YOUR_ADMIN")
tile_path = hexagon.get_tile(10,384,160) # get a tile in the coastal mountains near the north coast at a somewhat medium scale
print(tile_path) # or do work here - can loop over get_tile, but you'll want some kind of delay to not receive a 429 rate-limit response from the server. They don't specify the rate limit.
```
Note that Hexagon provides other methods for downloading bulk data that you can use, other than retrieving individual tiles, though
these other methods may introduce additional resampling into the data. See their information at https://docs.hxgncontent.com/hxgn-content-program-resources/Latest-version/streaming-api
for more details on options. Specifically, there may be a method to get all the data within an area of interest and then wait for the response, but they
do not provide documentation or examples of this capability.
"""
import os
import tempfile
import requests
from datetime import datetime, UTC
TOKEN_URL = "https://services.hxgncontent.com/streaming/oauth/token?grant_type=client_credentials"
BATCH_WMTS_URL = "https://services.hxgncontent.com/orders/wmts?"
BATCH_WMS_URL = "https://services.hxgncontent.com/orders/wms?"
PARAMS = "/1.0.0/HxGN_Imagery/default/WebMercator/"
# ZOOMS = {10: (list(range()))
class HexagonManager():
def __init__(self, client_id, client_secret, wmts_url=BATCH_WMTS_URL, url_params=PARAMS, token_url=TOKEN_URL):
self._token_info = None
self._reauthorize_after = datetime.now(tz=UTC)
self.token_url = token_url
self.client_id = client_id
self.client_secret = client_secret
self.wmts_url = wmts_url
self.url_params = url_params
self.default_folder = tempfile.mkdtemp(prefix="hexagon_") # where to save tiles
def _get_token(self):
"""Internal method - sends the request to a token URL to get the auth token to use, calculates its valid period
Raises:
RuntimeError: _description_
Returns:
_type_: _description_
"""
full_url = f"{self.token_url}&client_id={self.client_id}&client_secret={self.client_secret}" # not safe for untrusted inputs. Fine if we know our values
response = requests.get(full_url)
if response.status_code == 200:
body = response.json() # this has an access token and an expiration
reauthorize_after_seconds = datetime.now(tz=UTC).timestamp() + body["expires_in"] - 5 # add the expiration amount to the timestamp, then subtract 5 seconds to make sure we have a buffer
reauthorize_dt = datetime.fromtimestamp(reauthorize_after_seconds, tz=UTC) # get the timestamp we'll need to reauthorize after
body["reauthorize_after"] = reauthorize_dt
return body
else:
raise RuntimeError(f"Couldn't get access token, server returned status code {response.status_code} and message {response.content}")
@property
def token(self):
if datetime.now(UTC) > self._reauthorize_after: # if our existing token is no longer valid, or we don't have one at all
self._token_info = self._get_token() # authenticate for a token
self._reauthorize_after = self._token_info["reauthorize_after"] # keep track of when we should reauthorize again in the future.
return self._token_info["access_token"]
def get_tile(self, matrix, row, col, path=None):
"""
Automatically composes the correct request to the WMS server for the tile, then returns the path to the downloaded tile.
Returns the path to the downloaded tile if it downloaded one, otherwise raises the HTTP status code the server provided.
If the server returns status code 429, then you need to wait longer between calls to this function.
Args:
matrix (_type_): The WMTS tile matrix (ie Zoom) to request tiles from
row (_type_): The WMTS row within the tile matrix (ie, y)
col (_type_): The WMTS column within the tile matrix (ie, x)
path (str or None): The full output path for the downloaded tile, including ".jpg" extension. If None, then a path in Temp will be generated.
"""
filename = os.path.join(str(matrix),str(row),f"{col}.jpg")
file_url = f"{matrix}/{row}/{col}.jpg"
url = self.wmts_url + self.url_params + f"{file_url}&access_token={self.token}"
print(f"fetching {url}")
response = requests.get(url)
if response.status_code == 200:
if len(response.content) < 1000:
print("Likely empty tile")
if path is None:
path = os.path.join(self.default_folder, filename)
os.makedirs(os.path.split(path)[0], exist_ok=True) # make all the directories needed
with open(path, 'wb') as output:
output.write(response.content) # this isn't really a good way to do this for large files, but is likely fine enough for small ones
return path
else:
raise RuntimeError(f"Server returned alternative status code: {response.status_code}. Included body '{response.content}'")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment