Skip to content

Instantly share code, notes, and snippets.

@amotl
Last active January 19, 2025 03:09
Show Gist options
  • Save amotl/67251a472f849fa091a47355b31121e1 to your computer and use it in GitHub Desktop.
Save amotl/67251a472f849fa091a47355b31121e1 to your computer and use it in GitHub Desktop.
Concisely fetch data from remote resources in Python, with caching
#!/usr/bin/env python
"""
## About
Concisely fetch data from remote resources in Python, with caching.
## Synopsis
```
uv run acquire_dataset_cached.py
```
## Hishel
An elegant HTTP Cache implementation for HTTPX and HTTP Core.
https://hishel.com/
https://github.com/karpetrosyan/hishel
"""
# /// script
# requires-python = ">=3.10"
# dependencies = [
# "hishel",
# ]
# ///
import hishel
# The dataset resource URL.
DATASET_URL = "https://cdn.crate.io/downloads/datasets/cratedb-datasets/academy/chicago-data/taxi_rides_apr_2024.json.gz"
# A generic HTTP client, with caching.
http = hishel.CacheClient()
def main():
"""
Acquire resource, and display a few bytes worth of data.
"""
response = http.get(DATASET_URL)
print("response:", response.content[:1009] + b"...")
if __name__ == "__main__":
main()
@amotl
Copy link
Author

amotl commented Jan 19, 2025

Would probably be good to add to the README of cratedb-datasets. wdyt?

/cc @WalBeh, @simonprickett

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment