chatgpt-download-raster-landcover-data.md

Task

The download URLs look like:

https://s3-eu-west-1.amazonaws.com/vito.landcover.global/v3.0.1/2019/E000N60/E000N60_PROBAV_LC100_global_v3.0.1_2019-nrt_Bare-CoverFraction-layer_EPSG-4326.tif

template:

https://s3-eu-west-1.amazonaws.com/vito.landcover.global/v3.0.1/2019/{longitude}{latitude}/{longitude}{latitude}_PROBAV_LC100_global_v3.0.1_2019-nrt_{kind}-CoverFraction-layer_EPSG-4326.tif

latitudes: S40, S20, N00, N20, N40, N60, N80 longitudes: W180, W160, W140, W120, W100, W080, W060, W040, W020, E000, E020, E040, E060, E080, E100, E120, E140, E160 kinds: Snow, Bare, Grass, Crops, Tree, BuiltUp

Download all files into a sources folder.

Write a python script which loops through the filenames and downloads them to the sources folder with system calls using the cli tool axel which has parallel downloads with the -p -n 10 option.

Result

import os

# Define the base URL and the list of latitudes, longitudes, and kinds
base_url = "https://s3-eu-west-1.amazonaws.com/vito.landcover.global/v3.0.1/2019/"
latitudes = ["S40", "S20", "N00", "N20", "N40", "N60", "N80"]
longitudes = ["W180", "W160", "W140", "W120", "W100", "W080", "W060", "W040", "W020", "E000", "E020", "E040", "E060", "E080", "E100", "E120", "E140", "E160"]
kinds = ["Snow", "Bare", "Grass", "Crops", "Tree", "BuiltUp"]

# Create the sources folder if it doesn't already exist
if not os.path.exists("sources"):
    os.makedirs("sources")

# Loop through the combinations of latitude, longitude, and kind and download each file
for latitude in latitudes:
    for longitude in longitudes:
        for kind in kinds:
            filename = f"{longitude}{latitude}_PROBAV_LC100_global_v3.0.1_2019-nrt_{kind}-CoverFraction-layer_EPSG-4326.tif"
            url = base_url + f"{latitude}/{filename}"
            command = f"axel -p -n 10 {url} -o sources/{filename}"
            os.system(command)

Manual bug fix

-url = base_url + f"{latitude}/{filename}"
+url = base_url + f"{longitude}{latitude}/{filename}"