Skip to content

Instantly share code, notes, and snippets.

@robbibt

robbibt/s3_to_vrt.py

Created February 7, 2024 03:44
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save robbibt/3b85bfc5c394bc7bba536969455b1426 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save robbibt/3b85bfc5c394bc7bba536969455b1426 to your computer and use it in GitHub Desktop.
Download ZIP
Create virtual raster for files on S3
Raw
s3_to_vrt.py
import s3fs
import pandas as pd
s3 = s3fs.S3FileSystem(anon=True)
file_list = s3.glob(
"dea-public-data-dev/derivative/ga_ls8cls9c_gm_cyear_3/4-0-0/*/*/2023--P1Y/ga_ls8cls9c_gm_cyear_3_*_2023--P1Y_final_nbart_red.tif"
)
pd.DataFrame(file_list).iloc[:, 0].str.replace(
"dea-public-data-dev/",
"/vsicurl/https://dea-public-data-dev.s3-ap-southeast-2.amazonaws.com/",
).to_csv("geomad_red.txt", sep="\t", index=False, header=False)
!gdalbuildvrt -input_file_list geomad_red.txt -vrtnodata -9999 geomad_red.vrt
@robbibt
Copy link
Copy Markdown
Author

robbibt commented Feb 7, 2024

Also helps to manually add statistics:

    <Metadata>
      <MDI key="STATISTICS_MAXIMUM">10000</MDI>
      <MDI key="STATISTICS_MEAN">1000</MDI>
      <MDI key="STATISTICS_MINIMUM">0</MDI>
    </Metadata>

And remove the OverviewList line:

  <OverviewList resampling="nearest">2 4 8 16</OverviewList>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment