Skip to content

Instantly share code, notes, and snippets.

@mdsumner

mdsumner/coops3.md

Created December 28, 2024 23:46
Show Gist options
  • Save mdsumner/b239881e1cd126be057d3be4dea04710 to your computer and use it in GitHub Desktop.
Save mdsumner/b239881e1cd126be057d3be4dea04710 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
coops3.md

with these settings (region unset for good measure)

export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
export AWS_S3_ENDPOINT=data.source.coop

export AWS_VIRTUAL_HOSTING=FALSE
export CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE=YES
export CPL_VSIS3_CREATE_DIR_OBJECT=TRUE
export VSIS3_CHUNK_SIZE=44
export CPL_TMPDIR=$MYSCRATCH
unset AWS_REGION
unset AWS_DEFAULT_REGION

upload with awscli works, but not with gdal ...

aws s3 cp uploadtest.tif s3://ausantarctic/mdstest/ --endpoint-url=https://data.source.coop

gdalinfo --config CPL_DEBUG ON /vsis3/ausantarctic/mdstest/uploadtest.tif
CPL: Loading configuration from /home/mdsumner/.gdal/gdalrc
HTTP: libcurl/8.0.1 OpenSSL/1.1.1l-fips zlib/1.2.13 brotli/1.0.7 zstd/1.5.0 libidn2/2.2.0 libpsl/0.20.1 (+libidn2/2.2.0) libssh/0.9.8/openssl/zlib nghttp2/1.40.0
S3: <html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
</body>
</html>

S3: GetFileSize(https://data.source.coop/ausantarctic/mdstest/uploadtest.tif)=0  response_code=400
S3: <html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
</body>
</html>

ERROR 17: Invalid AWS response: <html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
</body>
</html>

gdalinfo failed - unable to open '/vsis3/ausantarctic/mdstest/uploadtest.tif'.
@mdsumner
Copy link
Author

mdsumner commented Dec 30, 2024 •
edited
Loading

a more basic read test

#unset LD_LIBRARY_PATH

export AWS_VIRTUAL_HOSTING=FALSE
export CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE=YES
export CPL_VSIS3_CREATE_DIR_OBJECT=TRUE
export VSIS3_CHUNK_SIZE=44
export CPL_TMPDIR=$MYSCRATCH
export AWS_REGION=
export AWS_NO_SIGN_REQUEST=YES
export AWS_S3_ENDPOINT=data.source.coop
gdalinfo --config CPL_DEBUG ON /vsis3/cboettig/carbon/cogs/irrecoverable_c_total_2018.tif

@mdsumner
Copy link
Author

mdsumner commented Dec 30, 2024 •
edited
Loading 

import os
os.environ["AWS_VIRTUAL_HOSTING"]="FALSE"
os.environ["AWS_S3_ENDPOINT"]='data.source.coop'
os.environ["AWS_NO_SIGN_REQUEST"]="TRUE"
os.environ["AWS_REGION"]=""

vfs_url = "s3://cboettig/carbon/cogs/irrecoverable_c_total_2018.tif"
import rasterio
r = rasterio.open(vfs_url)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment