stcaf prototype run with multiple containers. Jobs orchestrated with docker compose.

README.md

Example

Warning

The data setup downloads roughly ~25 GB from the internet. This can take an hour or longer, depending on your internet connection. Be sure you have adequate storage and bandwidth.

Download the scripts into an empty directory and run

# Sets up a ./data file with input and output directories.
sh ./setup_data.sh

# Runs container workflow from end to end.
docker compose up
# Output to ./data/output. All containers will show "Exited with status code 0" when complete.

# And cleanup once your run is complete.
docker compose down

# To cleanup, delete ./data to remove all input and output data.

Projected data and output from various modules is in ./data/output.

The containerized modules used in this example are prototypes with code available at:

• https://github.com/stcaf-org/fair-temperature
• https://github.com/stcaf-org/ssp-landwaterstorage
• https://github.com/stcaf-org/tlm-sterodynamics

See these repositories for additional information on each module.

docker-compose.yaml

services:
  fair:
    image: ghcr.io/stcaf-org/fair-temperature:0.1.0
    command:
      - "--pipeline-id=1234"
      - "--output-oceantemp-file=/output/oceantemp.nc"
      - "--nsamps=20000"
      - "--output-ohc-file=/output/ohc.nc"
      - "--output-gsat-file=/output/gsat.nc"
      - "--output-climate-file=/output/climate.nc"
      - "--rcmip-file=/input/rcmip/rcmip-emissions-annual-means-v5-1-0.csv"
      - "--param-file=/input/parameters/fair_ar6_climate_params_v4.0.nc"
    restart: no
    volumes:
      - ./data/input:/input
      - ./data/output/fair:/output

  lws:
    image: ghcr.io/stcaf-org/ssp-landwaterstorage:0.1.0
    command:
      - "--pipeline-id=1234"
      - "--nsamps=20000"
      - "--output-gslr-file=/output/gslr.nc"
      - "--output-lslr-file=/output/lslr.nc"
      - "--location-file=/input/location.lst"
      - "--pophist-file=/input/UNWPP2012 population historical.csv"
      - "--reservoir-file=/input/Chao2008 groundwater impoundment.csv"
      - "--popscen-file=/input/ssp_iam_baseline_popscenarios2100.csv"
      - "--gwd-file=/input/Konikow2011 GWD.csv"
      - "--gwd-file=/input/Wada2012 GWD.csv"
      - "--gwd-file=/input/Pokhrel2012 GWD.csv"
      - "--fp-file=/input/REL_GROUNDWATER_NOMASK.nc"
    restart: no
    volumes:
      - ./data/input:/input
      - ./data/output/lws:/output

  sterodynamics:
    image: ghcr.io/stcaf-org/tlm-sterodynamics:0.2.0
    command:
      - "--pipeline-id=1234"
      - "--scenario=ssp585"
      - "--nsamps=20000"
      - "--model-dir=/input/cmip6/"
      - "--location-file=/input/location.lst"
      - "--output-lslr-file=/output/lslr.nc"
      - "--output-gslr-file=/output/gslr.nc"
      - "--expansion-coefficients-file=/input/scmpy2LM_RCMIP_CMIP6calpm_n18_expcoefs.nc"
      - "--gsat-rmses-file=/input/scmpy2LM_RCMIP_CMIP6calpm_n17_gsat_rmse.nc"
      # Depends on output file from fair run. Output is attached in additional volume.
      - "--climate-data-file=/fair/climate.nc"
    volumes:
      - ./data/output/fair:/fair
      - ./data/input:/input
      - ./data/output/sterodynamics:/output
    restart: no
    depends_on:
      fair:
        condition: service_completed_successfully

setup_data.sh

#! /usr/bin/env bash

set -euxo pipefail


IN_DIR="./data/input"

mkdir -p "${IN_DIR}"

# Create common location file. Used by multiple modules.
echo "New_York	12	40.70	-74.01" > "${IN_DIR}/location.lst"

# Input data for ssp-landwaterstorage.
curl -sL https://zenodo.org/record/7478192/files/ssp_landwaterstorage_postprocess_data.tgz | tar -zx -C "${IN_DIR}"
curl -sL https://zenodo.org/record/7478192/files/ssp_landwaterstorage_preprocess_data.tgz | tar -zx -C "${IN_DIR}"


mkdir -p "./data/output/lws"


# Input for fair-temperature.
curl -sL https://zenodo.org/record/7478192/files/fair_temperature_fit_data.tgz | tar -zx -C "${IN_DIR}"
curl -sL https://zenodo.org/record/7478192/files/fair_temperature_preprocess_data.tgz | tar -zx -C "${IN_DIR}"

mkdir -p "./data/output/fair"


# Input data for tlm-sterodynamics
curl -sL https://zenodo.org/record/7478192/files/tlm_sterodynamics_preprocess_data.tgz | tar -zx -C "${IN_DIR}"
# BEWARE: This file is ~25 GB. It can take a long time to download and may error if your internet connection is interrupted.
curl -sL https://zenodo.org/record/7478192/files/tlm_sterodynamics_cmip6_data.tgz | tar -zx -C ./data/input

mkdir -p "./data/output/sterodynamics"