Skip to content

Instantly share code, notes, and snippets.

@emlys
emlys / ec2Setup.sh
Last active September 18, 2019 17:15
#!/bin/bash
# Setup for a new EC2 instance for testing Dockstore
# Install Dockstore (from https://staging.dockstore.org/quick-start)---------------------------------------------------------
# Install Java 11:
sudo apt-get update
sudo apt install -y openjdk-11-jdk
#Install the dockstore command-line program and add it to the path.
@emlys
emlys / queue_bug.py
Created November 18, 2020 21:04
Minimal example for datetime64 queue pickling behavior
import numpy
import multiprocessing
array_in = numpy.empty(1, dtype='datetime64,f4')
print(array_in.dtype['f0'].metadata) # prints None
# if you use queue.Queue instead, this doesn't happen
q = multiprocessing.Queue()
q.put(array_in)

Nested ARGS_SPEC data types examples and specification

Real model examples

leaving out all other arg properties besides "type" to save space

Coastal Blue Carbon selected args

"landcover_snapshot_csv": {
    "type": "csv",
    "structure": {
        "columns": {
@emlys
emlys / generate_users_guide.py
Last active March 29, 2021 16:52
tracer code to generate user's guide content from spec
from natcap.invest import carbon
def format_spec(name, spec):
"""Format an arg spec or subsection of an arg spec into text.
Works for the entire args spec, or any nested dictionary within it (individual args or parts of args).
This way, the generated content can be combined into the existing user's guide
at whatever level makes sense for each situation.
@emlys
emlys / wrap.py
Created April 5, 2021 22:50
Script to wrap multi-line text blocks in source code dictionary
"""
Script to wrap multi-line text blocks in dictionaries.
Made this for InVEST model ARGS_SPECs "about" properties since I'm
moving things around and it's tedious to wrap them all by hand.
This could be generalized to any multi-line text block or an entire file.
"""
import ast
import importlib
import textwrap
@emlys
emlys / is_near_comparison.py
Created April 21, 2021 21:47
comparing two methods for "is near" algorithm
import numpy
import math
from osgeo import gdal, osr
import pygeoprocessing
import timeit
FLOAT_NODATA = -1
UINT8_NODATA = 255
@emlys
emlys / post-commit
Created July 16, 2021 19:29
git post-commit hook to keep one branch updated with changes from another
#!/bin/sh
# post-commit hook to keep one branch updated with all the changes from another
# so that the target branch always has a superset of the changes in the source branch
# do this by rebasing target off of source after each commit to source
SOURCE_BRANCH=example/generate-docs
BRANCH_TO_REBASE=task/31/models-A-D
# get the current branch in the format "* branch_name"
import pygeoprocessing
import numpy
from osgeo import gdal, osr
srs = osr.SpatialReference()
srs.ImportFromEPSG(32731) # WGS84/UTM zone 31s
projection_wkt = srs.ExportToWkt()
arr = numpy.array([[0, 1, -1]], dtype=numpy.int16)
base_nodata = 0
target_datatype = gdal.GDT_Int16
@emlys
emlys / reclassify.md
Last active October 13, 2021 19:38
Using pygeoprocessing to reclassify raster values

Using pygeoprocessing to reclassify raster values

Here are some examples of how to use pygeoprocessing for reclassification. Note: tested with pygeoprocessing 2.3.2.

The reclassify_raster function

pygeoprocessing provides the reclassify_raster function which can handle basic cases. See the docstring for details.

Convert one valid value to another valid value

import pygeoprocessing
@emlys
emlys / model_references.md
Last active January 7, 2022 01:16
where to update for new model

invest

  • HISTORY.rst: Model names list in comment at top of file
  • HISTORY.rst: note that model was added
  • installer/windows/invest_installer.nsi: windows start menu link
  • installer/windows/invest_installer.nsi: windows data downloads list
  • Makefile: user's guide commit hash
  • Makefile: sample data commit hash
  • Makefile: test data commit hash (if needed)
  • Makefile: ZIPDIRS list
  • scripts/invest-autotest.py: add to dictionary