Timing pour points implementations

conclusions.txt

Conclusions:

* On the Colombia DEM (15030 x 20631 pixels) and the Gura DEM (1939 x 603 pixels) the
  delineateit implementation reliably runs in 1/3 the time of the proposed
  `_ManagedRaster`-based pour points implementation.

test_pour_points.py

import logging
import os
import pygeoprocessing.routing
import time
from natcap.invest.delineateit import delineateit
import numpy

logging.basicConfig(level=logging.INFO)

def timeit_dec(label, n=5):
    def new_func(orig_func):
        def wrapped_func(*args, **kwargs):
            print(f'Timing {label}')
            times = []
            for i in range(1, n+1):
                start_time = time.time()

                orig_func(*args, **kwargs)

                finish_time = time.time() - start_time
                times.append(finish_time)
                print(f'{label} #{i}: {finish_time}')

            print(f'{label} mean: {numpy.mean(times)}')
        return wrapped_func
    return new_func


@timeit_dec('delineateit', n=5)
def test_del(flow_dir, target_vector):
    if os.path.exists(target_vector):
        os.remove(target_vector)
    delineateit.detect_pour_points((flow_dir, 1), target_vector)

@timeit_dec('pygeoprocessing', n=5)
def test_pgp(flow_dir, target_vector):
    if os.path.exists(target_vector):
        os.remove(target_vector)
    pygeoprocessing.routing.detect_outlets((flow_dir, 1), target_vector)


if __name__ == '__main__':
    COL_DEM = '/Users/jdouglass/workspace/invest-natcap.permitting/data/colombia_tool_data/DEM.tif'
    GURA_DEM = '/Users/jdouglass/workspace/natcap/invest/data/invest-sample-data/RouteDEM/DEM_gura.tif'

    for dem, prefix in [
            (COL_DEM, 'colombia'),
            (GURA_DEM, 'gura')]:
        flow_dir = f'{prefix}_flow_dir.tif'

        if not os.path.exists(flow_dir):
            pygeoprocessing.routing.flow_dir_d8((dem, 1), flow_dir)

        test_del(flow_dir, f'{prefix}_outlets_del.gpkg')
        test_pgp(flow_dir, f'{prefix}_outlets_pgp.gpkg')

testing_colombia_log.txt

Timing delineateit
	- 'VirtualXPath'	[XML Path Language - XPath]
delineateit #1: 11.232605934143066
	- 'VirtualXPath'	[XML Path Language - XPath]
delineateit #2: 10.149876832962036
	- 'VirtualXPath'	[XML Path Language - XPath]
delineateit #3: 10.220314025878906
	- 'VirtualXPath'	[XML Path Language - XPath]
delineateit #4: 10.217549800872803
	- 'VirtualXPath'	[XML Path Language - XPath]
delineateit #5: 9.811689853668213
delineateit mean: 10.326407289505005
Timing pygeoprocessing
	- 'VirtualXPath'	[XML Path Language - XPath]
INFO:pygeoprocessing.routing.routing:outlet detection: 26.1 complete
INFO:pygeoprocessing.routing.routing:outlet detection: 43.4 complete
INFO:pygeoprocessing.routing.routing:outlet detection: 59.6 complete
INFO:pygeoprocessing.routing.routing:outlet detection: 75.7 complete
INFO:pygeoprocessing.routing.routing:outlet detection: committing transaction
INFO:pygeoprocessing.routing.routing:outlet detection: done
pygeoprocessing #1: 30.411698818206787
	- 'VirtualXPath'	[XML Path Language - XPath]
INFO:pygeoprocessing.routing.routing:outlet detection: 21.1 complete
INFO:pygeoprocessing.routing.routing:outlet detection: 37.2 complete
INFO:pygeoprocessing.routing.routing:outlet detection: 52.1 complete
INFO:pygeoprocessing.routing.routing:outlet detection: 64.5 complete
INFO:pygeoprocessing.routing.routing:outlet detection: 80.7 complete
INFO:pygeoprocessing.routing.routing:outlet detection: committing transaction
INFO:pygeoprocessing.routing.routing:outlet detection: done
pygeoprocessing #2: 35.109846115112305
	- 'VirtualXPath'	[XML Path Language - XPath]
INFO:pygeoprocessing.routing.routing:outlet detection: 21.1 complete
INFO:pygeoprocessing.routing.routing:outlet detection: 40.9 complete
INFO:pygeoprocessing.routing.routing:outlet detection: 55.8 complete
INFO:pygeoprocessing.routing.routing:outlet detection: 70.7 complete
INFO:pygeoprocessing.routing.routing:outlet detection: 93.1 complete
INFO:pygeoprocessing.routing.routing:outlet detection: committing transaction
INFO:pygeoprocessing.routing.routing:outlet detection: done
pygeoprocessing #3: 31.275635957717896
	- 'VirtualXPath'	[XML Path Language - XPath]
INFO:pygeoprocessing.routing.routing:outlet detection: 26.1 complete
INFO:pygeoprocessing.routing.routing:outlet detection: 43.4 complete
INFO:pygeoprocessing.routing.routing:outlet detection: 58.3 complete
INFO:pygeoprocessing.routing.routing:outlet detection: 73.2 complete
INFO:pygeoprocessing.routing.routing:outlet detection: 95.5 complete
INFO:pygeoprocessing.routing.routing:outlet detection: committing transaction
INFO:pygeoprocessing.routing.routing:outlet detection: done
pygeoprocessing #4: 31.349935054779053
	- 'VirtualXPath'	[XML Path Language - XPath]
INFO:pygeoprocessing.routing.routing:outlet detection: 24.8 complete
INFO:pygeoprocessing.routing.routing:outlet detection: 42.2 complete
INFO:pygeoprocessing.routing.routing:outlet detection: 58.3 complete
INFO:pygeoprocessing.routing.routing:outlet detection: 74.5 complete
INFO:pygeoprocessing.routing.routing:outlet detection: 96.8 complete
INFO:pygeoprocessing.routing.routing:outlet detection: committing transaction
INFO:pygeoprocessing.routing.routing:outlet detection: done
pygeoprocessing #5: 30.698140144348145
pygeoprocessing mean: 31.769051218032835

testing_gura_log.txt

Timing delineateit
	- 'VirtualXPath'	[XML Path Language - XPath]
delineateit #1: 0.02996683120727539
	- 'VirtualXPath'	[XML Path Language - XPath]
delineateit #2: 0.03027796745300293
	- 'VirtualXPath'	[XML Path Language - XPath]
delineateit #3: 0.031976938247680664
	- 'VirtualXPath'	[XML Path Language - XPath]
delineateit #4: 0.03491616249084473
	- 'VirtualXPath'	[XML Path Language - XPath]
delineateit #5: 0.031674861907958984
delineateit mean: 0.031762552261352536
Timing pygeoprocessing
	- 'VirtualXPath'	[XML Path Language - XPath]
INFO:pygeoprocessing.routing.routing:outlet detection: committing transaction
INFO:pygeoprocessing.routing.routing:outlet detection: done
pygeoprocessing #1: 0.10240483283996582
	- 'VirtualXPath'	[XML Path Language - XPath]
INFO:pygeoprocessing.routing.routing:outlet detection: committing transaction
INFO:pygeoprocessing.routing.routing:outlet detection: done
pygeoprocessing #2: 0.09025692939758301
	- 'VirtualXPath'	[XML Path Language - XPath]
INFO:pygeoprocessing.routing.routing:outlet detection: committing transaction
INFO:pygeoprocessing.routing.routing:outlet detection: done
pygeoprocessing #3: 0.09812188148498535
	- 'VirtualXPath'	[XML Path Language - XPath]
INFO:pygeoprocessing.routing.routing:outlet detection: committing transaction
INFO:pygeoprocessing.routing.routing:outlet detection: done
pygeoprocessing #4: 0.09271717071533203
	- 'VirtualXPath'	[XML Path Language - XPath]
INFO:pygeoprocessing.routing.routing:outlet detection: committing transaction
INFO:pygeoprocessing.routing.routing:outlet detection: done
pygeoprocessing #5: 0.0834190845489502
pygeoprocessing mean: 0.09338397979736328