Eugene Cheipesh echeipesh

Stream Processing for Polygonal Summary over Rasters

Problem

We have a stream of requests for aggregation for polygons over a set of rasters (eg: Sum of all values under polygon (AOI) in rasters A, B and C. The set of rasters is potentially unbounded and the aggregation itself can be sophisticated, where the aggregation value is continguent on per-pixel values of A and B and not their summaries (eg. Producing multi-dimensional histogram).

Challanges

The request stream does not have uniform rate
The requests in a stream have relative priority
Stream latency needs to be optimized
Stream throughput needs to be optimized

	{
	"baseLayer": {
	"tilejson": "2.2.0",
	"name": "base",
	"version": "1.0.0",
	"scheme": "xyz",
	"tiles": [
	"https://a.tile.openstreetmap.org/{z}/{x}/{y}.png"
	]
	},

	{
	"baseLayer": {
	"tilejson": "2.2.0",
	"name": "base",
	"version": "1.0.0",
	"scheme": "xyz",
	"tiles": [
	"https://a.tile.openstreetmap.org/{z}/{x}/{y}.png"
	]
	},

	/*
	* Copyright 2018 Azavea
	*
	* Licensed under the Apache License, Version 2.0 (the "License");
	* you may not use this file except in compliance with the License.
	* You may obtain a copy of the License at
	*
	* http://www.apache.org/licenses/LICENSE-2.0
	*
	* Unless required by applicable law or agreed to in writing, software

	package geotrellis.server.ogc.conf

	import pureconfig._
	import pureconfig.error.{ConfigReaderFailures, NoValidCoproductChoiceFound}
	import geotrellis.contrib.vlm.RasterSource
	import geotrellis.contrib.vlm.avro.GeotrellisRasterSource
	import geotrellis.contrib.vlm.gdal.GDALRasterSource
	import geotrellis.contrib.vlm.geotiff.GeoTiffRasterSource
	import geotrellis.spark.LayerId

	class CogLayerReader {
	def read(...): CogLayer
	}

	class Layer[K: SpatialComponent, V] {
	def extent: Extent
	def layout: GridBounds
	def get(key: K): V
	}

	{
	"cells": [
	{
	"cell_type": "code",
	"execution_count": 1,
	"metadata": {
	"collapsed": false
	},
	"outputs": [],
	"source": [

	def partitionWindows(
	segmentLayout: GeoTiffSegmentLayout,
	windows: IndexedSeq[GridBounds],
	maxPartitionSize: Long
	): Array[Array[GridBounds]] = {

	val partition = mutable.ArrayBuilder.make[GridBounds]
	val partitions = mutable.ArrayBuilder.make[Array[GridBounds]]
	var partitionSize: Long = 0l
	var windowsProcessed: Int = 0

Eugene Cheipesh echeipesh

Stream Processing for Polygonal Summary over Rasters

Problem

Table of Contents