Skip to content

Instantly share code, notes, and snippets.

@omad
Last active December 8, 2017 01:08
Show Gist options
  • Save omad/b00a74eb143ff93fbd2bb993df200144 to your computer and use it in GitHub Desktop.
Save omad/b00a74eb143ff93fbd2bb993df200144 to your computer and use it in GitHub Desktop.
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<!-- 2017-12-08 Fri 11:59 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Discussion points - DataCube S3 Refactor</title>
<meta name="generator" content="Org mode" />
<meta name="author" content="Damien Ayers" />
<style type="text/css">
<!--/*--><![CDATA[/*><!--*/
.title { text-align: center;
margin-bottom: .2em; }
.subtitle { text-align: center;
font-size: medium;
font-weight: bold;
margin-top:0; }
.todo { font-family: monospace; color: red; }
.done { font-family: monospace; color: green; }
.priority { font-family: monospace; color: orange; }
.tag { background-color: #eee; font-family: monospace;
padding: 2px; font-size: 80%; font-weight: normal; }
.timestamp { color: #bebebe; }
.timestamp-kwd { color: #5f9ea0; }
.org-right { margin-left: auto; margin-right: 0px; text-align: right; }
.org-left { margin-left: 0px; margin-right: auto; text-align: left; }
.org-center { margin-left: auto; margin-right: auto; text-align: center; }
.underline { text-decoration: underline; }
#postamble p, #preamble p { font-size: 90%; margin: .2em; }
p.verse { margin-left: 3%; }
pre {
border: 1px solid #ccc;
box-shadow: 3px 3px 3px #eee;
padding: 8pt;
font-family: monospace;
overflow: auto;
margin: 1.2em;
}
pre.src {
position: relative;
overflow: visible;
padding-top: 1.2em;
}
pre.src:before {
display: none;
position: absolute;
background-color: white;
top: -10px;
right: 10px;
padding: 3px;
border: 1px solid black;
}
pre.src:hover:before { display: inline;}
/* Languages per Org manual */
pre.src-asymptote:before { content: 'Asymptote'; }
pre.src-awk:before { content: 'Awk'; }
pre.src-C:before { content: 'C'; }
/* pre.src-C++ doesn't work in CSS */
pre.src-clojure:before { content: 'Clojure'; }
pre.src-css:before { content: 'CSS'; }
pre.src-D:before { content: 'D'; }
pre.src-ditaa:before { content: 'ditaa'; }
pre.src-dot:before { content: 'Graphviz'; }
pre.src-calc:before { content: 'Emacs Calc'; }
pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
pre.src-fortran:before { content: 'Fortran'; }
pre.src-gnuplot:before { content: 'gnuplot'; }
pre.src-haskell:before { content: 'Haskell'; }
pre.src-hledger:before { content: 'hledger'; }
pre.src-java:before { content: 'Java'; }
pre.src-js:before { content: 'Javascript'; }
pre.src-latex:before { content: 'LaTeX'; }
pre.src-ledger:before { content: 'Ledger'; }
pre.src-lisp:before { content: 'Lisp'; }
pre.src-lilypond:before { content: 'Lilypond'; }
pre.src-lua:before { content: 'Lua'; }
pre.src-matlab:before { content: 'MATLAB'; }
pre.src-mscgen:before { content: 'Mscgen'; }
pre.src-ocaml:before { content: 'Objective Caml'; }
pre.src-octave:before { content: 'Octave'; }
pre.src-org:before { content: 'Org mode'; }
pre.src-oz:before { content: 'OZ'; }
pre.src-plantuml:before { content: 'Plantuml'; }
pre.src-processing:before { content: 'Processing.js'; }
pre.src-python:before { content: 'Python'; }
pre.src-R:before { content: 'R'; }
pre.src-ruby:before { content: 'Ruby'; }
pre.src-sass:before { content: 'Sass'; }
pre.src-scheme:before { content: 'Scheme'; }
pre.src-screen:before { content: 'Gnu Screen'; }
pre.src-sed:before { content: 'Sed'; }
pre.src-sh:before { content: 'shell'; }
pre.src-sql:before { content: 'SQL'; }
pre.src-sqlite:before { content: 'SQLite'; }
/* additional languages in org.el's org-babel-load-languages alist */
pre.src-forth:before { content: 'Forth'; }
pre.src-io:before { content: 'IO'; }
pre.src-J:before { content: 'J'; }
pre.src-makefile:before { content: 'Makefile'; }
pre.src-maxima:before { content: 'Maxima'; }
pre.src-perl:before { content: 'Perl'; }
pre.src-picolisp:before { content: 'Pico Lisp'; }
pre.src-scala:before { content: 'Scala'; }
pre.src-shell:before { content: 'Shell Script'; }
pre.src-ebnf2ps:before { content: 'ebfn2ps'; }
/* additional language identifiers per "defun org-babel-execute"
in ob-*.el */
pre.src-cpp:before { content: 'C++'; }
pre.src-abc:before { content: 'ABC'; }
pre.src-coq:before { content: 'Coq'; }
pre.src-groovy:before { content: 'Groovy'; }
/* additional language identifiers from org-babel-shell-names in
ob-shell.el: ob-shell is the only babel language using a lambda to put
the execution function name together. */
pre.src-bash:before { content: 'bash'; }
pre.src-csh:before { content: 'csh'; }
pre.src-ash:before { content: 'ash'; }
pre.src-dash:before { content: 'dash'; }
pre.src-ksh:before { content: 'ksh'; }
pre.src-mksh:before { content: 'mksh'; }
pre.src-posh:before { content: 'posh'; }
/* Additional Emacs modes also supported by the LaTeX listings package */
pre.src-ada:before { content: 'Ada'; }
pre.src-asm:before { content: 'Assembler'; }
pre.src-caml:before { content: 'Caml'; }
pre.src-delphi:before { content: 'Delphi'; }
pre.src-html:before { content: 'HTML'; }
pre.src-idl:before { content: 'IDL'; }
pre.src-mercury:before { content: 'Mercury'; }
pre.src-metapost:before { content: 'MetaPost'; }
pre.src-modula-2:before { content: 'Modula-2'; }
pre.src-pascal:before { content: 'Pascal'; }
pre.src-ps:before { content: 'PostScript'; }
pre.src-prolog:before { content: 'Prolog'; }
pre.src-simula:before { content: 'Simula'; }
pre.src-tcl:before { content: 'tcl'; }
pre.src-tex:before { content: 'TeX'; }
pre.src-plain-tex:before { content: 'Plain TeX'; }
pre.src-verilog:before { content: 'Verilog'; }
pre.src-vhdl:before { content: 'VHDL'; }
pre.src-xml:before { content: 'XML'; }
pre.src-nxml:before { content: 'XML'; }
/* add a generic configuration mode; LaTeX export needs an additional
(add-to-list 'org-latex-listings-langs '(conf " ")) in .emacs */
pre.src-conf:before { content: 'Configuration File'; }
table { border-collapse:collapse; }
caption.t-above { caption-side: top; }
caption.t-bottom { caption-side: bottom; }
td, th { vertical-align:top; }
th.org-right { text-align: center; }
th.org-left { text-align: center; }
th.org-center { text-align: center; }
td.org-right { text-align: right; }
td.org-left { text-align: left; }
td.org-center { text-align: center; }
dt { font-weight: bold; }
.footpara { display: inline; }
.footdef { margin-bottom: 1em; }
.figure { padding: 1em; }
.figure p { text-align: center; }
.inlinetask {
padding: 10px;
border: 2px solid gray;
margin: 10px;
background: #ffffcc;
}
#org-div-home-and-up
{ text-align: right; font-size: 70%; white-space: nowrap; }
textarea { overflow-x: auto; }
.linenr { font-size: smaller }
.code-highlighted { background-color: #ffff00; }
.org-info-js_info-navigation { border-style: none; }
#org-info-js_console-label
{ font-size: 10px; font-weight: bold; white-space: nowrap; }
.org-info-js_search-highlight
{ background-color: #ffff00; color: #000000; font-weight: bold; }
.org-svg { width: 90%; }
/*]]>*/-->
</style>
<script type="text/javascript">
/*
@licstart The following is the entire license notice for the
JavaScript code in this tag.
Copyright (C) 2012-2017 Free Software Foundation, Inc.
The JavaScript code in this tag is free software: you can
redistribute it and/or modify it under the terms of the GNU
General Public License (GNU GPL) as published by the Free Software
Foundation, either version 3 of the License, or (at your option)
any later version. The code is distributed WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE. See the GNU GPL for more details.
As additional permission under GNU GPL version 3 section 7, you
may distribute non-source (e.g., minimized or compacted) forms of
that code without the copy of the GNU GPL normally required by
section 4, provided you include this license notice and a URL
through which recipients can access the Corresponding Source.
@licend The above is the entire license notice
for the JavaScript code in this tag.
*/
<!--/*--><![CDATA[/*><!--*/
function CodeHighlightOn(elem, id)
{
var target = document.getElementById(id);
if(null != target) {
elem.cacheClassElem = elem.className;
elem.cacheClassTarget = target.className;
target.className = "code-highlighted";
elem.className = "code-highlighted";
}
}
function CodeHighlightOff(elem, id)
{
var target = document.getElementById(id);
if(elem.cacheClassElem)
elem.className = elem.cacheClassElem;
if(elem.cacheClassTarget)
target.className = elem.cacheClassTarget;
}
/*]]>*///-->
</script>
</head>
<body>
<div id="content">
<h1 class="title">Discussion points - DataCube S3 Refactor</h1>
<div id="table-of-contents">
<h2>Table of Contents</h2>
<div id="text-table-of-contents">
<ul>
<li><a href="#org772f239">1. Must Change</a></li>
<li><a href="#orgcfdea83">2. Use Cases</a>
<ul>
<li><a href="#org6397913">2.1. Simple</a></li>
<li><a href="#org50348e2">2.2. Compute jobs at NCI</a></li>
</ul>
</li>
<li><a href="#org4733516">3. Configuration</a></li>
<li><a href="#orga3a5ec0">4. Types of drivers/extension points</a>
<ul>
<li><a href="#org169fa1a">4.1. Database extension</a></li>
<li><a href="#org663be17">4.2. Storage read</a></li>
<li><a href="#orgb70fda0">4.3. Storage write</a></li>
</ul>
</li>
<li><a href="#orga50e33b">5. Proposed architecture</a></li>
<li><a href="#org4216c9d">6. Driver resolution for querying for data</a></li>
<li><a href="#org9cb81e4">7. Driver resolution for loading data</a></li>
</ul>
</div>
</div>
<div id="outline-container-org772f239" class="outline-2">
<h2 id="org772f239"><span class="section-number-2">1</span> Must Change</h2>
<div class="outline-text-2" id="text-1">
<ol class="org-ol">
<li>URI schema cannot be <code>s3://</code>, that is already a standard protocol, the
<code>uri_body</code> that is generated is meaningless and is ignored.</li>
<li>Change how connections to database are managed</li>
<li></li>
</ol>
</div>
</div>
<div id="outline-container-orgcfdea83" class="outline-2">
<h2 id="orgcfdea83"><span class="section-number-2">2</span> Use Cases</h2>
<div class="outline-text-2" id="text-2">
</div>
<div id="outline-container-org6397913" class="outline-3">
<h3 id="org6397913"><span class="section-number-3">2.1</span> Simple</h3>
<div class="outline-text-3" id="text-2-1">
<div class="figure">
<p><object type="image/svg+xml" data="simple_use_case.svg" class="org-svg">
Sorry, your browser does not support SVG.</object>
</p>
</div>
</div>
</div>
<div id="outline-container-org50348e2" class="outline-3">
<h3 id="org50348e2"><span class="section-number-3">2.2</span> Compute jobs at NCI</h3>
<div class="outline-text-3" id="text-2-2">
<div class="figure">
<p><object type="image/svg+xml" data="nci_compute_job.svg" class="org-svg">
Sorry, your browser does not support SVG.</object>
</p>
</div>
</div>
</div>
</div>
<div id="outline-container-org4733516" class="outline-2">
<h2 id="org4733516"><span class="section-number-2">3</span> Configuration</h2>
<div class="outline-text-2" id="text-3">
<ul class="org-ul">
<li>When the library is loaded, it will find a list of available datacubes/environments</li>
<li>An environment specifies which database to connect to, it could also specify which
database driver to use.</li>
<li></li>
</ul>
<p>
I think it only makes sense for a `Datacube`
</p>
</div>
</div>
<div id="outline-container-orga3a5ec0" class="outline-2">
<h2 id="orga3a5ec0"><span class="section-number-2">4</span> Types of drivers/extension points</h2>
<div class="outline-text-2" id="text-4">
</div>
<div id="outline-container-org169fa1a" class="outline-3">
<h3 id="org169fa1a"><span class="section-number-3">4.1</span> Database extension</h3>
<div class="outline-text-3" id="text-4-1">
<ul class="org-ul">
<li>Options
<ul class="org-ul">
<li></li>
<li>Only one can exist for a single instance of DataCube</li>
</ul></li>
<li></li>
</ul>
<div class="figure">
<p><object type="image/svg+xml" data="proposed_database_extension_classes.svg" class="org-svg">
Sorry, your browser does not support SVG.</object>
</p>
</div>
</div>
</div>
<div id="outline-container-org663be17" class="outline-3">
<h3 id="org663be17"><span class="section-number-3">4.2</span> Storage read</h3>
<div class="outline-text-3" id="text-4-2">
<ul class="org-ul">
<li>Multiple should be available in the same Datacube Instance</li>
<li>Need a way to</li>
</ul>
<div class="figure">
<p><object type="image/svg+xml" data="proposed_classes.svg" class="org-svg">
Sorry, your browser does not support SVG.</object>
</p>
</div>
</div>
</div>
<div id="outline-container-orgb70fda0" class="outline-3">
<h3 id="orgb70fda0"><span class="section-number-3">4.3</span> Storage write</h3>
<div class="outline-text-3" id="text-4-3">
<ul class="org-ul">
<li>Multiple can be available, but we don't need automatic selection. User can
specify using either a configuration file or command line arguments.</li>
</ul>
<p>
<b>*</b>
</p>
</div>
</div>
</div>
<div id="outline-container-orga50e33b" class="outline-2">
<h2 id="orga50e33b"><span class="section-number-2">5</span> Proposed architecture</h2>
</div>
<div id="outline-container-org4216c9d" class="outline-2">
<h2 id="org4216c9d"><span class="section-number-2">6</span> Driver resolution for querying for data</h2>
</div>
<div id="outline-container-org9cb81e4" class="outline-2">
<h2 id="org9cb81e4"><span class="section-number-2">7</span> Driver resolution for loading data</h2>
<div class="outline-text-2" id="text-7">
<p>
dea-clean.e1607197
</p>
</div>
</div>
</div>
<div id="postamble" class="status">
<p class="author">Author: Damien Ayers</p>
<p class="date">Created: 2017-12-08 Fri 11:59</p>
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate</a></p>
</div>
</body>
</html>
Display the source blob
Display the rendered blob
Raw
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Display the source blob
Display the rendered blob
Raw
<?xml version="1.0" encoding="UTF-8" standalone="no"?><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" contentScriptType="application/ecmascript" contentStyleType="text/css" height="339px" preserveAspectRatio="none" style="width:685px;height:339px;" version="1.1" viewBox="0 0 685 339" width="685px" zoomAndPan="magnify"><defs><filter height="300%" id="f16xjsr8g1fs01" width="300%" x="-1" y="-1"><feGaussianBlur result="blurOut" stdDeviation="2.0"/><feColorMatrix in="blurOut" result="blurOut2" type="matrix" values="0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 .4 0"/><feOffset dx="4.0" dy="4.0" in="blurOut2" result="blurOut3"/><feBlend in="SourceGraphic" in2="blurOut3" mode="normal"/></filter></defs><g><text fill="#000000" font-family="sans-serif" font-size="18" lengthAdjust="spacingAndGlyphs" textLength="433" x="130" y="17.4023">Proposed Class Structure for Storage Read Drivers</text><!--class DriverManager--><rect fill="#FEFECE" filter="url(#f16xjsr8g1fs01)" height="67.2207" id="DriverManager" style="stroke: #A80036; stroke-width: 1.5;" width="217" x="37.5" y="65.1992"/><ellipse cx="99.25" cy="84.332" fill="#ADD1B2" rx="11" ry="11" style="stroke: #A80036; stroke-width: 1.0;"/><path d="M102.2231,89.9751 Q101.6421,90.2739 101.0029,90.4233 Q100.3638,90.5728 99.6582,90.5728 Q97.1514,90.5728 95.8315,88.9209 Q94.5117,87.269 94.5117,84.1479 Q94.5117,81.0186 95.8315,79.3667 Q97.1514,77.7148 99.6582,77.7148 Q100.3638,77.7148 101.0112,77.8643 Q101.6587,78.0137 102.2231,78.3125 L102.2231,81.0352 Q101.5923,80.4541 100.9988,80.1843 Q100.4053,79.9146 99.7744,79.9146 Q98.4297,79.9146 97.7449,80.9812 Q97.0601,82.0479 97.0601,84.1479 Q97.0601,86.2397 97.7449,87.3064 Q98.4297,88.373 99.7744,88.373 Q100.4053,88.373 100.9988,88.1033 Q101.5923,87.8335 102.2231,87.2524 Z "/><text fill="#000000" font-family="sans-serif" font-size="12" font-style="italic" lengthAdjust="spacingAndGlyphs" textLength="63" x="130.75" y="81.8008">«Singleton»</text><text fill="#000000" font-family="sans-serif" font-size="12" lengthAdjust="spacingAndGlyphs" textLength="85" x="119.75" y="95.9336">DriverManager</text><line style="stroke: #A80036; stroke-width: 1.5;" x1="38.5" x2="253.5" y1="103.4648" y2="103.4648"/><line style="stroke: #A80036; stroke-width: 1.5;" x1="38.5" x2="253.5" y1="111.4648" y2="111.4648"/><text fill="#000000" font-family="sans-serif" font-size="11" lengthAdjust="spacingAndGlyphs" textLength="205" x="43.5" y="126.0996">Driver findStorageDriverFor(Loadable)</text><!--class Loadable--><rect fill="#FEFECE" filter="url(#f16xjsr8g1fs01)" height="138.6855" id="Loadable" style="stroke: #A80036; stroke-width: 1.5;" width="385" x="289.5" y="29.1992"/><ellipse cx="451.25" cy="45.1992" fill="#B4A7E5" rx="11" ry="11" style="stroke: #A80036; stroke-width: 1.0;"/><path d="M447.1777,40.9644 L447.1777,38.8062 L454.5571,38.8062 L454.5571,40.9644 L452.0918,40.9644 L452.0918,49.041 L454.5571,49.041 L454.5571,51.1992 L447.1777,51.1992 L447.1777,49.041 L449.6431,49.041 L449.6431,40.9644 Z "/><text fill="#000000" font-family="sans-serif" font-size="12" font-style="italic" lengthAdjust="spacingAndGlyphs" textLength="53" x="471.75" y="49.7344">Loadable</text><line style="stroke: #A80036; stroke-width: 1.5;" x1="290.5" x2="673.5" y1="61.1992" y2="61.1992"/><text fill="#000000" font-family="sans-serif" font-size="11" lengthAdjust="spacingAndGlyphs" textLength="271" x="295.5" y="75.834">All the information required to load a hunk of data,</text><text fill="#000000" font-family="sans-serif" font-size="11" lengthAdjust="spacingAndGlyphs" textLength="177" x="295.5" y="88.7891">no need for an index connection.</text><line style="stroke: #A80036; stroke-width: 1.0;" x1="290.5" x2="673.5" y1="95.1094" y2="95.1094"/><text fill="#000000" font-family="sans-serif" font-size="11" lengthAdjust="spacingAndGlyphs" textLength="373" x="295.5" y="109.7441">This is currently (awkwardly) implemented as an xarray list of sources</text><text fill="#000000" font-family="sans-serif" font-size="11" lengthAdjust="spacingAndGlyphs" textLength="363" x="295.5" y="122.6992">There are also several different implementations. GridWorkFlow Tile,</text><text fill="#000000" font-family="sans-serif" font-size="11" lengthAdjust="spacingAndGlyphs" textLength="202" x="295.5" y="135.6543">The grouped DataArray used between</text><a target="_top" xlink:actuate="onRequest" xlink:href="http://datacube-core.readthedocs.io/en/latest/dev/generate/datacube.Datacube.group_datasets.html" xlink:show="new" xlink:title="http://datacube-core.readthedocs.io/en/latest/dev/generate/datacube.Datacube.group_datasets.html" xlink:type="simple"><text fill="#0000FF" font-family="sans-serif" font-size="11" lengthAdjust="spacingAndGlyphs" text-decoration="underline" textLength="145" x="500.5" y="135.6543">Datacube.group_datasets()</text><line style="stroke: #0000FF; stroke-width: 1.0;" x1="500.5" x2="645.5" y1="137.6543" y2="137.6543"/></a><text fill="#000000" font-family="sans-serif" font-size="11" lengthAdjust="spacingAndGlyphs" textLength="20" x="648.5" y="135.6543">and</text><text fill="#000000" font-family="sans-serif" font-size="11" lengthAdjust="spacingAndGlyphs" textLength="113" x="295.5" y="148.6094">Datacube.load_data()</text><a target="_top" xlink:actuate="onRequest" xlink:href="http://dask.pydata.org" xlink:show="new" xlink:title="http://dask.pydata.org" xlink:type="simple"><text fill="#0000FF" font-family="sans-serif" font-size="11" lengthAdjust="spacingAndGlyphs" text-decoration="underline" textLength="123" x="295.5" y="161.5645">http://dask.pydata.org</text><line style="stroke: #0000FF; stroke-width: 1.0;" x1="295.5" x2="418.5" y1="163.5645" y2="163.5645"/></a><!--class StorageDriver--><rect fill="#FEFECE" filter="url(#f16xjsr8g1fs01)" height="99.8203" id="StorageDriver" style="stroke: #A80036; stroke-width: 1.5;" width="280" x="6" y="228.1992"/><ellipse cx="102.75" cy="244.1992" fill="#B4A7E5" rx="11" ry="11" style="stroke: #A80036; stroke-width: 1.0;"/><path d="M98.6777,239.9644 L98.6777,237.8062 L106.0571,237.8062 L106.0571,239.9644 L103.5918,239.9644 L103.5918,248.041 L106.0571,248.041 L106.0571,250.1992 L98.6777,250.1992 L98.6777,248.041 L101.1431,248.041 L101.1431,239.9644 Z "/><text fill="#000000" font-family="sans-serif" font-size="12" font-style="italic" lengthAdjust="spacingAndGlyphs" textLength="78" x="123.25" y="248.7344">StorageDriver</text><line style="stroke: #A80036; stroke-width: 1.5;" x1="7" x2="285" y1="260.1992" y2="260.1992"/><line style="stroke: #A80036; stroke-width: 1.5;" x1="7" x2="285" y1="268.1992" y2="268.1992"/><text fill="#000000" font-family="sans-serif" font-size="11" lengthAdjust="spacingAndGlyphs" textLength="218" x="12" y="282.834">isAvailableInCurrentEnvironment() : bool</text><text fill="#000000" font-family="sans-serif" font-size="11" lengthAdjust="spacingAndGlyphs" textLength="162" x="12" y="295.7891">isAbleToLoad(Loadable) : bool</text><text fill="#000000" font-family="sans-serif" font-size="11" lengthAdjust="spacingAndGlyphs" textLength="218" x="12" y="308.7441">loadDataFrom(Loadable) : xarray.Dataset</text><text fill="#000000" font-family="sans-serif" font-size="11" lengthAdjust="spacingAndGlyphs" textLength="268" x="12" y="321.6992">lazyLoadDataFrom(Loadable) : xarray.DaskDataset</text><!--link DriverManager to Loadable--><!--link DriverManager to StorageDriver--><!--
@startuml
title Proposed Class Structure for Storage Read Drivers
class DriverManager <<Singleton>> {
Driver findStorageDriverFor(Loadable)
}
interface Loadable {
All the information required to load a hunk of data,
no need for an index connection.
- -
This is currently (awkwardly) implemented as an xarray list of sources
There are also several different implementations. GridWorkFlow Tile,
The grouped DataArray used between [[http://datacube-core.readthedocs.io/en/latest/dev/generate/datacube.Datacube.group_datasets.html Datacube.group_datasets()]] and
Datacube.load_data()
[[http://dask.pydata.org]]
}
interface StorageDriver {
isAvailableInCurrentEnvironment() : bool
isAbleToLoad(Loadable) : bool
loadDataFrom(Loadable) : xarray.Dataset
lazyLoadDataFrom(Loadable) : xarray.DaskDataset
}
@enduml
PlantUML version 1.2017.18(Sat Oct 07 03:56:32 AEDT 2017)
(GPL source distribution)
Java Runtime: Java(TM) SE Runtime Environment
JVM: Java HotSpot(TM) 64-Bit Server VM
Java Version: 1.8.0_152-b16
Operating System: Mac OS X
OS Version: 10.13.2
Default Encoding: UTF-8
Language: en
Country: AU
--></g></svg>
Display the source blob
Display the rendered blob
Raw
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Display the source blob
Display the rendered blob
Raw
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment