Skip to content

Instantly share code, notes, and snippets.

@omad

omad/index.html

Last active December 8, 2017 01:08
Show Gist options
  • Save omad/b00a74eb143ff93fbd2bb993df200144 to your computer and use it in GitHub Desktop.
Save omad/b00a74eb143ff93fbd2bb993df200144 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
index.html
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<!-- 2017-12-08 Fri 11:59 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Discussion points - DataCube S3 Refactor</title>
<meta name="generator" content="Org mode" />
<meta name="author" content="Damien Ayers" />
<style type="text/css">
<!--/*--><![CDATA[/*><!--*/
.title { text-align: center;
margin-bottom: .2em; }
.subtitle { text-align: center;
font-size: medium;
font-weight: bold;
margin-top:0; }
.todo { font-family: monospace; color: red; }
.done { font-family: monospace; color: green; }
.priority { font-family: monospace; color: orange; }
.tag { background-color: #eee; font-family: monospace;
padding: 2px; font-size: 80%; font-weight: normal; }
.timestamp { color: #bebebe; }
.timestamp-kwd { color: #5f9ea0; }
.org-right { margin-left: auto; margin-right: 0px; text-align: right; }
.org-left { margin-left: 0px; margin-right: auto; text-align: left; }
.org-center { margin-left: auto; margin-right: auto; text-align: center; }
.underline { text-decoration: underline; }
#postamble p, #preamble p { font-size: 90%; margin: .2em; }
p.verse { margin-left: 3%; }
pre {
border: 1px solid #ccc;
box-shadow: 3px 3px 3px #eee;
padding: 8pt;
font-family: monospace;
overflow: auto;
margin: 1.2em;
}
pre.src {
position: relative;
overflow: visible;
padding-top: 1.2em;
}
pre.src:before {
display: none;
position: absolute;
background-color: white;
top: -10px;
right: 10px;
padding: 3px;
border: 1px solid black;
}
pre.src:hover:before { display: inline;}
/* Languages per Org manual */
pre.src-asymptote:before { content: 'Asymptote'; }
pre.src-awk:before { content: 'Awk'; }
pre.src-C:before { content: 'C'; }
/* pre.src-C++ doesn't work in CSS */
pre.src-clojure:before { content: 'Clojure'; }
pre.src-css:before { content: 'CSS'; }
pre.src-D:before { content: 'D'; }
pre.src-ditaa:before { content: 'ditaa'; }
pre.src-dot:before { content: 'Graphviz'; }
pre.src-calc:before { content: 'Emacs Calc'; }
pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
pre.src-fortran:before { content: 'Fortran'; }
pre.src-gnuplot:before { content: 'gnuplot'; }
pre.src-haskell:before { content: 'Haskell'; }
pre.src-hledger:before { content: 'hledger'; }
pre.src-java:before { content: 'Java'; }
pre.src-js:before { content: 'Javascript'; }
pre.src-latex:before { content: 'LaTeX'; }
pre.src-ledger:before { content: 'Ledger'; }
pre.src-lisp:before { content: 'Lisp'; }
pre.src-lilypond:before { content: 'Lilypond'; }
pre.src-lua:before { content: 'Lua'; }
pre.src-matlab:before { content: 'MATLAB'; }
pre.src-mscgen:before { content: 'Mscgen'; }
pre.src-ocaml:before { content: 'Objective Caml'; }
pre.src-octave:before { content: 'Octave'; }
pre.src-org:before { content: 'Org mode'; }
pre.src-oz:before { content: 'OZ'; }
pre.src-plantuml:before { content: 'Plantuml'; }
pre.src-processing:before { content: 'Processing.js'; }
pre.src-python:before { content: 'Python'; }
pre.src-R:before { content: 'R'; }
pre.src-ruby:before { content: 'Ruby'; }
pre.src-sass:before { content: 'Sass'; }
pre.src-scheme:before { content: 'Scheme'; }
pre.src-screen:before { content: 'Gnu Screen'; }
pre.src-sed:before { content: 'Sed'; }
pre.src-sh:before { content: 'shell'; }
pre.src-sql:before { content: 'SQL'; }
pre.src-sqlite:before { content: 'SQLite'; }
/* additional languages in org.el's org-babel-load-languages alist */
pre.src-forth:before { content: 'Forth'; }
pre.src-io:before { content: 'IO'; }
pre.src-J:before { content: 'J'; }
pre.src-makefile:before { content: 'Makefile'; }
pre.src-maxima:before { content: 'Maxima'; }
pre.src-perl:before { content: 'Perl'; }
pre.src-picolisp:before { content: 'Pico Lisp'; }
pre.src-scala:before { content: 'Scala'; }
pre.src-shell:before { content: 'Shell Script'; }
pre.src-ebnf2ps:before { content: 'ebfn2ps'; }
/* additional language identifiers per "defun org-babel-execute"
in ob-*.el */
pre.src-cpp:before { content: 'C++'; }
pre.src-abc:before { content: 'ABC'; }
pre.src-coq:before { content: 'Coq'; }
pre.src-groovy:before { content: 'Groovy'; }
/* additional language identifiers from org-babel-shell-names in
ob-shell.el: ob-shell is the only babel language using a lambda to put
the execution function name together. */
pre.src-bash:before { content: 'bash'; }
pre.src-csh:before { content: 'csh'; }
pre.src-ash:before { content: 'ash'; }
pre.src-dash:before { content: 'dash'; }
pre.src-ksh:before { content: 'ksh'; }
pre.src-mksh:before { content: 'mksh'; }
pre.src-posh:before { content: 'posh'; }
/* Additional Emacs modes also supported by the LaTeX listings package */
pre.src-ada:before { content: 'Ada'; }
pre.src-asm:before { content: 'Assembler'; }
pre.src-caml:before { content: 'Caml'; }
pre.src-delphi:before { content: 'Delphi'; }
pre.src-html:before { content: 'HTML'; }
pre.src-idl:before { content: 'IDL'; }
pre.src-mercury:before { content: 'Mercury'; }
pre.src-metapost:before { content: 'MetaPost'; }
pre.src-modula-2:before { content: 'Modula-2'; }
pre.src-pascal:before { content: 'Pascal'; }
pre.src-ps:before { content: 'PostScript'; }
pre.src-prolog:before { content: 'Prolog'; }
pre.src-simula:before { content: 'Simula'; }
pre.src-tcl:before { content: 'tcl'; }
pre.src-tex:before { content: 'TeX'; }
pre.src-plain-tex:before { content: 'Plain TeX'; }
pre.src-verilog:before { content: 'Verilog'; }
pre.src-vhdl:before { content: 'VHDL'; }
pre.src-xml:before { content: 'XML'; }
pre.src-nxml:before { content: 'XML'; }
/* add a generic configuration mode; LaTeX export needs an additional
(add-to-list 'org-latex-listings-langs '(conf " ")) in .emacs */
pre.src-conf:before { content: 'Configuration File'; }
table { border-collapse:collapse; }
caption.t-above { caption-side: top; }
caption.t-bottom { caption-side: bottom; }
td, th { vertical-align:top; }
th.org-right { text-align: center; }
th.org-left { text-align: center; }
th.org-center { text-align: center; }
td.org-right { text-align: right; }
td.org-left { text-align: left; }
td.org-center { text-align: center; }
dt { font-weight: bold; }
.footpara { display: inline; }
.footdef { margin-bottom: 1em; }
.figure { padding: 1em; }
.figure p { text-align: center; }
.inlinetask {
padding: 10px;
border: 2px solid gray;
margin: 10px;
background: #ffffcc;
}
#org-div-home-and-up
{ text-align: right; font-size: 70%; white-space: nowrap; }
textarea { overflow-x: auto; }
.linenr { font-size: smaller }
.code-highlighted { background-color: #ffff00; }
.org-info-js_info-navigation { border-style: none; }
#org-info-js_console-label
{ font-size: 10px; font-weight: bold; white-space: nowrap; }
.org-info-js_search-highlight
{ background-color: #ffff00; color: #000000; font-weight: bold; }
.org-svg { width: 90%; }
/*]]>*/-->
</style>
<script type="text/javascript">
/*
@licstart The following is the entire license notice for the
JavaScript code in this tag.
Copyright (C) 2012-2017 Free Software Foundation, Inc.
The JavaScript code in this tag is free software: you can
redistribute it and/or modify it under the terms of the GNU
General Public License (GNU GPL) as published by the Free Software
Foundation, either version 3 of the License, or (at your option)
any later version. The code is distributed WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE. See the GNU GPL for more details.
As additional permission under GNU GPL version 3 section 7, you
may distribute non-source (e.g., minimized or compacted) forms of
that code without the copy of the GNU GPL normally required by
section 4, provided you include this license notice and a URL
through which recipients can access the Corresponding Source.
@licend The above is the entire license notice
for the JavaScript code in this tag.
*/
<!--/*--><![CDATA[/*><!--*/
function CodeHighlightOn(elem, id)
{
var target = document.getElementById(id);
if(null != target) {
elem.cacheClassElem = elem.className;
elem.cacheClassTarget = target.className;
target.className = "code-highlighted";
elem.className = "code-highlighted";
}
}
function CodeHighlightOff(elem, id)
{
var target = document.getElementById(id);
if(elem.cacheClassElem)
elem.className = elem.cacheClassElem;
if(elem.cacheClassTarget)
target.className = elem.cacheClassTarget;
}
/*]]>*///-->
</script>
</head>
<body>
<div id="content">
<h1 class="title">Discussion points - DataCube S3 Refactor</h1>
<div id="table-of-contents">
<h2>Table of Contents</h2>
<div id="text-table-of-contents">
<ul>
<li><a href="#org772f239">1. Must Change</a></li>
<li><a href="#orgcfdea83">2. Use Cases</a>
<ul>
<li><a href="#org6397913">2.1. Simple</a></li>
<li><a href="#org50348e2">2.2. Compute jobs at NCI</a></li>
</ul>
</li>
<li><a href="#org4733516">3. Configuration</a></li>
<li><a href="#orga3a5ec0">4. Types of drivers/extension points</a>
<ul>
<li><a href="#org169fa1a">4.1. Database extension</a></li>
<li><a href="#org663be17">4.2. Storage read</a></li>
<li><a href="#orgb70fda0">4.3. Storage write</a></li>
</ul>
</li>
<li><a href="#orga50e33b">5. Proposed architecture</a></li>
<li><a href="#org4216c9d">6. Driver resolution for querying for data</a></li>
<li><a href="#org9cb81e4">7. Driver resolution for loading data</a></li>
</ul>
</div>
</div>
<div id="outline-container-org772f239" class="outline-2">
<h2 id="org772f239"><span class="section-number-2">1</span> Must Change</h2>
<div class="outline-text-2" id="text-1">
<ol class="org-ol">
<li>URI schema cannot be <code>s3://</code>, that is already a standard protocol, the
<code>uri_body</code> that is generated is meaningless and is ignored.</li>
<li>Change how connections to database are managed</li>
<li></li>
</ol>
</div>
</div>
<div id="outline-container-orgcfdea83" class="outline-2">
<h2 id="orgcfdea83"><span class="section-number-2">2</span> Use Cases</h2>
<div class="outline-text-2" id="text-2">
</div>
<div id="outline-container-org6397913" class="outline-3">
<h3 id="org6397913"><span class="section-number-3">2.1</span> Simple</h3>
<div class="outline-text-3" id="text-2-1">
<div class="figure">
<p><object type="image/svg+xml" data="simple_use_case.svg" class="org-svg">
Sorry, your browser does not support SVG.</object>
</p>
</div>
</div>
</div>
<div id="outline-container-org50348e2" class="outline-3">
<h3 id="org50348e2"><span class="section-number-3">2.2</span> Compute jobs at NCI</h3>
<div class="outline-text-3" id="text-2-2">
<div class="figure">
<p><object type="image/svg+xml" data="nci_compute_job.svg" class="org-svg">
Sorry, your browser does not support SVG.</object>
</p>
</div>
</div>
</div>
</div>
<div id="outline-container-org4733516" class="outline-2">
<h2 id="org4733516"><span class="section-number-2">3</span> Configuration</h2>
<div class="outline-text-2" id="text-3">
<ul class="org-ul">
<li>When the library is loaded, it will find a list of available datacubes/environments</li>
<li>An environment specifies which database to connect to, it could also specify which
database driver to use.</li>
<li></li>
</ul>
<p>
I think it only makes sense for a `Datacube`
</p>
</div>
</div>
<div id="outline-container-orga3a5ec0" class="outline-2">
<h2 id="orga3a5ec0"><span class="section-number-2">4</span> Types of drivers/extension points</h2>
<div class="outline-text-2" id="text-4">
</div>
<div id="outline-container-org169fa1a" class="outline-3">
<h3 id="org169fa1a"><span class="section-number-3">4.1</span> Database extension</h3>
<div class="outline-text-3" id="text-4-1">
<ul class="org-ul">
<li>Options
<ul class="org-ul">
<li></li>
<li>Only one can exist for a single instance of DataCube</li>
</ul></li>
<li></li>
</ul>
<div class="figure">
<p><object type="image/svg+xml" data="proposed_database_extension_classes.svg" class="org-svg">
Sorry, your browser does not support SVG.</object>
</p>
</div>
</div>
</div>
<div id="outline-container-org663be17" class="outline-3">
<h3 id="org663be17"><span class="section-number-3">4.2</span> Storage read</h3>
<div class="outline-text-3" id="text-4-2">
<ul class="org-ul">
<li>Multiple should be available in the same Datacube Instance</li>
<li>Need a way to</li>
</ul>
<div class="figure">
<p><object type="image/svg+xml" data="proposed_classes.svg" class="org-svg">
Sorry, your browser does not support SVG.</object>
</p>
</div>
</div>
</div>
<div id="outline-container-orgb70fda0" class="outline-3">
<h3 id="orgb70fda0"><span class="section-number-3">4.3</span> Storage write</h3>
<div class="outline-text-3" id="text-4-3">
<ul class="org-ul">
<li>Multiple can be available, but we don't need automatic selection. User can
specify using either a configuration file or command line arguments.</li>
</ul>
<p>
<b>*</b>
</p>
</div>
</div>
</div>
<div id="outline-container-orga50e33b" class="outline-2">
<h2 id="orga50e33b"><span class="section-number-2">5</span> Proposed architecture</h2>
</div>
<div id="outline-container-org4216c9d" class="outline-2">
<h2 id="org4216c9d"><span class="section-number-2">6</span> Driver resolution for querying for data</h2>
</div>
<div id="outline-container-org9cb81e4" class="outline-2">
<h2 id="org9cb81e4"><span class="section-number-2">7</span> Driver resolution for loading data</h2>
<div class="outline-text-2" id="text-7">
<p>
dea-clean.e1607197
</p>
</div>
</div>
</div>
<div id="postamble" class="status">
<p class="author">Author: Damien Ayers</p>
<p class="date">Created: 2017-12-08 Fri 11:59</p>
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate</a></p>
</div>
</body>
</html>
Display the source blob
Display the rendered blob
Raw
nci_compute_job.svg
<?xml version="1.0" encoding="UTF-8" standalone="no"?><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" contentScriptType="application/ecmascript" contentStyleType="text/css" height="630px" preserveAspectRatio="none" style="width:679px;height:630px;" version="1.1" viewBox="0 0 679 630" width="679px" zoomAndPan="magnify"><defs><filter height="300%" id="fahac0md765gz" width="300%" x="-1" y="-1"><feGaussianBlur result="blurOut" stdDeviation="2.0"/><feColorMatrix in="blurOut" result="blurOut2" type="matrix" values="0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 .4 0"/><feOffset dx="4.0" dy="4.0" in="blurOut2" result="blurOut3"/><feBlend in="SourceGraphic" in2="blurOut3" mode="normal"/></filter></defs><g><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="125" x="272.5" y="23.5352">NCI Compute Job</text><line style="stroke: #A80036; stroke-width: 1.0; stroke-dasharray: 5.0,5.0;" x1="26" x2="26" y1="116.9766" y2="542.7031"/><line style="stroke: #A80036; stroke-width: 1.0; stroke-dasharray: 5.0,5.0;" x1="285.5" x2="285.5" y1="116.9766" y2="542.7031"/><line style="stroke: #A80036; stroke-width: 1.0; stroke-dasharray: 5.0,5.0;" x1="484" x2="484" y1="116.9766" y2="542.7031"/><line style="stroke: #A80036; stroke-width: 1.0; stroke-dasharray: 5.0,5.0;" x1="589" x2="589" y1="116.9766" y2="542.7031"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="31" x="8" y="114.0234">User</text><ellipse cx="26.5" cy="43.4883" fill="#FEFECE" filter="url(#fahac0md765gz)" rx="8" ry="8" style="stroke: #A80036; stroke-width: 2.0;"/><path d="M26.5,51.4883 L26.5,78.4883 M13.5,59.4883 L39.5,59.4883 M26.5,78.4883 L13.5,93.4883 M26.5,78.4883 L39.5,93.4883 " fill="#FEFECE" filter="url(#fahac0md765gz)" style="stroke: #A80036; stroke-width: 2.0;"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="31" x="8" y="555.2383">User</text><ellipse cx="26.5" cy="568.1914" fill="#FEFECE" filter="url(#fahac0md765gz)" rx="8" ry="8" style="stroke: #A80036; stroke-width: 2.0;"/><path d="M26.5,576.1914 L26.5,603.1914 M13.5,584.1914 L39.5,584.1914 M26.5,603.1914 L13.5,618.1914 M26.5,603.1914 L39.5,618.1914 " fill="#FEFECE" filter="url(#fahac0md765gz)" style="stroke: #A80036; stroke-width: 2.0;"/><rect fill="#FEFECE" filter="url(#fahac0md765gz)" height="30.4883" style="stroke: #A80036; stroke-width: 1.5;" width="78" x="244.5" y="81.4883"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="64" x="251.5" y="102.0234">Datacube</text><rect fill="#FEFECE" filter="url(#fahac0md765gz)" height="30.4883" style="stroke: #A80036; stroke-width: 1.5;" width="78" x="244.5" y="541.7031"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="64" x="251.5" y="562.2383">Datacube</text><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="39" x="462" y="114.0234">Index</text><path d="M466.5,64.4883 C466.5,54.4883 484.5,54.4883 484.5,54.4883 C484.5,54.4883 502.5,54.4883 502.5,64.4883 L502.5,90.4883 C502.5,100.4883 484.5,100.4883 484.5,100.4883 C484.5,100.4883 466.5,100.4883 466.5,90.4883 L466.5,64.4883 " fill="#FEFECE" filter="url(#fahac0md765gz)" style="stroke: #000000; stroke-width: 1.5;"/><path d="M466.5,64.4883 C466.5,74.4883 484.5,74.4883 484.5,74.4883 C484.5,74.4883 502.5,74.4883 502.5,64.4883 " fill="none" style="stroke: #000000; stroke-width: 1.5;"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="39" x="462" y="555.2383">Index</text><path d="M466.5,568.1914 C466.5,558.1914 484.5,558.1914 484.5,558.1914 C484.5,558.1914 502.5,558.1914 502.5,568.1914 L502.5,594.1914 C502.5,604.1914 484.5,604.1914 484.5,604.1914 C484.5,604.1914 466.5,604.1914 466.5,594.1914 L466.5,568.1914 " fill="#FEFECE" filter="url(#fahac0md765gz)" style="stroke: #000000; stroke-width: 1.5;"/><path d="M466.5,568.1914 C466.5,578.1914 484.5,578.1914 484.5,578.1914 C484.5,578.1914 502.5,578.1914 502.5,568.1914 " fill="none" style="stroke: #000000; stroke-width: 1.5;"/><rect fill="#FEFECE" filter="url(#fahac0md765gz)" height="30.4883" style="stroke: #A80036; stroke-width: 1.5;" width="137" x="521" y="77.4883"/><rect fill="#FEFECE" filter="url(#fahac0md765gz)" height="30.4883" style="stroke: #A80036; stroke-width: 1.5;" width="137" x="517" y="81.4883"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="123" x="524" y="102.0234">Disk/S3/HTTP/etc</text><rect fill="#FEFECE" filter="url(#fahac0md765gz)" height="30.4883" style="stroke: #A80036; stroke-width: 1.5;" width="137" x="521" y="541.7031"/><rect fill="#FEFECE" filter="url(#fahac0md765gz)" height="30.4883" style="stroke: #A80036; stroke-width: 1.5;" width="137" x="517" y="545.7031"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="123" x="524" y="566.2383">Disk/S3/HTTP/etc</text><rect fill="#EEEEEE" filter="url(#fahac0md765gz)" height="3" style="stroke: #EEEEEE; stroke-width: 1.0;" width="664" x="3" y="147.6318"/><line style="stroke: #000000; stroke-width: 1.0;" x1="3" x2="667" y1="147.6318" y2="147.6318"/><line style="stroke: #000000; stroke-width: 1.0;" x1="3" x2="667" y1="150.6318" y2="150.6318"/><rect fill="#EEEEEE" filter="url(#fahac0md765gz)" height="23.3105" style="stroke: #000000; stroke-width: 2.0;" width="116" x="277" y="136.9766"/><text fill="#000000" font-family="sans-serif" font-size="13" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="98" x="283" y="153.5449">On Head Node</text><polygon fill="#A80036" points="273.5,187.2871,283.5,191.2871,273.5,195.2871,277.5,191.2871" style="stroke: #A80036; stroke-width: 1.0;"/><line style="stroke: #A80036; stroke-width: 1.0;" x1="26.5" x2="279.5" y1="191.2871" y2="191.2871"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="153" x="33.5" y="186.8555">Request a region of data</text><polygon fill="#A80036" points="472.5,216.5977,482.5,220.5977,472.5,224.5977,476.5,220.5977" style="stroke: #A80036; stroke-width: 1.0;"/><line style="stroke: #A80036; stroke-width: 1.0;" x1="285.5" x2="478.5" y1="220.5977" y2="220.5977"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="175" x="292.5" y="216.166">Query for available datasets</text><polygon fill="#A80036" points="296.5,245.9082,286.5,249.9082,296.5,253.9082,292.5,249.9082" style="stroke: #A80036; stroke-width: 1.0;"/><line style="stroke: #A80036; stroke-width: 1.0; stroke-dasharray: 2.0,2.0;" x1="290.5" x2="483.5" y1="249.9082" y2="249.9082"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="94" x="302.5" y="245.4766">list of datasets</text><line style="stroke: #A80036; stroke-width: 1.0;" x1="285.5" x2="327.5" y1="279.5293" y2="279.5293"/><line style="stroke: #A80036; stroke-width: 1.0;" x1="327.5" x2="327.5" y1="279.5293" y2="292.5293"/><line style="stroke: #A80036; stroke-width: 1.0;" x1="286.5" x2="327.5" y1="292.5293" y2="292.5293"/><polygon fill="#A80036" points="296.5,288.5293,286.5,292.5293,296.5,296.5293,292.5,292.5293" style="stroke: #A80036; stroke-width: 1.0;"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="55" x="292.5" y="274.7871">turn into</text><text fill="#000000" font-family="sans-serif" font-size="13" font-style="italic" lengthAdjust="spacingAndGlyphs" textLength="71" x="351.5" y="274.7871">Loadable(s)</text><polygon fill="#A80036" points="37.5,317.5293,27.5,321.5293,37.5,325.5293,33.5,321.5293" style="stroke: #A80036; stroke-width: 1.0;"/><line style="stroke: #A80036; stroke-width: 1.0; stroke-dasharray: 2.0,2.0;" x1="31.5" x2="284.5" y1="321.5293" y2="321.5293"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="235" x="43.5" y="317.0977">Return a serialisable list of Loadables</text><rect fill="#EEEEEE" filter="url(#fahac0md765gz)" height="3" style="stroke: #EEEEEE; stroke-width: 1.0;" width="664" x="3" y="350.4951"/><line style="stroke: #000000; stroke-width: 1.0;" x1="3" x2="667" y1="350.4951" y2="350.4951"/><line style="stroke: #000000; stroke-width: 1.0;" x1="3" x2="667" y1="353.4951" y2="353.4951"/><rect fill="#EEEEEE" filter="url(#fahac0md765gz)" height="23.3105" style="stroke: #000000; stroke-width: 2.0;" width="136" x="267" y="339.8398"/><text fill="#000000" font-family="sans-serif" font-size="13" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="118" x="273" y="356.4082">On Worker Nodes</text><line style="stroke: #A80036; stroke-width: 1.0; stroke-dasharray: 2.0,2.0;" x1="26.5" x2="68.5" y1="394.4609" y2="394.4609"/><line style="stroke: #A80036; stroke-width: 1.0; stroke-dasharray: 2.0,2.0;" x1="68.5" x2="68.5" y1="394.4609" y2="407.4609"/><line style="stroke: #A80036; stroke-width: 1.0; stroke-dasharray: 2.0,2.0;" x1="27.5" x2="68.5" y1="407.4609" y2="407.4609"/><polygon fill="#A80036" points="37.5,403.4609,27.5,407.4609,37.5,411.4609,33.5,407.4609" style="stroke: #A80036; stroke-width: 1.0;"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="240" x="33.5" y="389.7188">Read Loadables from disk/somewhere</text><polygon fill="#A80036" points="273.5,432.4609,283.5,436.4609,273.5,440.4609,277.5,436.4609" style="stroke: #A80036; stroke-width: 1.0;"/><line style="stroke: #A80036; stroke-width: 1.0;" x1="26.5" x2="279.5" y1="436.4609" y2="436.4609"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="205" x="33.5" y="432.0293">Here is a Loadable, give me Data</text><polygon fill="#A80036" points="577.5,461.7715,587.5,465.7715,577.5,469.7715,581.5,465.7715" style="stroke: #A80036; stroke-width: 1.0;"/><line style="stroke: #A80036; stroke-width: 1.0;" x1="285.5" x2="583.5" y1="465.7715" y2="465.7715"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="207" x="292.5" y="461.3398">Load data from disk/s3/http/etc</text><polygon fill="#A80036" points="296.5,491.082,286.5,495.082,296.5,499.082,292.5,495.082" style="stroke: #A80036; stroke-width: 1.0;"/><line style="stroke: #A80036; stroke-width: 1.0; stroke-dasharray: 2.0,2.0;" x1="290.5" x2="588.5" y1="495.082" y2="495.082"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="157" x="302.5" y="490.6504">data loaded into memory</text><polygon fill="#A80036" points="37.5,520.3926,27.5,524.3926,37.5,528.3926,33.5,524.3926" style="stroke: #A80036; stroke-width: 1.0;"/><line style="stroke: #A80036; stroke-width: 1.0; stroke-dasharray: 2.0,2.0;" x1="31.5" x2="284.5" y1="524.3926" y2="524.3926"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="166" x="43.5" y="519.9609">Return an in memory array</text><!--
@startuml
title NCI Compute Job
== On Head Node ==
actor User
participant Datacube
database Index
collections "Disk/S3/HTTP/etc" as Storage
User -> Datacube: Request a region of data
Datacube -> Index : Query for available datasets
Index - -> Datacube : list of datasets
Datacube -> Datacube : turn into //Loadable(s)//
Datacube - -> User : Return a serialisable list of Loadables
== On Worker Nodes ==
User - -> User : Read Loadables from disk/somewhere
User -> Datacube : Here is a Loadable, give me Data
Datacube -> Storage : Load data from disk/s3/http/etc
Storage - -> Datacube : data loaded into memory
Datacube - -> User : Return an in memory array
@enduml
PlantUML version 1.2017.18(Sat Oct 07 03:56:32 AEDT 2017)
(GPL source distribution)
Java Runtime: Java(TM) SE Runtime Environment
JVM: Java HotSpot(TM) 64-Bit Server VM
Java Version: 1.8.0_152-b16
Operating System: Mac OS X
OS Version: 10.13.2
Default Encoding: UTF-8
Language: en
Country: AU
--></g></svg>
Display the source blob
Display the rendered blob
Raw
proposed_classes.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Display the source blob
Display the rendered blob
Raw
proposed_database_extension_classes.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Display the source blob
Display the rendered blob
Raw
simple_use_case.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment