Skip to content

Instantly share code, notes, and snippets.

@akutz

akutz/README.md

Last active June 14, 2017 23:29
Show Gist options
  • Save akutz/0dcd0123dbc7e56fbf130c46a942b2f3 to your computer and use it in GitHub Desktop.
Save akutz/0dcd0123dbc7e56fbf130c46a942b2f3 to your computer and use it in GitHub Desktop.
Download ZIP
REX-Ray+CSI: Once more unto the breach
Raw
README.md

REX-Ray+CSI: Once more unto the breach

This document outlines the roadmap for supporting CSI with libStorage and REX-Ray:

Release Description
R1 Initial CSI compatibility
R2 CNCF-ready
R3 Reference CSI implementation

R1 - Initial CSI compatibility

The first release will support both CSI architectures: centralized and decentralized, as well as the hybrid model necessary to satisfy Mesos. Many thanks to @codenrhoden for providing the illustrations!

Arch Diagram
Centralized
Decentralized
Hybrid

Currently the only analogue to the CSI centralized model is K8s where there is a CO master. Docker and Mesos both function in a decentralized model with the libStorage API controller providing optional centralization.

Work Summary Story Points
#1 New REX-Ray module for CSI Ident+Node+Controller Services 21
21

R2 - CNCF-ready

The second release will represent a project ready to donate to the CNCF as the default implementation of CSI. The primary feature of this release will be support for out-of-tree drivers via Go plug-ins, making it trivial to deliver CSI compatibility for a given storage platform via REX-Ray.

Work Summary Story Points
#1 Convert existing drivers into plug-ins 40
#2 Deprecate libStorage executor 21
#3 Update build process to account for plug-ins 21
82

R3 - Reference implementation

Rounding out the roadmap is R3, the release that transforms REX-Ray from a program that embeds the libStorage API to the reference implementation of CSI. This release introduces support for distributed configuration via one or more well-known providers such as Etcd or abstraction libraries like Docker's libkv.

The introduction of distributed configuration support will sunset the libStorage API and complete the transformation of REX-Ray into a reference implementation of the CSI specification.

Work Summary Story Points
#1 Distributed configuration support 40
#2 Relocate CSI endpoint from REX-Ray to libStorage 20
60
Raw
csi+docker.puml
@startuml
database storage [
Backend
Storage
<U+0020><U+0020><U+0020><U+0020>API
]
rectangle "Docker Node 1" as dnode1 {
frame "Docker" as dnode1d {
component "dockerd" as dnode1dd
}
frame "REX-Ray" as dnode1rr {
component dnode1csi [
**CSI Node+Controller**
---
libStorage Client
]
component "libS Server" as dnode1lss
}
dnode1dd <-> dnode1csi: gRPC
dnode1csi <-> dnode1lss: HTTP(s)\nJSON
}
rectangle "Docker Node 2" as dnode2 {
frame "Docker" as dnode2d {
component "dockerd" as dnode2dd
}
frame "REX-Ray" as dnode2rr {
component dnode2csi [
**CSI Node+Controller**
---
libStorage Client
]
component "libS Server" as dnode2lss
}
dnode2dd <-> dnode2csi: gRPC
dnode2csi <-> dnode2lss: HTTP(s)\nJSON
}
storage <-> dnode1lss
storage <-> dnode2lss
@enduml
Display the source blob
Display the rendered blob
Raw
csi+docker.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Raw
csi+k8s.puml
@startuml
database storage [
Backend
Storage
<U+0020><U+0020><U+0020><U+0020>API
]
rectangle "k8s Master" as knode1 {
frame "k8s" as knode1k {
component "kube-controller-manager" as knode1kd
}
frame "REX-Ray" as knode1rr {
component knode1csi [
**CSI Controller**
---
libStorage Client
]
component "libS Server" as knode1lss
}
knode1kd <-> knode1csi: gRPC
knode1csi <-> knode1lss: HTTP(s)\nJSON
}
rectangle "k8s Minion" as knode2 {
frame "k8s" as knode2k {
component "kubelet" as knode2kd
}
frame "REX-Ray" as knode2rr {
component knode2csi [
**CSI Node**
---
libStorage Client
]
}
knode2kd <-> knode2csi: gRPC
knode2csi <-> knode1lss: HTTP(s)\nJSON
}
knode1lss -> storage
knode1 -[hidden]> storage
storage -[hidden]> knode2
storage -[hidden]> knode2csi
@enduml
Display the source blob
Display the rendered blob
Raw
csi+k8s.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Raw
csi+mesos.puml
@startuml
database storage [
Backend
Storage
<U+0020><U+0020><U+0020><U+0020>API
]
rectangle "Mesos Node 1" as mnode1 {
frame "Mesos" as mnode1m {
component "mesosd" as mnode1md
}
frame "REX-Ray" as mnode1rr {
component mnode1csi [
**CSI Node+Controller**
---
libStorage Client
]
component "libS Server" as mnode1lss
}
mnode1md <-> mnode1csi: gRPC
mnode1csi <-> mnode1lss: HTTP(s)\nJSON
}
rectangle "Mesos Node 2" as mnode2 {
frame "Mesos" as mnode2m {
component "mesosd" as mnode2md
}
frame "REX-Ray" as mnode2rr {
component mnode2csi [
**CSI Node+Controller**
---
libStorage Client
]
}
mnode2md <-> mnode2csi: gRPC
mnode2csi <-> mnode1lss: HTTP(s)\nJSON
}
mnode1lss -> storage
mnode1 -[hidden]> storage
storage -[hidden]> mnode2
storage -[hidden]> mnode2csi
@enduml
Display the source blob
Display the rendered blob
Raw
csi+mesos.svg
<?xml version="1.0" encoding="UTF-8" standalone="no"?><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" contentScriptType="application/ecmascript" contentStyleType="text/css" height="405px" preserveAspectRatio="none" style="width:930px;height:405px;" version="1.1" viewBox="0 0 930 405" width="930px" zoomAndPan="magnify"><defs><filter height="300%" id="f18tsn972gsjbi" width="300%" x="-1" y="-1"><feGaussianBlur result="blurOut" stdDeviation="2.0"/><feColorMatrix in="blurOut" result="blurOut2" type="matrix" values="0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 .4 0"/><feOffset dx="4.0" dy="4.0" in="blurOut2" result="blurOut3"/><feBlend in="SourceGraphic" in2="blurOut3" mode="normal"/></filter></defs><g><!--cluster mnode1--><rect fill="#FFFFFF" filter="url(#f18tsn972gsjbi)" height="382" style="stroke: #000000; stroke-width: 2.0;" width="593" x="14" y="16"/><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="102" x="259.5" y="31.5352">Mesos Node 1</text><!--cluster mnode1m--><rect fill="#FFFFFF" filter="url(#f18tsn972gsjbi)" height="71" style="stroke: #000000; stroke-width: 2.0;" width="90" x="80" y="43"/><path d="M136,44 L136,52.4883 L126,62.4883 L80,62.4883 " fill="none" style="stroke: #000000; stroke-width: 2.0;"/><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="46" x="83" y="57.5352">Mesos</text><!--cluster mnode1rr--><rect fill="#FFFFFF" filter="url(#f18tsn972gsjbi)" height="106" style="stroke: #000000; stroke-width: 2.0;" width="577" x="22" y="284"/><path d="M94,285 L94,293.4883 L84,303.4883 L22,303.4883 " fill="none" style="stroke: #000000; stroke-width: 2.0;"/><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="62" x="25" y="298.5352">REX-Ray</text><!--cluster mnode2--><rect fill="#FFFFFF" filter="url(#f18tsn972gsjbi)" height="382" style="stroke: #000000; stroke-width: 2.0;" width="222" x="701" y="16"/><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="102" x="761" y="31.5352">Mesos Node 2</text><!--cluster mnode2m--><rect fill="#FFFFFF" filter="url(#f18tsn972gsjbi)" height="71" style="stroke: #000000; stroke-width: 2.0;" width="90" x="767" y="43"/><path d="M823,44 L823,52.4883 L813,62.4883 L767,62.4883 " fill="none" style="stroke: #000000; stroke-width: 2.0;"/><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="46" x="770" y="57.5352">Mesos</text><!--cluster mnode2rr--><rect fill="#FFFFFF" filter="url(#f18tsn972gsjbi)" height="106" style="stroke: #000000; stroke-width: 2.0;" width="206" x="709" y="284"/><path d="M781,285 L781,293.4883 L771,303.4883 L709,303.4883 " fill="none" style="stroke: #000000; stroke-width: 2.0;"/><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="62" x="712" y="298.5352">REX-Ray</text><!--entity mnode1md--><rect fill="#FEFECE" filter="url(#f18tsn972gsjbi)" height="36.4883" style="stroke: #A80036; stroke-width: 1.5;" width="73" x="88.5" y="70"/><rect fill="#FEFECE" height="5" style="stroke: #A80036; stroke-width: 1.5;" width="10" x="83.5" y="75"/><rect fill="#FEFECE" height="5" style="stroke: #A80036; stroke-width: 1.5;" width="10" x="83.5" y="96.4883"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="53" x="98.5" y="93.5352">mesosd</text><!--entity mnode1csi--><rect fill="#FEFECE" filter="url(#f18tsn972gsjbi)" height="70.9766" style="stroke: #A80036; stroke-width: 1.5;" width="189" x="30.5" y="311"/><rect fill="#FEFECE" height="10" style="stroke: #A80036; stroke-width: 1.5;" width="15" x="199.5" y="316"/><rect fill="#FEFECE" height="2" style="stroke: #A80036; stroke-width: 1.5;" width="4" x="197.5" y="318"/><rect fill="#FEFECE" height="2" style="stroke: #A80036; stroke-width: 1.5;" width="4" x="197.5" y="322"/><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="149" x="45.5" y="344.5352">CSI Node+Controller</text><line style="stroke: #A80036; stroke-width: 1.0;" x1="31.5" x2="218.5" y1="347.4883" y2="347.4883"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="114" x="51.5" y="365.0234">libStorage Client</text><!--entity mnode1lss--><rect fill="#FEFECE" filter="url(#f18tsn972gsjbi)" height="36.4883" style="stroke: #A80036; stroke-width: 1.5;" width="92" x="499" y="328.5"/><rect fill="#FEFECE" height="5" style="stroke: #A80036; stroke-width: 1.5;" width="10" x="494" y="333.5"/><rect fill="#FEFECE" height="5" style="stroke: #A80036; stroke-width: 1.5;" width="10" x="494" y="354.9883"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="72" x="509" y="352.0352">libS Server</text><!--entity mnode2md--><rect fill="#FEFECE" filter="url(#f18tsn972gsjbi)" height="36.4883" style="stroke: #A80036; stroke-width: 1.5;" width="73" x="775.5" y="70"/><rect fill="#FEFECE" height="5" style="stroke: #A80036; stroke-width: 1.5;" width="10" x="770.5" y="75"/><rect fill="#FEFECE" height="5" style="stroke: #A80036; stroke-width: 1.5;" width="10" x="770.5" y="96.4883"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="53" x="785.5" y="93.5352">mesosd</text><!--entity mnode2csi--><rect fill="#FEFECE" filter="url(#f18tsn972gsjbi)" height="70.9766" style="stroke: #A80036; stroke-width: 1.5;" width="189" x="717.5" y="311"/><rect fill="#FEFECE" height="10" style="stroke: #A80036; stroke-width: 1.5;" width="15" x="886.5" y="316"/><rect fill="#FEFECE" height="2" style="stroke: #A80036; stroke-width: 1.5;" width="4" x="884.5" y="318"/><rect fill="#FEFECE" height="2" style="stroke: #A80036; stroke-width: 1.5;" width="4" x="884.5" y="322"/><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="149" x="732.5" y="344.5352">CSI Node+Controller</text><line style="stroke: #A80036; stroke-width: 1.0;" x1="718.5" x2="905.5" y1="347.4883" y2="347.4883"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="114" x="738.5" y="365.0234">libStorage Client</text><!--entity storage--><path d="M615.5,177 C615.5,167 654,167 654,167 C654,167 692.5,167 692.5,177 L692.5,235.4648 C692.5,245.4648 654,245.4648 654,245.4648 C654,245.4648 615.5,245.4648 615.5,235.4648 L615.5,177 " fill="#FEFECE" filter="url(#f18tsn972gsjbi)" style="stroke: #000000; stroke-width: 1.5;"/><path d="M615.5,177 C615.5,187 654,187 654,187 C654,187 692.5,187 692.5,177 " fill="none" style="stroke: #000000; stroke-width: 1.5;"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="57" x="625.5" y="204.5352">Backend</text><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="53" x="625.5" y="221.0234">Storage</text><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="22" x="641.5" y="237.5117">API</text><!--link mnode1md to mnode1csi--><path d="M125,111.4315 C125,155.379 125,250.6763 125,305.4812 " fill="none" id="mnode1md-mnode1csi" style="stroke: #A80036; stroke-width: 1.0;"/><polygon fill="#A80036" points="125,310.6554,129,301.6554,125,305.6554,121,301.6554,125,310.6554" style="stroke: #A80036; stroke-width: 1.0;"/><polygon fill="#A80036" points="125,106.1772,121,115.1772,125,111.1772,129,115.1772,125,106.1772" style="stroke: #A80036; stroke-width: 1.0;"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="32" x="126" y="211.0684">gRPC</text><!--link mnode1csi to mnode1lss--><path d="M224.6638,346.5 C309.2787,346.5 427.2726,346.5 493.8979,346.5 " fill="none" id="mnode1csi-mnode1lss" style="stroke: #A80036; stroke-width: 1.0;"/><polygon fill="#A80036" points="498.999,346.5,489.999,342.5,493.999,346.5,489.999,350.5,498.999,346.5" style="stroke: #A80036; stroke-width: 1.0;"/><polygon fill="#A80036" points="219.5179,346.5,228.5179,350.5,224.5179,346.5,228.5179,342.5,219.5179,346.5" style="stroke: #A80036; stroke-width: 1.0;"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="48" x="335.25" y="325.0684">HTTP(s)</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="31" x="343.75" y="340.3789">JSON</text><!--link mnode2md to mnode2csi--><path d="M812,111.4315 C812,155.379 812,250.6763 812,305.4812 " fill="none" id="mnode2md-mnode2csi" style="stroke: #A80036; stroke-width: 1.0;"/><polygon fill="#A80036" points="812,310.6554,816,301.6554,812,305.6554,808,301.6554,812,310.6554" style="stroke: #A80036; stroke-width: 1.0;"/><polygon fill="#A80036" points="812,106.1772,808,115.1772,812,111.1772,816,115.1772,812,106.1772" style="stroke: #A80036; stroke-width: 1.0;"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="32" x="813" y="211.0684">gRPC</text><!--link mnode2csi to mnode1lss--><path d="M712.1963,346.5 C672.8685,346.5 629.3662,346.5 596.5955,346.5 " fill="none" id="mnode2csi-mnode1lss" style="stroke: #A80036; stroke-width: 1.0;"/><polygon fill="#A80036" points="591.203,346.5,600.203,350.5,596.203,346.5,600.203,342.5,591.203,346.5" style="stroke: #A80036; stroke-width: 1.0;"/><polygon fill="#A80036" points="717.4669,346.5,708.4669,342.5,712.4669,346.5,708.4669,350.5,717.4669,346.5" style="stroke: #A80036; stroke-width: 1.0;"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="48" x="630.25" y="325.0684">HTTP(s)</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="31" x="638.75" y="340.3789">JSON</text><!--link mnode1lss to storage--><path d="M558.985,328.4734 C574.3954,308.6096 599.7325,275.9503 620.3681,249.3512 " fill="none" id="mnode1lss-storage" style="stroke: #A80036; stroke-width: 1.0;"/><polygon fill="#A80036" points="623.6374,245.1371,614.9601,249.7959,620.5724,249.0875,621.2807,254.6999,623.6374,245.1371" style="stroke: #A80036; stroke-width: 1.0;"/><!--link mnode1 to storage--><!--link storage to mnode2--><!--link storage to mnode2csi--><!--
@startuml
database storage [
Backend
Storage
<U+0020><U+0020><U+0020><U+0020>API
]
rectangle "Mesos Node 1" as mnode1 {
frame "Mesos" as mnode1m {
component "mesosd" as mnode1md
}
frame "REX-Ray" as mnode1rr {
component mnode1csi [
**CSI Node+Controller**
- - -
libStorage Client
]
component "libS Server" as mnode1lss
}
mnode1md <-> mnode1csi: gRPC
mnode1csi <-> mnode1lss: HTTP(s)\nJSON
}
rectangle "Mesos Node 2" as mnode2 {
frame "Mesos" as mnode2m {
component "mesosd" as mnode2md
}
frame "REX-Ray" as mnode2rr {
component mnode2csi [
**CSI Node+Controller**
- - -
libStorage Client
]
}
mnode2md <-> mnode2csi: gRPC
mnode2csi <-> mnode1lss: HTTP(s)\nJSON
}
mnode1lss -> storage
mnode1 -[hidden]> storage
storage -[hidden]> mnode2
storage -[hidden]> mnode2csi
@enduml
PlantUML version 1.2017.13(Wed May 10 11:52:33 CDT 2017)
(GPL source distribution)
Java Runtime: Java(TM) SE Runtime Environment
JVM: Java HotSpot(TM) 64-Bit Server VM
Java Version: 1.8.0_20-b26
Operating System: Mac OS X
OS Version: 10.12.5
Default Encoding: UTF-8
Language: en
Country: US
--></g></svg>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment