Skip to content

Instantly share code, notes, and snippets.

@akutz

akutz/README.md

Last active June 14, 2017 23:58
Show Gist options
  • Save akutz/59bfe230f557a3f0ffef9198429f6d64 to your computer and use it in GitHub Desktop.
Save akutz/59bfe230f557a3f0ffef9198429f6d64 to your computer and use it in GitHub Desktop.
Download ZIP
REX-Ray+CSI: Once more unto the breach
Raw
README.md

REX-Ray+CSI: Once more unto the breach

This document outlines the roadmap for supporting CSI with libStorage and REX-Ray:

Release Description
R1 Initial CSI compatibility
R2 CNCF-ready
R3 Reference CSI implementation

R1 - Initial CSI compatibility

The first release will support both CSI architectures: centralized and decentralized, as well as the hybrid model necessary to satisfy Mesos. Many thanks to @codenrhoden for providing the illustrations!

Arch Diagram
Centralized
Decentralized
Hybrid

Currently the only analogue to the CSI centralized model is K8s where there is a CO master. Docker and Mesos both function in a decentralized model with the libStorage API controller providing optional centralization.

Work Summary Story Points
#1 New REX-Ray module for CSI Ident+Node+Controller Services 21
21

R2 - CNCF-ready

The second release represents a project ready to donate to the CNCF as the default implementation of CSI. The primary feature of this release will be support for out-of-tree drivers via Go plug-ins, making it trivial to deliver CSI compatibility for a given storage platform via REX-Ray.

The diagrams for R2 do not change in any significant way from R1 and will not be repeated. While there is now support for Go plug-ins and the executor will be deprecated, the absence of distributed configuration support means a central libStorage API controller is still required in order to provide single, centralized configuration for storage platforms such as ScaleIO.

Work Summary Story Points
#1 Convert existing drivers into plug-ins 40
#2 Deprecate libStorage executor 21
#3 Update build process to account for plug-ins 21
82

R3 - Reference implementation

Rounding out the roadmap is R3, the release that transforms REX-Ray from a program that embeds the libStorage API into the reference implementation of CSI. This release introduces support for distributed configuration via one or more well-known providers such as Etcd or abstraction libraries like Docker's libkv.

Arch Diagram
Centralized
Decentralized
Hybrid This release no longer includes the hybrid model as the reasons for such a design have been rendered moot by the introduction of distributed configuration support. Please be aware that the Decentralized model below is applicable to both Docker and Mesos.

The introduction of distributed configuration support will also sunset the libStorage API and complete the transformation of REX-Ray into a reference implementation of the CSI specification.

Work Summary Story Points
#1 Distributed configuration support 40
#2 Relocate CSI endpoint from REX-Ray to libStorage 20
60
Raw
r1-c.puml
@startuml
database storage [
Backend
Storage
<U+0020><U+0020><U+0020><U+0020>API
]
rectangle "k8s Master" as knode1 {
frame "k8s" as knode1k {
component "kube-controller-manager" as knode1kd
}
frame "REX-Ray" as knode1rr {
component knode1csi [
**CSI Controller**
---
libStorage Client
]
component "libS Server" as knode1lss
}
knode1kd <-> knode1csi: gRPC
knode1csi <-> knode1lss: HTTP(s)\nJSON
}
rectangle "k8s Minion" as knode2 {
frame "k8s" as knode2k {
component "kubelet" as knode2kd
}
frame "REX-Ray" as knode2rr {
component knode2csi [
**CSI Node**
---
libStorage Client
]
}
knode2kd <-> knode2csi: gRPC
knode2csi <-> knode1lss: HTTP(s)\nJSON
}
knode1lss -> storage
knode1 -[hidden]> storage
storage -[hidden]> knode2
storage -[hidden]> knode2csi
@enduml
Display the source blob
Display the rendered blob
Raw
r1-c.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Raw
r1-d.puml
@startuml
database storage [
Backend
Storage
<U+0020><U+0020><U+0020><U+0020>API
]
rectangle "Docker Node 1" as dnode1 {
frame "Docker" as dnode1d {
component "dockerd" as dnode1dd
}
frame "REX-Ray" as dnode1rr {
component dnode1csi [
**CSI Node+Controller**
---
libStorage Client
]
component "libS Server" as dnode1lss
}
dnode1dd <-> dnode1csi: gRPC
dnode1csi <-> dnode1lss: HTTP(s)\nJSON
}
rectangle "Docker Node 2" as dnode2 {
frame "Docker" as dnode2d {
component "dockerd" as dnode2dd
}
frame "REX-Ray" as dnode2rr {
component dnode2csi [
**CSI Node+Controller**
---
libStorage Client
]
component "libS Server" as dnode2lss
}
dnode2dd <-> dnode2csi: gRPC
dnode2csi <-> dnode2lss: HTTP(s)\nJSON
}
storage <-> dnode1lss
storage <-> dnode2lss
@enduml
Display the source blob
Display the rendered blob
Raw
r1-d.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Raw
r1-h.puml
@startuml
database storage [
Backend
Storage
<U+0020><U+0020><U+0020><U+0020>API
]
rectangle "Mesos Node 1" as mnode1 {
frame "Mesos" as mnode1m {
component "mesosd" as mnode1md
}
frame "REX-Ray" as mnode1rr {
component mnode1csi [
**CSI Node+Controller**
---
libStorage Client
]
component "libS Server" as mnode1lss
}
mnode1md <-> mnode1csi: gRPC
mnode1csi <-> mnode1lss: HTTP(s)\nJSON
}
rectangle "Mesos Node 2" as mnode2 {
frame "Mesos" as mnode2m {
component "mesosd" as mnode2md
}
frame "REX-Ray" as mnode2rr {
component mnode2csi [
**CSI Node+Controller**
---
libStorage Client
]
}
mnode2md <-> mnode2csi: gRPC
mnode2csi <-> mnode1lss: HTTP(s)\nJSON
}
mnode1lss -> storage
mnode1 -[hidden]> storage
storage -[hidden]> mnode2
storage -[hidden]> mnode2csi
@enduml
Display the source blob
Display the rendered blob
Raw
r1-h.svg
<?xml version="1.0" encoding="UTF-8" standalone="no"?><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" contentScriptType="application/ecmascript" contentStyleType="text/css" height="405px" preserveAspectRatio="none" style="width:930px;height:405px;" version="1.1" viewBox="0 0 930 405" width="930px" zoomAndPan="magnify"><defs><filter height="300%" id="f18tsn972gsjbi" width="300%" x="-1" y="-1"><feGaussianBlur result="blurOut" stdDeviation="2.0"/><feColorMatrix in="blurOut" result="blurOut2" type="matrix" values="0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 .4 0"/><feOffset dx="4.0" dy="4.0" in="blurOut2" result="blurOut3"/><feBlend in="SourceGraphic" in2="blurOut3" mode="normal"/></filter></defs><g><!--cluster mnode1--><rect fill="#FFFFFF" filter="url(#f18tsn972gsjbi)" height="382" style="stroke: #000000; stroke-width: 2.0;" width="593" x="14" y="16"/><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="102" x="259.5" y="31.5352">Mesos Node 1</text><!--cluster mnode1m--><rect fill="#FFFFFF" filter="url(#f18tsn972gsjbi)" height="71" style="stroke: #000000; stroke-width: 2.0;" width="90" x="80" y="43"/><path d="M136,44 L136,52.4883 L126,62.4883 L80,62.4883 " fill="none" style="stroke: #000000; stroke-width: 2.0;"/><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="46" x="83" y="57.5352">Mesos</text><!--cluster mnode1rr--><rect fill="#FFFFFF" filter="url(#f18tsn972gsjbi)" height="106" style="stroke: #000000; stroke-width: 2.0;" width="577" x="22" y="284"/><path d="M94,285 L94,293.4883 L84,303.4883 L22,303.4883 " fill="none" style="stroke: #000000; stroke-width: 2.0;"/><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="62" x="25" y="298.5352">REX-Ray</text><!--cluster mnode2--><rect fill="#FFFFFF" filter="url(#f18tsn972gsjbi)" height="382" style="stroke: #000000; stroke-width: 2.0;" width="222" x="701" y="16"/><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="102" x="761" y="31.5352">Mesos Node 2</text><!--cluster mnode2m--><rect fill="#FFFFFF" filter="url(#f18tsn972gsjbi)" height="71" style="stroke: #000000; stroke-width: 2.0;" width="90" x="767" y="43"/><path d="M823,44 L823,52.4883 L813,62.4883 L767,62.4883 " fill="none" style="stroke: #000000; stroke-width: 2.0;"/><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="46" x="770" y="57.5352">Mesos</text><!--cluster mnode2rr--><rect fill="#FFFFFF" filter="url(#f18tsn972gsjbi)" height="106" style="stroke: #000000; stroke-width: 2.0;" width="206" x="709" y="284"/><path d="M781,285 L781,293.4883 L771,303.4883 L709,303.4883 " fill="none" style="stroke: #000000; stroke-width: 2.0;"/><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="62" x="712" y="298.5352">REX-Ray</text><!--entity mnode1md--><rect fill="#FEFECE" filter="url(#f18tsn972gsjbi)" height="36.4883" style="stroke: #A80036; stroke-width: 1.5;" width="73" x="88.5" y="70"/><rect fill="#FEFECE" height="5" style="stroke: #A80036; stroke-width: 1.5;" width="10" x="83.5" y="75"/><rect fill="#FEFECE" height="5" style="stroke: #A80036; stroke-width: 1.5;" width="10" x="83.5" y="96.4883"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="53" x="98.5" y="93.5352">mesosd</text><!--entity mnode1csi--><rect fill="#FEFECE" filter="url(#f18tsn972gsjbi)" height="70.9766" style="stroke: #A80036; stroke-width: 1.5;" width="189" x="30.5" y="311"/><rect fill="#FEFECE" height="10" style="stroke: #A80036; stroke-width: 1.5;" width="15" x="199.5" y="316"/><rect fill="#FEFECE" height="2" style="stroke: #A80036; stroke-width: 1.5;" width="4" x="197.5" y="318"/><rect fill="#FEFECE" height="2" style="stroke: #A80036; stroke-width: 1.5;" width="4" x="197.5" y="322"/><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="149" x="45.5" y="344.5352">CSI Node+Controller</text><line style="stroke: #A80036; stroke-width: 1.0;" x1="31.5" x2="218.5" y1="347.4883" y2="347.4883"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="114" x="51.5" y="365.0234">libStorage Client</text><!--entity mnode1lss--><rect fill="#FEFECE" filter="url(#f18tsn972gsjbi)" height="36.4883" style="stroke: #A80036; stroke-width: 1.5;" width="92" x="499" y="328.5"/><rect fill="#FEFECE" height="5" style="stroke: #A80036; stroke-width: 1.5;" width="10" x="494" y="333.5"/><rect fill="#FEFECE" height="5" style="stroke: #A80036; stroke-width: 1.5;" width="10" x="494" y="354.9883"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="72" x="509" y="352.0352">libS Server</text><!--entity mnode2md--><rect fill="#FEFECE" filter="url(#f18tsn972gsjbi)" height="36.4883" style="stroke: #A80036; stroke-width: 1.5;" width="73" x="775.5" y="70"/><rect fill="#FEFECE" height="5" style="stroke: #A80036; stroke-width: 1.5;" width="10" x="770.5" y="75"/><rect fill="#FEFECE" height="5" style="stroke: #A80036; stroke-width: 1.5;" width="10" x="770.5" y="96.4883"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="53" x="785.5" y="93.5352">mesosd</text><!--entity mnode2csi--><rect fill="#FEFECE" filter="url(#f18tsn972gsjbi)" height="70.9766" style="stroke: #A80036; stroke-width: 1.5;" width="189" x="717.5" y="311"/><rect fill="#FEFECE" height="10" style="stroke: #A80036; stroke-width: 1.5;" width="15" x="886.5" y="316"/><rect fill="#FEFECE" height="2" style="stroke: #A80036; stroke-width: 1.5;" width="4" x="884.5" y="318"/><rect fill="#FEFECE" height="2" style="stroke: #A80036; stroke-width: 1.5;" width="4" x="884.5" y="322"/><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="149" x="732.5" y="344.5352">CSI Node+Controller</text><line style="stroke: #A80036; stroke-width: 1.0;" x1="718.5" x2="905.5" y1="347.4883" y2="347.4883"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="114" x="738.5" y="365.0234">libStorage Client</text><!--entity storage--><path d="M615.5,177 C615.5,167 654,167 654,167 C654,167 692.5,167 692.5,177 L692.5,235.4648 C692.5,245.4648 654,245.4648 654,245.4648 C654,245.4648 615.5,245.4648 615.5,235.4648 L615.5,177 " fill="#FEFECE" filter="url(#f18tsn972gsjbi)" style="stroke: #000000; stroke-width: 1.5;"/><path d="M615.5,177 C615.5,187 654,187 654,187 C654,187 692.5,187 692.5,177 " fill="none" style="stroke: #000000; stroke-width: 1.5;"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="57" x="625.5" y="204.5352">Backend</text><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="53" x="625.5" y="221.0234">Storage</text><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="22" x="641.5" y="237.5117">API</text><!--link mnode1md to mnode1csi--><path d="M125,111.4315 C125,155.379 125,250.6763 125,305.4812 " fill="none" id="mnode1md-mnode1csi" style="stroke: #A80036; stroke-width: 1.0;"/><polygon fill="#A80036" points="125,310.6554,129,301.6554,125,305.6554,121,301.6554,125,310.6554" style="stroke: #A80036; stroke-width: 1.0;"/><polygon fill="#A80036" points="125,106.1772,121,115.1772,125,111.1772,129,115.1772,125,106.1772" style="stroke: #A80036; stroke-width: 1.0;"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="32" x="126" y="211.0684">gRPC</text><!--link mnode1csi to mnode1lss--><path d="M224.6638,346.5 C309.2787,346.5 427.2726,346.5 493.8979,346.5 " fill="none" id="mnode1csi-mnode1lss" style="stroke: #A80036; stroke-width: 1.0;"/><polygon fill="#A80036" points="498.999,346.5,489.999,342.5,493.999,346.5,489.999,350.5,498.999,346.5" style="stroke: #A80036; stroke-width: 1.0;"/><polygon fill="#A80036" points="219.5179,346.5,228.5179,350.5,224.5179,346.5,228.5179,342.5,219.5179,346.5" style="stroke: #A80036; stroke-width: 1.0;"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="48" x="335.25" y="325.0684">HTTP(s)</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="31" x="343.75" y="340.3789">JSON</text><!--link mnode2md to mnode2csi--><path d="M812,111.4315 C812,155.379 812,250.6763 812,305.4812 " fill="none" id="mnode2md-mnode2csi" style="stroke: #A80036; stroke-width: 1.0;"/><polygon fill="#A80036" points="812,310.6554,816,301.6554,812,305.6554,808,301.6554,812,310.6554" style="stroke: #A80036; stroke-width: 1.0;"/><polygon fill="#A80036" points="812,106.1772,808,115.1772,812,111.1772,816,115.1772,812,106.1772" style="stroke: #A80036; stroke-width: 1.0;"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="32" x="813" y="211.0684">gRPC</text><!--link mnode2csi to mnode1lss--><path d="M712.1963,346.5 C672.8685,346.5 629.3662,346.5 596.5955,346.5 " fill="none" id="mnode2csi-mnode1lss" style="stroke: #A80036; stroke-width: 1.0;"/><polygon fill="#A80036" points="591.203,346.5,600.203,350.5,596.203,346.5,600.203,342.5,591.203,346.5" style="stroke: #A80036; stroke-width: 1.0;"/><polygon fill="#A80036" points="717.4669,346.5,708.4669,342.5,712.4669,346.5,708.4669,350.5,717.4669,346.5" style="stroke: #A80036; stroke-width: 1.0;"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="48" x="630.25" y="325.0684">HTTP(s)</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="31" x="638.75" y="340.3789">JSON</text><!--link mnode1lss to storage--><path d="M558.985,328.4734 C574.3954,308.6096 599.7325,275.9503 620.3681,249.3512 " fill="none" id="mnode1lss-storage" style="stroke: #A80036; stroke-width: 1.0;"/><polygon fill="#A80036" points="623.6374,245.1371,614.9601,249.7959,620.5724,249.0875,621.2807,254.6999,623.6374,245.1371" style="stroke: #A80036; stroke-width: 1.0;"/><!--link mnode1 to storage--><!--link storage to mnode2--><!--link storage to mnode2csi--><!--
@startuml
database storage [
Backend
Storage
<U+0020><U+0020><U+0020><U+0020>API
]
rectangle "Mesos Node 1" as mnode1 {
frame "Mesos" as mnode1m {
component "mesosd" as mnode1md
}
frame "REX-Ray" as mnode1rr {
component mnode1csi [
**CSI Node+Controller**
- - -
libStorage Client
]
component "libS Server" as mnode1lss
}
mnode1md <-> mnode1csi: gRPC
mnode1csi <-> mnode1lss: HTTP(s)\nJSON
}
rectangle "Mesos Node 2" as mnode2 {
frame "Mesos" as mnode2m {
component "mesosd" as mnode2md
}
frame "REX-Ray" as mnode2rr {
component mnode2csi [
**CSI Node+Controller**
- - -
libStorage Client
]
}
mnode2md <-> mnode2csi: gRPC
mnode2csi <-> mnode1lss: HTTP(s)\nJSON
}
mnode1lss -> storage
mnode1 -[hidden]> storage
storage -[hidden]> mnode2
storage -[hidden]> mnode2csi
@enduml
PlantUML version 1.2017.13(Wed May 10 11:52:33 CDT 2017)
(GPL source distribution)
Java Runtime: Java(TM) SE Runtime Environment
JVM: Java HotSpot(TM) 64-Bit Server VM
Java Version: 1.8.0_20-b26
Operating System: Mac OS X
OS Version: 10.12.5
Default Encoding: UTF-8
Language: en
Country: US
--></g></svg>
Raw
r3-c.puml
@startuml
database storage [
Backend
Storage
<U+0020><U+0020><U+0020><U+0020>API
]
rectangle "k8s Master" as knode1 {
frame "k8s" as knode1k {
component "kube-controller-manager" as knode1kd
}
frame "REX-Ray" as knode1rr {
component "CSI Controller" as knode1csi
}
knode1kd <-> knode1csi: gRPC
}
rectangle "k8s Minion" as knode2 {
frame "k8s" as knode2k {
component "kubelet" as knode2kd
}
frame "REX-Ray" as knode2rr {
component "CSI Node" as knode2csi
}
knode2kd <-> knode2csi: gRPC
}
knode1csi -> storage
knode2csi -> storage
@enduml
Display the source blob
Display the rendered blob
Raw
r3-c.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Raw
r3-d.puml
@startuml
database storage [
Backend
Storage
<U+0020><U+0020><U+0020><U+0020>API
]
rectangle "Docker Node 1" as dnode1 {
frame "Docker" as dnode1d {
component "dockerd" as dnode1dd
}
frame "REX-Ray" as dnode1rr {
component "CSI Node+Controller" as dnode1csi
}
dnode1dd <-> dnode1csi: gRPC
}
rectangle "Docker Node 2" as dnode2 {
frame "Docker" as dnode2d {
component "dockerd" as dnode2dd
}
frame "REX-Ray" as dnode2rr {
component "CSI Node+Controller" as dnode2csi
}
dnode2dd <-> dnode2csi: gRPC
}
dnode1csi -> storage
dnode2csi -> storage
@enduml
Display the source blob
Display the rendered blob
Raw
r3-d.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment