REX-Ray+CSI: Once more unto the breach

README.md

REX-Ray+CSI: Once more unto the breach

This document outlines the roadmap for supporting CSI with libStorage and REX-Ray:

Release	Description
R1	Initial CSI compatibility
R2	CNCF-ready
R3	Reference CSI implementation

R1 - Initial CSI compatibility

The first release will support both CSI architectures: centralized and decentralized, as well as the hybrid model necessary to satisfy Mesos. Many thanks to @codenrhoden for providing the illustrations!

Arch	Diagram
Centralized
Decentralized
Hybrid

Currently the only analogue to the CSI centralized model is K8s where there is a CO master. Docker and Mesos both function in a decentralized model with the libStorage API controller providing optional centralization.

Work	Summary	Story Points
#1	New REX-Ray module for CSI Ident+Node+Controller Services	21
		21

R2 - CNCF-ready

The second release represents a project ready to donate to the CNCF as the default implementation of CSI. The primary feature of this release will be support for out-of-tree drivers via Go plug-ins, making it trivial to deliver CSI compatibility for a given storage platform via REX-Ray.

The diagrams for R2 do not change in any significant way from R1 and will not be repeated. While there is now support for Go plug-ins and the executor will be deprecated, the absence of distributed configuration support means a central libStorage API controller is still required in order to provide single, centralized configuration for storage platforms such as ScaleIO.

Work	Summary	Story Points
#1	Convert existing drivers into plug-ins	40
#2	Deprecate libStorage executor	21
#3	Update build process to account for plug-ins	21
		82

R3 - Reference implementation

Rounding out the roadmap is R3, the release that transforms REX-Ray from a program that embeds the libStorage API into the reference implementation of CSI. This release introduces support for distributed configuration via one or more well-known providers such as Etcd or abstraction libraries like Docker's libkv.

Arch	Diagram
Centralized
Decentralized
Hybrid	This release no longer includes the hybrid model as the reasons for such a design have been rendered moot by the introduction of distributed configuration support. Please be aware that the Decentralized model below is applicable to both Docker and Mesos.

The introduction of distributed configuration support will also sunset the libStorage API and complete the transformation of REX-Ray into a reference implementation of the CSI specification.

Work	Summary	Story Points
#1	Distributed configuration support	40
#2	Relocate CSI endpoint from REX-Ray to libStorage	20
		60

r1-c.puml

@startuml

database storage [
    Backend
    Storage
    <U+0020><U+0020><U+0020><U+0020>API
]

rectangle "k8s Master" as knode1 {
    frame "k8s" as knode1k {
        component "kube-controller-manager" as knode1kd
    }
    frame "REX-Ray" as knode1rr {
        component knode1csi [
            **CSI Controller**
            ---
            libStorage Client
        ]
        component "libS Server" as knode1lss
    }
    knode1kd <-> knode1csi: gRPC
    knode1csi <-> knode1lss: HTTP(s)\nJSON
}

rectangle "k8s Minion" as knode2 {
    frame "k8s" as knode2k {
        component "kubelet" as knode2kd
    }
    frame "REX-Ray" as knode2rr {
        component knode2csi [
            **CSI Node**
            ---
            libStorage Client
        ]
    }
    knode2kd <-> knode2csi: gRPC
    knode2csi <-> knode1lss: HTTP(s)\nJSON
}

knode1lss -> storage
knode1 -[hidden]> storage
storage -[hidden]> knode2
storage -[hidden]> knode2csi

@enduml

r1-c.svg

r1-d.puml

@startuml

database storage [
    Backend
    Storage
    <U+0020><U+0020><U+0020><U+0020>API
]

rectangle "Docker Node 1" as dnode1 {
    frame "Docker" as dnode1d {
        component "dockerd" as dnode1dd
    }
    frame "REX-Ray" as dnode1rr {
        component dnode1csi [
            **CSI Node+Controller**
            ---
            libStorage Client
        ]
        component "libS Server" as dnode1lss
    }
    dnode1dd <-> dnode1csi: gRPC
    dnode1csi <-> dnode1lss: HTTP(s)\nJSON
}

rectangle "Docker Node 2" as dnode2 {
    frame "Docker" as dnode2d {
        component "dockerd" as dnode2dd
    }
    frame "REX-Ray" as dnode2rr {
        component dnode2csi [
            **CSI Node+Controller**
            ---
            libStorage Client
        ]
        component "libS Server" as dnode2lss
    }
    dnode2dd <-> dnode2csi: gRPC
    dnode2csi <-> dnode2lss: HTTP(s)\nJSON
}

storage <-> dnode1lss
storage <-> dnode2lss

@enduml

r1-d.svg

r1-h.puml

@startuml

database storage [
    Backend
    Storage
    <U+0020><U+0020><U+0020><U+0020>API
]

rectangle "Mesos Node 1" as mnode1 {
    frame "Mesos" as mnode1m {
        component "mesosd" as mnode1md
    }
    frame "REX-Ray" as mnode1rr {
        component mnode1csi [
            **CSI Node+Controller**
            ---
            libStorage Client
        ]
        component "libS Server" as mnode1lss
    }
    mnode1md <-> mnode1csi: gRPC
    mnode1csi <-> mnode1lss: HTTP(s)\nJSON
}

rectangle "Mesos Node 2" as mnode2 {
    frame "Mesos" as mnode2m {
        component "mesosd" as mnode2md
    }
    frame "REX-Ray" as mnode2rr {
        component mnode2csi [
            **CSI Node+Controller**
            ---
            libStorage Client
        ]
    }
    mnode2md <-> mnode2csi: gRPC
    mnode2csi <-> mnode1lss: HTTP(s)\nJSON
}

mnode1lss -> storage
mnode1 -[hidden]> storage
storage -[hidden]> mnode2
storage -[hidden]> mnode2csi

@enduml

r1-h.svg

r3-c.puml

@startuml

database storage [
    Backend
    Storage
    <U+0020><U+0020><U+0020><U+0020>API
]

rectangle "k8s Master" as knode1 {
    frame "k8s" as knode1k {
        component "kube-controller-manager" as knode1kd
    }
    frame "REX-Ray" as knode1rr {
        component "CSI Controller" as knode1csi
    }
    knode1kd <-> knode1csi: gRPC
}

rectangle "k8s Minion" as knode2 {
    frame "k8s" as knode2k {
        component "kubelet" as knode2kd
    }
    frame "REX-Ray" as knode2rr {
        component "CSI Node" as knode2csi
    }
    knode2kd <-> knode2csi: gRPC
}

knode1csi -> storage
knode2csi -> storage

@enduml

r3-c.svg

<?xml version="1.0" encoding="UTF-8" standalone="no"?><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" contentScriptType="application/ecmascript" contentStyleType="text/css" height="432px" preserveAspectRatio="none" style="width:373px;height:432px;" version="1.1" viewBox="0 0 373 432" width="373px" zoomAndPan="magnify"><defs><filter height="300%" id="fulh4uh540myv" width="300%" x="-1" y="-1"><feGaussianBlur result="blurOut" stdDeviation="2.0"/><feColorMatrix in="blurOut" result="blurOut2" type="matrix" values="0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 .4 0"/><feOffset dx="4.0" dy="4.0" in="blurOut2" result="blurOut3"/><feBlend in="SourceGraphic" in2="blurOut3" mode="normal"/></filter></defs><g><!--cluster knode1--><rect fill="#FFFFFF" filter="url(#fulh4uh540myv)" height="286" style="stroke: #000000; stroke-width: 2.0;" width="230" x="14" y="16"/><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="80" x="89" y="31.5352">k8s Master</text><!--cluster knode1k--><rect fill="#FFFFFF" filter="url(#fulh4uh540myv)" height="71" style="stroke: #000000; stroke-width: 2.0;" width="214" x="22" y="43"/><path d="M58,44 L58,52.4883 L48,62.4883 L22,62.4883 " fill="none" style="stroke: #000000; stroke-width: 2.0;"/><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="26" x="25" y="57.5352">k8s</text><!--cluster knode1rr--><rect fill="#FFFFFF" filter="url(#fulh4uh540myv)" height="71" style="stroke: #000000; stroke-width: 2.0;" width="132" x="93" y="223"/><path d="M165,224 L165,232.4883 L155,242.4883 L93,242.4883 " fill="none" style="stroke: #000000; stroke-width: 2.0;"/><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="62" x="96" y="237.5352">REX-Ray</text><!--cluster knode2--><rect fill="#FFFFFF" filter="url(#fulh4uh540myv)" height="286" style="stroke: #000000; stroke-width: 2.0;" width="114" x="252" y="16"/><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="81" x="268.5" y="31.5352">k8s Minion</text><!--cluster knode2k--><rect fill="#FFFFFF" filter="url(#fulh4uh540myv)" height="71" style="stroke: #000000; stroke-width: 2.0;" width="88" x="265" y="43"/><path d="M301,44 L301,52.4883 L291,62.4883 L265,62.4883 " fill="none" style="stroke: #000000; stroke-width: 2.0;"/><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="26" x="268" y="57.5352">k8s</text><!--cluster knode2rr--><rect fill="#FFFFFF" filter="url(#fulh4uh540myv)" height="71" style="stroke: #000000; stroke-width: 2.0;" width="98" x="260" y="223"/><path d="M332,224 L332,232.4883 L322,242.4883 L260,242.4883 " fill="none" style="stroke: #000000; stroke-width: 2.0;"/><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="62" x="263" y="237.5352">REX-Ray</text><!--entity knode1kd--><rect fill="#FEFECE" filter="url(#fulh4uh540myv)" height="36.4883" style="stroke: #A80036; stroke-width: 1.5;" width="198" x="30" y="70"/><rect fill="#FEFECE" height="5" style="stroke: #A80036; stroke-width: 1.5;" width="10" x="25" y="75"/><rect fill="#FEFECE" height="5" style="stroke: #A80036; stroke-width: 1.5;" width="10" x="25" y="96.4883"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="178" x="40" y="93.5352">kube-controller-manager</text><!--entity knode1csi--><rect fill="#FEFECE" filter="url(#fulh4uh540myv)" height="36.4883" style="stroke: #A80036; stroke-width: 1.5;" width="116" x="101" y="250"/><rect fill="#FEFECE" height="5" style="stroke: #A80036; stroke-width: 1.5;" width="10" x="96" y="255"/><rect fill="#FEFECE" height="5" style="stroke: #A80036; stroke-width: 1.5;" width="10" x="96" y="276.4883"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="96" x="111" y="273.5352">CSI Controller</text><!--entity knode2kd--><rect fill="#FEFECE" filter="url(#fulh4uh540myv)" height="36.4883" style="stroke: #A80036; stroke-width: 1.5;" width="71" x="273.5" y="70"/><rect fill="#FEFECE" height="5" style="stroke: #A80036; stroke-width: 1.5;" width="10" x="268.5" y="75"/><rect fill="#FEFECE" height="5" style="stroke: #A80036; stroke-width: 1.5;" width="10" x="268.5" y="96.4883"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="51" x="283.5" y="93.5352">kubelet</text><!--entity knode2csi--><rect fill="#FEFECE" filter="url(#fulh4uh540myv)" height="36.4883" style="stroke: #A80036; stroke-width: 1.5;" width="82" x="268" y="250"/><rect fill="#FEFECE" height="5" style="stroke: #A80036; stroke-width: 1.5;" width="10" x="263" y="255"/><rect fill="#FEFECE" height="5" style="stroke: #A80036; stroke-width: 1.5;" width="10" x="263" y="276.4883"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="62" x="278" y="273.5352">CSI Node</text><!--entity storage--><path d="M199.5,357 C199.5,347 238,347 238,347 C238,347 276.5,347 276.5,357 L276.5,415.4648 C276.5,425.4648 238,425.4648 238,425.4648 C238,425.4648 199.5,425.4648 199.5,415.4648 L199.5,357 " fill="#FEFECE" filter="url(#fulh4uh540myv)" style="stroke: #000000; stroke-width: 1.5;"/><path d="M199.5,357 C199.5,367 238,367 238,367 C238,367 276.5,367 276.5,357 " fill="none" style="stroke: #000000; stroke-width: 1.5;"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="57" x="209.5" y="384.5352">Backend</text><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="53" x="209.5" y="401.0234">Storage</text><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="22" x="225.5" y="417.5117">API</text><!--link knode1kd to knode1csi--><path d="M132.895,111.3702 C138.6512,145.907 149.3666,210.1999 155.1163,244.6976 " fill="none" id="knode1kd-knode1csi" style="stroke: #A80036; stroke-width: 1.0;"/><polygon fill="#A80036" points="155.9949,249.9694,158.4609,240.4343,155.1729,245.0374,150.5697,241.7495,155.9949,249.9694" style="stroke: #A80036; stroke-width: 1.0;"/><polygon fill="#A80036" points="132.0417,106.2499,129.5757,115.785,132.8637,111.1819,137.4669,114.4698,132.0417,106.2499" style="stroke: #A80036; stroke-width: 1.0;"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="32" x="145" y="180.5684">gRPC</text><!--link knode2kd to knode2csi--><path d="M309,111.3702 C309,145.907 309,210.1999 309,244.6976 " fill="none" id="knode2kd-knode2csi" style="stroke: #A80036; stroke-width: 1.0;"/><polygon fill="#A80036" points="309,249.9694,313,240.9694,309,244.9694,305,240.9694,309,249.9694" style="stroke: #A80036; stroke-width: 1.0;"/><polygon fill="#A80036" points="309,106.2499,305,115.2499,309,111.2499,313,115.2499,309,106.2499" style="stroke: #A80036; stroke-width: 1.0;"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="32" x="310" y="180.5684">gRPC</text><!--link knode1csi to storage--><path d="M171.1706,286.1789 C181.1497,301.0844 195.6933,322.8078 208.6669,342.186 " fill="none" id="knode1csi-storage" style="stroke: #A80036; stroke-width: 1.0;"/><polygon fill="#A80036" points="211.5809,346.5385,209.8977,336.8345,208.7992,342.3837,203.2501,341.2852,211.5809,346.5385" style="stroke: #A80036; stroke-width: 1.0;"/><!--link knode2csi to storage--><path d="M298.0618,286.1789 C289.0933,301.0844 276.0224,322.8078 264.3627,342.186 " fill="none" id="knode2csi-storage" style="stroke: #A80036; stroke-width: 1.0;"/><polygon fill="#A80036" points="261.7438,346.5385,269.8113,340.8891,264.3216,342.2543,262.9565,336.7646,261.7438,346.5385" style="stroke: #A80036; stroke-width: 1.0;"/><!--
@startuml

database storage [
    Backend
    Storage
    <U+0020><U+0020><U+0020><U+0020>API
]

rectangle "k8s Master" as knode1 {
    frame "k8s" as knode1k {
        component "kube-controller-manager" as knode1kd
    }
    frame "REX-Ray" as knode1rr {
        component "CSI Controller" as knode1csi
    }
    knode1kd <-> knode1csi: gRPC
}

rectangle "k8s Minion" as knode2 {
    frame "k8s" as knode2k {
        component "kubelet" as knode2kd
    }
    frame "REX-Ray" as knode2rr {
        component "CSI Node" as knode2csi
    }
    knode2kd <-> knode2csi: gRPC
}

knode1csi -> storage
knode2csi -> storage

@enduml

PlantUML version 1.2017.13(Wed May 10 11:52:33 CDT 2017)
(GPL source distribution)
Java Runtime: Java(TM) SE Runtime Environment
JVM: Java HotSpot(TM) 64-Bit Server VM
Java Version: 1.8.0_20-b26
Operating System: Mac OS X
OS Version: 10.12.5
Default Encoding: UTF-8
Language: en
Country: US
--></g></svg>

r3-d.puml

@startuml

database storage [
    Backend
    Storage
    <U+0020><U+0020><U+0020><U+0020>API
]

rectangle "Docker Node 1" as dnode1 {
    frame "Docker" as dnode1d {
        component "dockerd" as dnode1dd
    }
    frame "REX-Ray" as dnode1rr {
        component "CSI Node+Controller" as dnode1csi
    }
    dnode1dd <-> dnode1csi: gRPC
}

rectangle "Docker Node 2" as dnode2 {
    frame "Docker" as dnode2d {
        component "dockerd" as dnode2dd
    }
    frame "REX-Ray" as dnode2rr {
        component "CSI Node+Controller" as dnode2csi
    }
    dnode2dd <-> dnode2csi: gRPC
}

dnode1csi -> storage
dnode2csi -> storage

@enduml

r3-d.svg