REX-Ray+CSI

README.md

REX-Ray+CSI

The introduction of the Container Storage Interface (CSI) provides an exciting opportunity to the world of Container Orchestrator (CO) and storage. This document reviews how CSI adds value to REX-Ray, strengthening its position as the leading CO storage management solution.

Note: Before reading this document please take time to review the several REX-Ray/CSI architecture and component types.

The Proposal

This section illustrates the proposed solution for introducing CSI to REX-Ray and libStorage:

1. Create a new libStorage interface, EndpointProvider:

type EndpointProvider interface {
	// Close immediately closes all active net.Listeners and any
	// connections in state StateNew, StateActive, or StateIdle. For a
	// graceful shutdown, use Shutdown.
	Close() error

	// Serve accepts incoming connections on the Listener l, creating a
	// new service goroutine for each.
	Serve(l net.Listener) error

	// Shutdown gracefully shuts down the server without interrupting any
	// active connections. Shutdown works by first closing all open
	// listeners, then closing all idle connections, and then waiting
	// indefinitely for connections to return to idle and then shut down.
	// If the provided context expires before the shutdown is complete,
	// then the context's error is returned.
	Shutdown(ctx context.Context) error
}

2. Refactor the libStorage API (HTTP/REST) components into an implementation of the EndpointProvider interface.
3. Relocate the Docker Volume Driver endpoint from REX-Ray to libStorage and then refactor the endpoint provider as an implementation of the EndpointProvider interface.
4. Create CSI implementations of the EndpointProvider interface:
   • csi.node
   • csi.controller
   • csi.node+controller
5. Introduce the following configuration enhancements:
   • Use libstorage.server.endpoints.NAME.type to specify the endpoint's provider type
   • Use libstorage.server.endpoints.NAME.services to define which configured storage service(s) an endpoint will expose. Please note that if an endpoint provider is not capable of consuming more than one service the property can be defined as a string, otherwise it would be []string.

Diagrams

This section visualizes the proposed changes as two diagrams that represent a centralized architecture with Docker as the CO, without and with CSI support.

Centralized Architecture w Docker

The first diagram illustrates the centralized architecture as it is today using Docker as the CO:

Centralized Architecture w Docker & CSI

The second diagram also outlines the centralized architecture, but this time includes the model as it would appear with the changes required to support CSI described above:

Endpoint Providers

This section lists the supported endpoint provider types:

Type	Multi-Service Support	Description
libstorage	✓	The libStorage API (HTTP/REST) endpoint provider
docker		The Docker Volume Driver (HTTP/REST) endpoint provider
csi		The CSI node+controller (gRPC) endpoint provider
csi.node		The CSI node (gRPC) endpoint provider
csi.controller		The CSI controller (gRPC) endpoint provider

Estimates

This section estimates the story points for the issues above:

Issue	Summary	Story Points
#1	EndpointProvider interface	8
#2	Refactor libStorage w EndpointProvider interface	21
#3	Relocate/Refactor Docker Vol Driver	13
#4	Create CSI endpoint providers	21
#4	Enhance libStorage configuration	13
		76

docker-rr-cz.puml

@startuml
frame "Master Host" as masterHost {
    frame "REX-Ray\nController" as masterControllerProc {
        rectangle  "libStorage\nPackage" as masterLibstoragePkg {
            rectangle "libStorage\nController" as masterControllerLibStorageEndpoint
        }
        rectangle masterController [
            **libStorage Package**
            ==
            libStorage Controller
        ]
    }
}
frame "Node Host(s)" as nodeHost {
    rectangle "Docker" as nodeClient
    frame "REX-Ray\nAgent" as nodeAgentProc {
        rectangle nodeAgentEndpoint [
            **REX-Ray Package**
            ==
            Docker Volume
            Driver Endpoint
        ]
        rectangle masterControllerClient [
            **libStorage Package**
            ==
            libStorage API
            Client
        ]
    }
    nodeClient -> nodeAgentEndpoint: "Docker Volume\nDriver API (HTTP/REST)"
    nodeAgentEndpoint -> masterControllerClient
    masterHost -[hidden]d-> nodeHost
}
masterControllerClient -u-> masterControllerLibStorageEndpoint: "libStorage API\n(HTTP/REST)"
@enduml

docker-rr-cz.svg

docker-rr.csi-cz.puml

@startuml
frame "Master Host" as masterHost {
    rectangle "Docker" as masterClient
    frame "REX-Ray\nController" as masterControllerProc {
        rectangle  "libStorage\nPackage" as masterLibstoragePkg {
            rectangle "CSI\nController\nService" as masterControllerCSIEndpoint
            rectangle "libStorage\nController" as masterControllerLibStorageEndpoint
        }
    }
    masterClient -> masterControllerCSIEndpoint: "CSI\n(gRPC)"
}
frame "Node Host(s)" as nodeHost {
    rectangle "Docker" as nodeClient
    frame "REX-Ray\nAgent" as nodeAgentProc {
        rectangle "libStorage\nPackage" as nodeLibstoragePkg {
            rectangle "CSI\nNode\nService" as nodeAgentEndpoint
            rectangle "libStorage\nAPI\nClient" as masterControllerClient
        }
    }
    nodeClient -> nodeAgentEndpoint: "CSI\n(gRPC)"
    nodeAgentEndpoint -> masterControllerClient
}
masterControllerClient -u-> masterControllerLibStorageEndpoint: "libStorage API\n(HTTP/REST)"
@enduml

docker-rr.csi-cz.svg

<?xml version="1.0" encoding="UTF-8" standalone="no"?><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" contentScriptType="application/ecmascript" contentStyleType="text/css" height="633px" preserveAspectRatio="none" style="width:1058px;height:633px;" version="1.1" viewBox="0 0 1058 633" width="1058px" zoomAndPan="magnify"><defs><filter height="300%" id="finivoepnffck" width="300%" x="-1" y="-1"><feGaussianBlur result="blurOut" stdDeviation="2.0"/><feColorMatrix in="blurOut" result="blurOut2" type="matrix" values="0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 .4 0"/><feOffset dx="4.0" dy="4.0" in="blurOut2" result="blurOut3"/><feBlend in="SourceGraphic" in2="blurOut3" mode="normal"/></filter></defs><g><!--cluster masterHost--><rect fill="#FFFFFF" filter="url(#finivoepnffck)" height="383" style="stroke: #000000; stroke-width: 2.0;" width="263" x="14" y="16"/><path d="M112,17 L112,25.4883 L102,35.4883 L14,35.4883 " fill="none" style="stroke: #000000; stroke-width: 2.0;"/><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="88" x="17" y="30.5352">Master Host</text><!--cluster masterControllerProc--><rect fill="#FFFFFF" filter="url(#finivoepnffck)" height="171" style="stroke: #000000; stroke-width: 2.0;" width="247" x="22" y="220"/><path d="M105,221 L105,245.9766 L95,255.9766 L22,255.9766 " fill="none" style="stroke: #000000; stroke-width: 2.0;"/><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="62" x="30.5" y="234.5352">REX-Ray</text><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="73" x="25" y="251.0234">Controller</text><!--cluster masterLibstoragePkg--><rect fill="#FFFFFF" filter="url(#finivoepnffck)" height="120" style="stroke: #000000; stroke-width: 2.0;" width="231" x="30" y="263"/><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="73" x="109" y="278.5352">libStorage</text><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="58" x="116.5" y="295.0234">Package</text><!--cluster nodeHost--><rect fill="#FFFFFF" filter="url(#finivoepnffck)" height="508" style="stroke: #000000; stroke-width: 2.0;" width="766" x="285" y="118"/><path d="M389,119 L389,127.4883 L379,137.4883 L285,137.4883 " fill="none" style="stroke: #000000; stroke-width: 2.0;"/><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="94" x="288" y="132.5352">Node Host(s)</text><!--cluster nodeAgentProc--><rect fill="#FFFFFF" filter="url(#finivoepnffck)" height="171" style="stroke: #000000; stroke-width: 2.0;" width="750" x="293" y="447"/><path d="M365,448 L365,472.9766 L355,482.9766 L293,482.9766 " fill="none" style="stroke: #000000; stroke-width: 2.0;"/><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="62" x="296" y="461.5352">REX-Ray</text><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="42" x="306" y="478.0234">Agent</text><!--cluster nodeLibstoragePkg--><rect fill="#FFFFFF" filter="url(#finivoepnffck)" height="120" style="stroke: #000000; stroke-width: 2.0;" width="734" x="301" y="490"/><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="73" x="631.5" y="505.5352">libStorage</text><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="58" x="639" y="522.0234">Package</text><!--entity masterClient--><rect fill="#FEFECE" filter="url(#finivoepnffck)" height="36.4883" style="stroke: #000000; stroke-width: 1.5;" width="68" x="174" y="43"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="48" x="184" y="66.5352">Docker</text><!--entity masterControllerCSIEndpoint--><rect fill="#FEFECE" filter="url(#finivoepnffck)" height="69.4648" style="stroke: #000000; stroke-width: 1.5;" width="90" x="163" y="306"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="22" x="173" y="329.5352">CSI</text><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="70" x="173" y="346.0234">Controller</text><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="48" x="173" y="362.5117">Service</text><!--entity masterControllerLibStorageEndpoint--><rect fill="#FEFECE" filter="url(#finivoepnffck)" height="52.9766" style="stroke: #000000; stroke-width: 1.5;" width="90" x="38" y="314"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="70" x="48" y="337.5352">libStorage</text><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="70" x="48" y="354.0234">Controller</text><!--entity nodeClient--><rect fill="#FEFECE" filter="url(#finivoepnffck)" height="36.4883" style="stroke: #000000; stroke-width: 1.5;" width="68" x="959" y="145"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="48" x="969" y="168.5352">Docker</text><!--entity nodeAgentEndpoint--><rect fill="#FEFECE" filter="url(#finivoepnffck)" height="69.4648" style="stroke: #000000; stroke-width: 1.5;" width="68" x="959" y="533"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="22" x="969" y="556.5352">CSI</text><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="36" x="969" y="573.0234">Node</text><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="48" x="969" y="589.5117">Service</text><!--entity masterControllerClient--><rect fill="#FEFECE" filter="url(#finivoepnffck)" height="69.4648" style="stroke: #000000; stroke-width: 1.5;" width="90" x="309" y="533"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="70" x="319" y="556.5352">libStorage</text><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="22" x="319" y="573.0234">API</text><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="40" x="319" y="589.5117">Client</text><!--link masterClient to masterControllerCSIEndpoint--><path d="M208,79.3388 C208,123.9907 208,239.1842 208,300.5831 " fill="none" id="masterClient-masterControllerCSIEndpoint" style="stroke: #A80036; stroke-width: 1.0;"/><polygon fill="#A80036" points="208,305.631,212,296.631,208,300.631,204,296.631,208,305.631" style="stroke: #A80036; stroke-width: 1.0;"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="20" x="219" y="160.5684">CSI</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="40" x="209" y="175.8789">(gRPC)</text><!--link nodeClient to nodeAgentEndpoint--><path d="M993,181.1663 C993,242.2999 993,441.5326 993,527.4591 " fill="none" id="nodeClient-nodeAgentEndpoint" style="stroke: #A80036; stroke-width: 1.0;"/><polygon fill="#A80036" points="993,532.7049,997,523.7049,993,527.7049,989,523.7049,993,532.7049" style="stroke: #A80036; stroke-width: 1.0;"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="20" x="1004" y="338.0684">CSI</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="40" x="994" y="353.3789">(gRPC)</text><!--link nodeAgentEndpoint to masterControllerClient--><path d="M958.7893,567.5 C774.0038,567.5 589.2183,567.5 404.4328,567.5 " fill="none" id="nodeAgentEndpoint-masterControllerClient" style="stroke: #A80036; stroke-width: 1.0;"/><polygon fill="#A80036" points="399.2417,567.5,408.2417,571.5,404.2417,567.5,408.2417,563.5,399.2417,567.5" style="stroke: #A80036; stroke-width: 1.0;"/><!--link masterControllerLibStorageEndpoint to masterControllerClient--><path d="M113.7338,370.7917 C134.2981,390.6921 162.2183,417.0138 188,439 C227.6552,472.8173 274.8046,508.9498 308.7023,534.2591 " fill="none" id="masterControllerLibStorageEndpoint-masterControllerClient" style="stroke: #A80036; stroke-width: 1.0;"/><polygon fill="#A80036" points="109.9817,367.1531,113.6578,376.2902,113.571,370.634,119.2272,370.5473,109.9817,367.1531" style="stroke: #A80036; stroke-width: 1.0;"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="87" x="189" y="420.5684">libStorage API</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="78" x="193.5" y="435.8789">(HTTP/REST)</text><!--
@startuml
frame "Master Host" as masterHost {
    rectangle "Docker" as masterClient
    frame "REX-Ray\nController" as masterControllerProc {
        rectangle  "libStorage\nPackage" as masterLibstoragePkg {
            rectangle "CSI\nController\nService" as masterControllerCSIEndpoint
            rectangle "libStorage\nController" as masterControllerLibStorageEndpoint
        }
    }
    masterClient -> masterControllerCSIEndpoint: "CSI\n(gRPC)"
}
frame "Node Host(s)" as nodeHost {
    rectangle "Docker" as nodeClient
    frame "REX-Ray\nAgent" as nodeAgentProc {
        rectangle "libStorage\nPackage" as nodeLibstoragePkg {
            rectangle "CSI\nNode\nService" as nodeAgentEndpoint
            rectangle "libStorage\nAPI\nClient" as masterControllerClient
        }
    }
    nodeClient -> nodeAgentEndpoint: "CSI\n(gRPC)"
    nodeAgentEndpoint -> masterControllerClient
}
masterControllerClient -u-> masterControllerLibStorageEndpoint: "libStorage API\n(HTTP/REST)"
@enduml

PlantUML version 1.2017.13(Wed May 10 11:52:33 CDT 2017)
(GPL source distribution)
Java Runtime: Java(TM) SE Runtime Environment
JVM: Java HotSpot(TM) 64-Bit Server VM
Java Version: 1.8.0_20-b26
Operating System: Mac OS X
OS Version: 10.12.5
Default Encoding: UTF-8
Language: en
Country: US
--></g></svg>