Skip to content

Instantly share code, notes, and snippets.

@akutz
Last active June 5, 2017 07:24
Show Gist options
  • Save akutz/70448a3c25cd4ac45b51806bc1d5c878 to your computer and use it in GitHub Desktop.
Save akutz/70448a3c25cd4ac45b51806bc1d5c878 to your computer and use it in GitHub Desktop.
REX-Ray/CSI Architecture Diagrams

REX-Ray/CSI Architecture Review

REX-Ray and the Container Storage Interface (CSI) support three architectural models: centralized, decentralized-split, and decentralized-unified. This document identifies the components of each architecture and their relationship to the three, common component types: client, agent, and controller.

Terms

This document uses the following terms:

Term Description
CO Container orchestrator (ex. Docker, K8s, Mesos). In the models below a CO is a client
agent Long-running process on the same host as the CO that processes identity and filesystem requests and may proxy other requests to controller
client Issues requests to agent and/or controller
controller Long-running process on the same host as the CO or a remote host that processes requests issue by a client or agent
master host A single host (or cluster) responsible for running the controller and sometimes a version of the CO as well
node host(s) One or more hosts running the CO and agent in the centralized model and CO, agent, and controller in the decentralized models

Notes

This section lists some notes of interest:

  • The Docker client knows nothing about the executor binary that the REX-Ray agent downloads from the controller. The executor binary is executed on the node host in order to provide information such as the node host's ID and local device information.
  • The diagrams below do not illustrate REX-Ray CLI->Agent or REX-Ray CLI->Controller relationships. For the purposes of this document the REX-Ray CLI should not be considered a client, rather the CO is the client.
  • In fact there isn't a REX-Ray CLI->Agent relationship. The REX-Ray CLI accesses the REX-Ray Controller directly as the REX-Ray Agent isn't a libStorage API endpoint but rather a Docker Volume Driver endpoint.
  • The only CLI tool that could be considered a client with regards to the diagrams below is the Docker Volume Driver CLI (dvdcli) tool as it accesses the REX-Ray Agent, just like Docker.

Analogues

The Diagrams section below attempts to organize the architectural implementations in such a way that analogues are apparent, but sometimes it is difficult to create the structure necessary to add the desired clarity.

Role Components
Agent Node Plugin, REX-Ray Agent
Client CO, Docker, dvdcli
Controller Node Controller, REX-Ray Agent

Diagrams

This section visually illustrates the relationship between client, agent, and controller for each of the architectural types and implementations:

Centralized Decentralized-Split Decentralized-Unified
Abstract
Docker w REX-Ray
dvdcli w REX-Ray
CSI
Docker w REX-Ray.CSI
@startuml
frame "Master Host" as masterHost {
rectangle "CO" as masterClient
rectangle "Controller Plugin" as masterController
}
frame "Node Host(s)" as nodeHost {
rectangle "CO" as nodeClient
rectangle "Node Plugin" as nodeAgent
}
masterHost -[hidden]- nodeHost
masterClient -> masterController : gRPC
nodeClient -> nodeAgent : gRPC
@enduml
Display the source blob
Display the rendered blob
Raw
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@startuml
frame "Node Host(s)" as nodeHost {
rectangle "CO" as nodeClient
rectangle "Controller Plugin" as nodeController
rectangle "Node Plugin" as nodeAgent
}
nodeClient --> nodeAgent: gRPC
nodeClient --> nodeController: gRPC
@enduml
Display the source blob
Display the rendered blob
Raw
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@startuml
frame "Node Host(s)" as nodeHost {
rectangle "CO" as nodeClient
rectangle "Controller/Node Plugin" as nodeControllerAgent
}
nodeClient --> nodeControllerAgent: gRPC
@enduml
Display the source blob
Display the rendered blob
Raw
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@startuml
frame "Master Host" as masterHost {
rectangle "Controller" as masterController
}
frame "Node Host(s)" as nodeHost {
rectangle "Client" as nodeClient
rectangle "Agent" as nodeAgent
nodeClient -> nodeAgent
nodeAgent -u-> masterController
}
@enduml
Display the source blob
Display the rendered blob
Raw
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@startuml
skinparam linetype ortho
frame "Node Host(s)" as nodeHost {
rectangle "Client" as nodeClient
rectangle "Agent" as nodeAgent
rectangle "Controller" as nodeController
nodeClient --> nodeAgent
nodeAgent -> nodeController
}
@enduml
Display the source blob
Display the rendered blob
Raw
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@startuml
skinparam linetype ortho
frame "Node Host(s)" as nodeHost {
rectangle "Client" as nodeClient
frame "Agent/Controller" as nodeAgentController {
rectangle "Agent" as nodeAgent
rectangle "Controller" as nodeController
}
nodeClient -> nodeAgent
nodeAgent -> nodeController
}
@enduml
Display the source blob
Display the rendered blob
Raw
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@startuml
frame "Master Host" as masterHost {
rectangle "REX-Ray Controller" as masterController
}
frame "Node Host(s)" as nodeHost {
rectangle "Docker" as nodeClient
rectangle "REX-Ray Agent" as nodeAgent
nodeClient -> nodeAgent: "Docker Volume\nDriver API (HTTP/REST)"
nodeAgent -u-> masterController: "libStorage API\n(HTTP/REST)"
}
@enduml
Display the source blob
Display the rendered blob
Raw
<?xml version="1.0" encoding="UTF-8" standalone="no"?><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" contentScriptType="application/ecmascript" contentStyleType="text/css" height="227px" preserveAspectRatio="none" style="width:423px;height:227px;" version="1.1" viewBox="0 0 423 227" width="423px" zoomAndPan="magnify"><defs><filter height="300%" id="fd8w9qdw3ugfi" width="300%" x="-1" y="-1"><feGaussianBlur result="blurOut" stdDeviation="2.0"/><feColorMatrix in="blurOut" result="blurOut2" type="matrix" values="0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 .4 0"/><feOffset dx="4.0" dy="4.0" in="blurOut2" result="blurOut3"/><feBlend in="SourceGraphic" in2="blurOut3" mode="normal"/></filter></defs><g><!--cluster masterHost--><rect fill="#FFFFFF" filter="url(#fd8w9qdw3ugfi)" height="71" style="stroke: #000000; stroke-width: 2.0;" width="168" x="248" y="16"/><path d="M346,17 L346,25.4883 L336,35.4883 L248,35.4883 " fill="none" style="stroke: #000000; stroke-width: 2.0;"/><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="88" x="251" y="30.5352">Master Host</text><!--cluster nodeHost--><rect fill="#FFFFFF" filter="url(#fd8w9qdw3ugfi)" height="71" style="stroke: #000000; stroke-width: 2.0;" width="388" x="14" y="149"/><path d="M118,150 L118,158.4883 L108,168.4883 L14,168.4883 " fill="none" style="stroke: #000000; stroke-width: 2.0;"/><text fill="#000000" font-family="sans-serif" font-size="14" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="94" x="17" y="163.5352">Node Host(s)</text><!--entity masterController--><rect fill="#FEFECE" filter="url(#fd8w9qdw3ugfi)" height="36.4883" style="stroke: #000000; stroke-width: 1.5;" width="152" x="256" y="43"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="132" x="266" y="66.5352">REX-Ray Controller</text><!--entity nodeClient--><rect fill="#FEFECE" filter="url(#fd8w9qdw3ugfi)" height="36.4883" style="stroke: #000000; stroke-width: 1.5;" width="68" x="22" y="176"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="48" x="32" y="199.5352">Docker</text><!--entity nodeAgent--><rect fill="#FEFECE" filter="url(#fd8w9qdw3ugfi)" height="36.4883" style="stroke: #000000; stroke-width: 1.5;" width="123" x="270.5" y="176"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="103" x="280.5" y="199.5352">REX-Ray Agent</text><!--link nodeClient to nodeAgent--><path d="M90.1198,194 C133.6425,194 209.8852,194 265.1597,194 " fill="none" id="nodeClient-nodeAgent" style="stroke: #A80036; stroke-width: 1.0;"/><polygon fill="#A80036" points="270.4416,194,261.4416,190,265.4416,194,261.4416,198,270.4416,194" style="stroke: #A80036; stroke-width: 1.0;"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="96" x="132.25" y="172.5684">Docker Volume</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="144" x="108.25" y="187.8789">Driver API (HTTP/REST)</text><!--link masterController to nodeAgent--><path d="M332,84.5162 C332,110.5087 332,151.8313 332,175.579 " fill="none" id="masterController-nodeAgent" style="stroke: #A80036; stroke-width: 1.0;"/><polygon fill="#A80036" points="332,79.2142,328,88.2142,332,84.2142,336,88.2142,332,79.2142" style="stroke: #A80036; stroke-width: 1.0;"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="87" x="333" y="122.5684">libStorage API</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="78" x="337.5" y="137.8789">(HTTP/REST)</text><!--
@startuml
frame "Master Host" as masterHost {
rectangle "REX-Ray Controller" as masterController
}
frame "Node Host(s)" as nodeHost {
rectangle "Docker" as nodeClient
rectangle "REX-Ray Agent" as nodeAgent
nodeClient -> nodeAgent: "Docker Volume\nDriver API (HTTP/REST)"
nodeAgent -u-> masterController: "libStorage API\n(HTTP/REST)"
}
@enduml
PlantUML version 1.2017.13(Wed May 10 11:52:33 CDT 2017)
(GPL source distribution)
Java Runtime: Java(TM) SE Runtime Environment
JVM: Java HotSpot(TM) 64-Bit Server VM
Java Version: 1.8.0_20-b26
Operating System: Mac OS X
OS Version: 10.12.5
Default Encoding: UTF-8
Language: en
Country: US
--></g></svg>
@startuml
frame "Node Host(s)" as nodeHost {
rectangle "Docker" as nodeClient
rectangle "REX-Ray Agent" as nodeAgent
rectangle "REX-Ray Controller" as nodeController
nodeClient --> nodeAgent: Docker Volume\nDriver API\n(HTTP/REST)
nodeAgent -> nodeController: libStorage API\n(HTTP/REST)
}
@enduml
Display the source blob
Display the rendered blob
Raw
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@startuml
frame "Node Host(s)" as nodeHost {
rectangle "Docker" as nodeClient
frame "Agent/Controller" as nodeAgentController {
rectangle "REX-Ray Agent" as nodeAgent
rectangle "REX-Ray Controller" as nodeController
}
nodeClient --> nodeAgent: Docker Volume\nDriver API\n(HTTP/REST)
nodeAgent -> nodeController: libStorage API\n(HTTP/REST)
}
@enduml
Display the source blob
Display the rendered blob
Raw
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@startuml
frame "Master Host" as masterHost {
rectangle "Docker" as masterClient
rectangle "REX-Ray\nController\nPlugin" as masterController
masterClient -> masterController: "CSI API (gRPC)"
}
frame "Node Host(s)" as nodeHost {
rectangle "Docker" as nodeClient
rectangle "REX-Ray\nAgent\nPlugin" as nodeAgent
nodeClient -> nodeAgent: "CSI API (gRPC)"
nodeAgent -u-> masterController: "libStorage API\n(HTTP/REST)"
}
@enduml
Display the source blob
Display the rendered blob
Raw
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@startuml
frame "Node Host(s)" as nodeHost {
rectangle "Docker" as nodeClient
rectangle "REX-Ray\nController\nPlugin" as nodeController
rectangle "REX-Ray\nAgent\nPlugin" as nodeAgent
}
nodeClient --> nodeController: " CSI API\n (gRPC)"
nodeClient --> nodeAgent: "CSI API\n (gRPC)"
nodeAgent -r-> nodeController: "libStorage API\n(HTTP/REST)"
@enduml
Display the source blob
Display the rendered blob
Raw
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@startuml
frame "Node Host(s)" as nodeHost {
rectangle "Docker" as nodeClient
frame "REX-Ray\nController/Agent\nPlugin" as nodeAgentController {
rectangle "REX-Ray\nController" as nodeController
rectangle "REX-Ray\nAgent" as nodeAgent
}
nodeClient --> nodeController: " CSI API\n (gRPC)"
nodeClient --> nodeAgent: "CSI API\n (gRPC)"
nodeAgent -r-> nodeController: "libStorage API\n(HTTP/REST)"
}
@enduml
Display the source blob
Display the rendered blob
Raw
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@startuml
frame "Master Host" as masterHost {
rectangle "REX-Ray Controller" as masterController
}
frame "Node Host(s)" as nodeHost {
rectangle "dvdcli" as nodeClient
rectangle "REX-Ray Agent" as nodeAgent
nodeClient -> nodeAgent: "Docker Volume\nDriver API (HTTP/REST)"
nodeAgent -u-> masterController: "libStorage API\n(HTTP/REST)"
}
@enduml
Display the source blob
Display the rendered blob
Raw
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@startuml
frame "Node Host(s)" as nodeHost {
rectangle "dvdcli" as nodeClient
rectangle "REX-Ray Agent" as nodeAgent
rectangle "REX-Ray Controller" as nodeController
nodeClient --> nodeAgent: Docker Volume\nDriver API\n(HTTP/REST)
nodeAgent -> nodeController: libStorage API\n(HTTP/REST)
}
@enduml
Display the source blob
Display the rendered blob
Raw
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@startuml
frame "Node Host(s)" as nodeHost {
rectangle "dvdcli" as nodeClient
frame "Agent/Controller" as nodeAgentController {
rectangle "REX-Ray Agent" as nodeAgent
rectangle "REX-Ray Controller" as nodeController
}
nodeClient --> nodeAgent: Docker Volume\nDriver API\n(HTTP/REST)
nodeAgent -> nodeController: libStorage API\n(HTTP/REST)
}
@enduml
Display the source blob
Display the rendered blob
Raw
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment