Rook DataPolicy Prototype

notes.md

TODO

• Define defaults and override rules.
  • Should there be a "default" DataPolicy that is assumed unless overridden?
  • The DataPolicies, DataEndpoints, and StorageNodes lists are ordered. How can we use that ordering to make predictable results from selections?
    • Proposal for StorageNodes: The last group of selectors (all must match) to match a host is what applies.
    • Proposal for DataPolicies: The last policy to match a device or directory is what applies.
    • Proposal for DataEndpoints: Inclusive match. If any one rule describes a device or directory then it's added to the group.
• Seems like it could be possible to combine DataPolicies and DataEndpoints into one resource type.

Hopefully there are few flaws in the logic and this proposal is a good compromise between simplicity and power. This will also hopefully conform to most Kubernetes conventions and make a clear link between user configuration and data structures within Rook.

Builtin policies could be created that handle most use-cases, so they only need to be named by the user kind of like they do now with settings that correspond to CLI arguments.

policy-spec.md

StorageNode

Information needed to select storage nodes in a cluster as well as specific configuration details of data endpoints on those nodes.

Specification

Attribute Name	Type	Description
name	string	Unique identifier
nodeLabels	map	Node labels (key: value) to select Storage Nodes
nodeNames	list of strings	Names to select Storage Nodes
dataPolicyIncludes	list of strings	Names of pre-defined data policies to include
dataPolicies	list of objects	List of policies to apply to devices on matched nodes

StorageNode Examples

storageNodes:
  - name: default
    nodeLabels:
      storage-node: "true"
    dataPolicyIncludes:
      - use-all-devices # builtin or pre-defined policy
  - name: node4 # node4 gets a special policy
    nodeLabels:
      storage-node: "true"
    nodeNames:
      - node4
    dataPolicies:
      - name: ssd-only
        dataEndpoints:
          - deviceTypes:
            - ssd

DataPolicy

Rules that select data endpoints and govern how they should be configured. These show up in dataPolicies lists.

Specification

Attribute Name	Type	Description
name	string	Unique identifier
includes	list of strings	The names of pre-defined policies to include
dataEndpoints	list of objects	List of rules for matched storage directories/devices
metadataEndpoints	list of objects	List of rules for matched metadata directories/devices

DataPolicy Examples:

See StorageNode Examples

DataEndpointGroup

Rules that group one or more devices and directories and defines their expected configuration details. These are what go in dataEndpoints and metadataEndpoints lists.

Specification

Attribute Name	Type	Description
name	string	Unique identifier
deviceNames	list of strings	Matches devices by name
devicePaths	list of strings	Matches devices by full path
deviceFilters	list of strings	Matches devices by regular expression
deviceTypes	list of strings	Matches devices by type (e.g. ssd, hdd, ramdisk, loop)
paths	list of strings	Matches directories by path
pathFilters	list of strings	Matches directories by regular expression
settings	list of objects	Configuration details for matched data endpoints. Used to override or add to default settings.

DataEndpointGroup Examples

dataEndpoints:
  - name: use-all-devices
    deviceFilters:
      - ^sd.*
    settings:
      forceFormatEnabled: "true"
  - name: ssd-data-group
    deviceTypes:
      - ssd
      - ramdisk
    settings:
      location:
        root: ssd
        hostgroup: {{.nodeName}}-ssd
      store: bluestore
metadataEndpoints:
  - name: metadata-device
    devicePaths:
      - /dev/disk/by-label/rook-metadata
    settings:
      store: rocksdb
      forceFormatEnabled: "true"

DataEndpointSettings

Configuration information that can be applied to data endpoints. These are used in settings lists.

Specification

Attribute Name	Type	Description
name	string	Unique identifier
location	map	Key/Value pairs to be used in crushmap rules
store	string	The data store (e.g. bluestore, filestore) or metadata store (e.g. rocksdb, file)
forceFormatEnabled	bool	Decides whether to destroy data (format, partition, rm -rf, etc) on an endpoint when the device or directory does not match the expected configuration.

DataEndpointSettings Examples

See: DataEndpointGroup Examples