data-sources-design.md

DataSource Proposal

Background

This document proposes the extension of the DataSources field for PVC creation in Kubernetes. Snapshots as DataSources have already been added to Kubernetes, enabling a user to create a new PVC based off of a specified Snapshot. There are other common DataSources that can be used for PVC creation, including Cloning and Populators.

DataSources in general provides a clean abstraction to expose these other storage device related features (Cloning) and Services that retrieve data from remote sources (Populators).

Terminology

• Clone - a duplicated volume created by the same storage technology as the original
• Snapshot - a point in time copy of the data the resides on a PVC, often used as a backup; the Snapshot is NOT a PVC but is it's own unique object.
• Populator - an application used to write data to a PVC, this includes sources like tar files which can be transferred to and expanded on a PVC during the creation process.

Goals

• Provide a descriptive API to pre-populate PVCs with data during the provisioning process, via various sources (Clone, Snapshot, Populators). The result being an independent unique PVC that includes the desired/expected data already available on the PVC.
• Enable common features provided by storage devices to provide better performance, automation and disaster recovery

Non Goals

• This proposal does NOT suggest implementing DataSources (Cloning, Snapshot-Volumes or Populators) in Kubernetes. The goal is instead to just expose these features in the API so that they can be used by Kubernetes users if they're available.

Value add to Kubernetes

Clones, Snapshots and Populators are commonly used in storage to automate the process of providing specific/expected data on a Volume. Most backend devices provide optimizations for things like Clones and Snapshots, and recommend their usage as best practices. Adding the ability to use these features in Kubernetes provides Kubernetes users with the features they're already using and expecting.

Use Cases

Specific use cases around cloning are included in the use cases listed below as follows:

• A user wishes to create a volume pre-seeded with a known data set or data base initialization
• A user wishes to create a duplicate environment for debug purposes
• A user wishes to use Clones or Snapshots as part of their DR strategy

Design Overview

The propsed design is to expand the DataSource Field that already exists in the PVC object. Currently, the only accepted use of DataSource is to specify a Snapshot to create a PVC from. This design would extend the DataSource field to accept:

• Other PVCs in the user namespace (Clones) for SC's that support it
• Populators which would take the form of an external controller or CRD

It's important to note that the process of Cloning, Creating from Snapshot or Populating is NOT implemented in Kubernetes but is a capability of the Storage Device, or provided via an external controller.

API

Clone

The following example shows what a request to create a Clone of an existing PVC would look like. This assumes that there is only one Storage Class on the system and that the Storage Class supports cloning (CSI Capabilities):

** Request:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-2
  Namespace: myns
spec:
  capacity:
    storage: 10Gi
  dataSource:
    kind: PersistentVolumeClaim
    name: pvc-1

** Result:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-2
  Namespace: myns
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

Where pvc-2 is a clone ov pvc-1 and is a new independent object, with it's own provisioned backend volume.

Populator

The following example shows what a request to create a PVC from a populator would look like. This assumes that the specified populator has been deployed on the Cluster.

** Request:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-2
  Namespace: myns
spec:
  capacity:
    storage: 10Gi
  dataSource:
    APIGroup: populator.acme.io
    kind: HTTPSource
    name: myHTTPPopulator
kind: HTTPSource
metadata: 
  name: my-data 
  url: https://github.com/golang/go/archive/go1.11.2.tar.gz
  secretRef: "" #optional

** Result:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-2
  Namespace: myns
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

For the clone example, should spec.dataSource.kind be PersistentVolumeClaim?

For the populator example, I wonder if something like the following makes sense and is more consistent with how VolumeSnapshot is used:

apiVersion: v1
kind: PersistentVolumeClaim                                                                                                                                                   
metadata:                                                                                                                                                        
 name: pvc-2                                                                                                                                                    
 namespace: myns                                                                                                                                                
spec:                                                                                                                                                            
 capacity:                                                                                                                                                      
   storage: 10Gi                                                                                                                                                
 dataSource:                                                                                                                                                    
   kind: URLPopulator                                                                                                                                                
   name: my-golang-archive                                                                                                                                 
   apiVersion: cdi.kubevirt.io/v1alpha1
---
apiVersion: cdi.kubevirt.io/v1alpha1
kind: URLPopulator
metadata:
 name: my-golang-archive
spec:
 url: https://github.com/golang/go/archive/go1.11.2.tar.gz

@mhenriks oops! YES it should be PVC NOT PV! Thanks! On the populator that might work, I agree being as consistent/close to snapshots as possible is the best answer. Thanks for the feedback!!

On this line 'Other PVCs in the user namespace (Clones) for SC's that support it' does SC stand for Storage Class? I am guessing yes, but it took me some time to figure out.