gistfile1.txt

vSphere Cloud provider and VMDK dynamic provisioning 

The release of OpenShift Container Platform 3.6 brings support for vsphere cloud provider. This provides vsphere VMDK dynamic provisioning for persistent volumes for container workloads. The storage presented to vsphere virtual machines as a VMDK has ReadWriteOnce access mode. 

VMDKs have a one to one relationship with the virtual machines they are attached to hence the RWO access. Configuring the OCP cluster for vsphere cloud provider support requires configuration changes on both the master and node configuration and requires some vsphere specific changes. The kubernetes docs (https://kubernetes.io/docs/getting-started-guides/vsphere/) to a great job of highlighting the requirements for vsphere. 

Both master and nodes need the following parameters set cloud-provider=vsphere and cloud-config=<Path of vsphere.conf file>. 

- Master configuration
```bash

/etc/origin/master/master-config.yaml

kubernetesMasterConfig:
  apiServerArguments: 
    cloud-config:
    - /etc/vsphere/vsphere.conf
    cloud-provider:
    - vsphere
  controllerArguments: 
    cloud-config:
    - /etc/vsphere/vsphere.conf
    cloud-provider:
    - vsphere
```

- Node configuration
```bash
/etc/origin/node/node-config.yaml

kubeletArguments:
  cloud-config:
  - /etc/vsphere/vsphere.conf
  cloud-provider:
  - vsphere
```
In the OCP 3.6 on vSphere reference architecture (https://access.redhat.com/documentation/en-us/reference_architectures/2017/html/deploying_and_managing_openshift_container_platform_3.6_on_vmware_vsphere/) there is some discussion and automation on applying this configuration. 

The vsphere.conf file should loosely resemble this:
```bash
$ cat /etc/vsphere/vsphere.conf 
[Global]
user = "[email protected]"
password = "vcenter_password"
server = "10.*.*.25" 
port = 443
insecure-flag = 1
datacenter = Boston
datastore = ose3-vmware-prod
working-dir = /Boston/vm/ocp36/
[Disk]
scsicontrollertype = pvscsi
```
The variables are discussed more in the Kubernetes document, but an important format to examine would be the working-dir or folder that houses the OpenShift guest machines. 

In vSphere the folder syntax will be /Datacenter/vm/foldername.  The tool govc is a GO based application for interacting with vSphere and vCenter. For more information take a look at the github page for govc - https://github.com/vmware/govmomi/tree/master/govc

First, export the vars that govc needs then query vCenter.

```bash
export GOVC_URL='vCenter IP OR FQDN'
export GOVC_USERNAME='[email protected]'
export GOVC_PASSWORD='vCenter Password'
export GOVC_INSECURE=1
$ govc ls
/Boston/vm
/Boston/network
/Boston/host
/Boston/datastore
```

- vSphere pre-requisites

- disk.enableUUID
This option is necessary so that the VMDK always presents a consistent UUID to the VM, this allows the new disk to be mounted properly. 

[IMAGE=enableUUID.png]

This option can be applied to the template that is being deployed from for consistency across all new VMs.

Additionally, govc can be used to set this as well:

for each VM in `govc ls /Boston/vm/ocp36/`;do govc vm.change -e="disk.enableUUID=1" -vm="$VM"