rook-on-crc.md

• download CRC from here

note that you would need a Red Hat Developer account. But there is no subscriptions needed for CRC

• CRC requires a "pull secret" taken from the download page for crc start to work

if CRC is already running use: crc delete first

• CRC doesn’t provide any additional disk so users on linux can try something following to make it work:

qemu-img create -f raw crc-extra-disk 30G
sudo virsh attach-disk crc --source /full/path/to/crc-extra-disk --target vdb --cache none

note: no restart is needed for CRC

• now use "oc" to login to CRC

oc login -u <user> -p <password> https://api.crc.testing:6443

• to see the user/passowrd use:

crc console --credentials

• after CRC is up and running, you can use the following to install Rook:

oc apply -f https://raw.githubusercontent.com/rook/rook/master/deploy/examples/crds.yaml
oc apply -f https://raw.githubusercontent.com/rook/rook/master/deploy/examples/common.yaml
oc apply -f https://raw.githubusercontent.com/rook/rook/master/deploy/examples/operator-openshift.yaml

• and the Rook resources (cluster and object store):

oc apply -f https://raw.githubusercontent.com/rook/rook/master/deploy/examples/cluster-test.yaml
oc apply -f https://raw.githubusercontent.com/rook/rook/master/deploy/examples/object-test.yaml

• use oc get pods -n rook-ceph to make sure that the: MONs, OSDs and RGW are running
• now create the bucket, the topic and the notifications:

oc apply -f https://raw.githubusercontent.com/rook/rook/master/deploy/examples/storageclass-bucket-delete.yaml
oc apply -f https://raw.githubusercontent.com/rook/rook/master/deploy/examples/object-bucket-claim-delete.yaml

Ceph Object Store resource which is created by oc apply -f https://raw.githubusercontent.com/rook/rook/release-1.8/deploy/examples/object-test.yaml only use insecure port 80. rook/rook#9565 is merged recently which enable controllers for notification and topics if the request is sent to TLS enabled RGW endpoints. This PR is still not part of the 1.8 release so we have to use master branch for test.

oc apply -f https://raw.githubusercontent.com/rook/rook/master/deploy/examples/crds.yaml
oc apply -f https://raw.githubusercontent.com/rook/rook/master/deploy/examples/common.yaml
oc apply -f https://raw.githubusercontent.com/rook/rook/master/deploy/examples/operator-openshift.yaml
oc apply -f https://raw.githubusercontent.com/rook/rook/master/deploy/examples/cluster-test.yaml

Then we use secure port for object store resource.

cat <<EOF | oc apply -f -
#################################################################################################################
# Create an object store with settings for a test environment. Only a single OSD is required in this example.
#  kubectl create -f object-test.yaml
#################################################################################################################

apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
  name: my-store
  namespace: rook-ceph # namespace:cluster
spec:
  metadataPool:
    replicated:
      size: 1
  dataPool:
    replicated:
      size: 1
  preservePoolsOnDelete: false
  gateway:
    service:
      annotations:
        service.beta.openshift.io/serving-cert-secret-name: my-store-tls 
    securePort: 443
    instances: 1
EOF

Check the pods in rook-ceph namespace to make sure all the pods are running.

$ oc get pods -n rook-ceph
NAME                                                READY   STATUS      RESTARTS   AGE
csi-cephfsplugin-provisioner-6f54f6c477-5sp9k       6/6     Running     0          29m
csi-cephfsplugin-z96pz                              3/3     Running     0          29m
csi-rbdplugin-provisioner-6d765b47d5-pkc8j          6/6     Running     0          29m
csi-rbdplugin-ssgc9                                 3/3     Running     0          29m
rook-ceph-mgr-a-5b8f9998c6-vrglx                    1/1     Running     0          27m
rook-ceph-mon-a-7445f49f8-6tfjj                     1/1     Running     0          27m
rook-ceph-operator-5df4d596d5-sfrtw                 1/1     Running     0          31m
rook-ceph-osd-0-5f46f4cb58-498w6                    1/1     Running     0          26m
rook-ceph-osd-prepare-crc-8rwmc-master-0--1-zjcgc   0/1     Completed   0          26m
rook-ceph-rgw-my-store-a-6847bcf96b-cwc9s           1/1     Running     0          17m

Object bucket Claim (OBC) in Rook

Based on the OBC configuration doc and the notification configuration doc. Let's create a storage class, and a bucket preconfigured with a notification:

$ oc apply -f https://raw.githubusercontent.com/rook/rook/master/deploy/examples/storageclass-bucket-delete.yaml
$ oc apply -f https://raw.githubusercontent.com/rook/rook/master/deploy/examples/object-bucket-claim-notification.yaml

Create bucket notifications as mentioned https://ceph.io/en/news/blog/2022/bucket-notifications-with-knative-and-rook-on-minikube2/#bucket-notifications here.

External Access

Openshift provide routes (ingress) to make available of a service.

$ oc get svc
$ oc create route passthrough --service=rook-ceph-rgw-my-store
$ oc get route
NAME                     HOST/PORT                                           PATH   SERVICES                 PORT    TERMINATION   WILDCARD
rook-ceph-rgw-my-store   rook-ceph-rgw-my-store-rook-ceph.apps-crc.testing          rook-ceph-rgw-my-store   https   passthrough   None
$ export AWS_URL=https://rook-ceph-rgw-my-store-rook-ceph.apps-crc.testing

user credentials should be fetched using https://ceph.io/en/news/blog/2022/bucket-notifications-with-knative-and-rook-on-minikube2/#user-credentials one.

Upload a file to ceph storage.

$ echo "hello world" > hello.txt
$ aws --no-verify-ssl --endpoint-url "$AWS_URL" s3 cp hello.txt s3://"$BUCKET_NAME"

you can download the aws client binary from https://docs.aws.amazon.com/cli/v1/userguide/install-linux.html