cephalocon_rook.md

1. setup kubernetes cluster in minikube for testing:

$ minikube stop && minikube delete && minikube start --force --cpus="6" -b kubeadm --kubernetes-version="v1.24.3" --driver="kvm2" --extra-config="kubelet.cgroup-driver=systemd"

2. Next we deploy Jaeger Opentelemetry Resources:

We'll use Jaeger Operator for this, uses webhooks to validate Jaeger custom resources (CRs). This requires an installed version of the cert-manager.

$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.yaml

Deploy Jaeger operator:

$  kubectl create namespace observability
$  kubectl apply -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.42.0/jaeger-operator.yaml -n observability

• For Production cluster please refer to: https://www.jaegertracing.io/docs/1.21/operator/

• We deploy simplest (agent, collector, query, ingestor, Jaeger UI) in a single pod, using in-memory storage by default

$ cat << EOF | kubectl apply -f -
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: simplest
  namespace: observability
EOF

Expose Query API:

$ cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: simplest-query-external
  namespace: observability
  labels:
    app: jaeger
    app.kubernetes.io/component: service-query
    app.kubernetes.io/instance: simplest
    app.kubernetes.io/managed-by: jaeger-operator
    app.kubernetes.io/name: simplest-query
    app.kubernetes.io/part-of: jaeger
spec:
  ports:
  - name: http-query
    port: 16686
    protocol: TCP
    targetPort: 16686
  selector:
    app.kubernetes.io/component: all-in-one
    app.kubernetes.io/instance: simplest
    app.kubernetes.io/managed-by: jaeger-operator
    app.kubernetes.io/name: simplest
    app.kubernetes.io/part-of: jaeger
    app: jaeger
  sessionAffinity: None
  type: NodePort
EOF

3. Will be available in Reef, change Ceph image for cluster.yaml/cluster-test.yaml and use a v18+/Ceph main builds:

4. add jaeger configuration to cluster-test ConfigMap under [global] settings

jaeger_tracing_enable = true
jaeger_agent_port = 6831

Jaeger requires a sidecar agent for the instrumented code to collect traces from, for that we need to inject Jaeger as a sidecar on OSDs and RGW

add annotations to the cluster. so that jaeger will inject an agent side-car to OSD pods:

spec:
  annotations:
    osd:
      sidecar.jaegertracing.io/inject: "true" 

$ kubectl create -f cluster-test.yaml

add annotations to the cluster. so that jaeger will inject an agent side-car to RGW object store object-test.yaml:

gateway:
  annotations:
    sidecar.jaegertracing.io/inject: "true"

$ kubectl create -f object-test.yaml

Test

• we will create storage class and a bucket:

$ kubectl create -f storageclass-bucket-delete.yaml
$ kubectl create -f object-bucket-claim-delete.yaml

• create a service so that it could be accessed from outside of k8s:

$ cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: rook-ceph-rgw-my-store-external
  namespace: rook-ceph
  labels:
    app: rook-ceph-rgw
    rook_cluster: rook-ceph
    rook_object_store: my-store
spec:
  ports:
  - name: rgw
    port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: rook-ceph-rgw
    rook_cluster: rook-ceph
    rook_object_store: my-store
  sessionAffinity: None
  type: NodePort
EOF

• fetch the URL that allow access to the RGW service from the host running the minikube VM:

$ export AWS_URL=$(minikube service --url rook-ceph-rgw-my-store-external -n rook-ceph)

• user credentials and bucket name:

$ export AWS_ACCESS_KEY_ID=$(kubectl -n default get secret ceph-delete-bucket -o jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 --decode)
$ export AWS_SECRET_ACCESS_KEY=$(kubectl -n default get secret ceph-delete-bucket -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 --decode)
$ export BUCKET_NAME=$(kubectl get objectbucketclaim ceph-delete-bucket -o jsonpath='{.spec.bucketName}')

• now use them to upload an object:

$ echo "hello world" > hello.txt
$ aws --endpoint-url "$AWS_URL" s3 cp hello.txt s3://"$BUCKET_NAME"

• fetch the URL that allow access to the jaeger query service from the host running the minikube VM:

$ export JAEGER_URL=$(minikube service --url simplest-query-external -n observability)

• query traces:

$ curl "$JAEGER_URL/api/traces?service=rgw&limit=20&lookback=1h" | jq