Harbor Private Registry.md

Harbor Private Registry

Software

• Ubuntu 20.04.4 TLS
  • Router - 42.118.242.163
  • K8s-master - 10.0.0.200
  • Elastic-01 - 10.0.0.51
  • Elastic-02 - 10.0.0.52
  • Elastic-03 - 10.0.0.53
• Haproxy v2.2
• Kubernetes:
  • kubectl - v1.23.4
  • kubeadm - v1.23.7
  • kubelet - v1.23.7
  • nginx-ingress-controller - v1.3
• Docker v20.10.12
• Harbor v2.5.3
• Helm v3.9.0

Network Topology

Installation

Helm

• Refer helm install documents

Kubernetes

• Create folders on all worker nodes and master node

  • /mnt/harbor/registry
  • /mnt/harbor/redis
  • /mnt/harbor/database
  • /mnt/harbor/chartmuseum
  • /mnt/harbor/jobservice
  • /mnt/harbor/trivy

• Create storage class, we use local storage provisioner

  apiVersion: storage.k8s.io/v1
  kind: StorageClass
  metadata:
    name: harbor
    annotations:
      storageclass.kubernetes.io/is-default-class: "true"
  provisioner: kubernetes.io/no-provisioner
  reclaimPolicy: Retain
  volumeBindingMode: WaitForFirstConsumer

• Create persistent volume

  • Registry persistent volume

  apiVersion: v1
  kind: PersistentVolume
  metadata:
    name: harbor-registry
  spec:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 5Gi
    local:
      path: /mnt/harbor/registry
    nodeAffinity:
      required:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/os
            operator: In
            values:
            - linux
    persistentVolumeReclaimPolicy: Delete
    storageClassName: harbor

  • Chartmuseum persistent volume

  apiVersion: v1
  kind: PersistentVolume
  metadata:
    name: harbor-chartmuseum
  spec:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 5Gi
    local:
      path: /mnt/harbor/chartmuseum
    nodeAffinity:
      required:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/os
            operator: In
            values:
            - linux
    persistentVolumeReclaimPolicy: Delete
    storageClassName: harbor

  • Jobservice persistent volume

  apiVersion: v1
  kind: PersistentVolume
  metadata:
    name: harbor-jobservice
  spec:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 5Gi
    local:
      path: /mnt/harbor/jobservice
    nodeAffinity:
      required:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/os
            operator: In
            values:
            - linux
    persistentVolumeReclaimPolicy: Delete
    storageClassName: harbor

  • Database persistent volume

  apiVersion: v1
  kind: PersistentVolume
  metadata:
    name: harbor-database
  spec:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 5Gi
    local:
      path: /mnt/harbor/database
    nodeAffinity:
      required:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/os
            operator: In
            values:
            - linux
    persistentVolumeReclaimPolicy: Delete
    storageClassName: harbor

  • Redis persistent volume

  apiVersion: v1
  kind: PersistentVolume
  metadata:
    name: harbor-redis
  spec:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 5Gi
    local:
      path: /mnt/harbor/redis
    nodeAffinity:
      required:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/os
            operator: In
            values:
            - linux
    persistentVolumeReclaimPolicy: Delete
    storageClassName: harbor

  • Trivy persistent volume

  apiVersion: v1
  kind: PersistentVolume
  metadata:
    name: harbor-trivy
  spec:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 5Gi
    local:
      path: /mnt/harbor/trivy
    nodeAffinity:
      required:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/os
            operator: In
            values:
            - linux
    persistentVolumeReclaimPolicy: Delete
    storageClassName: harbor

Harbor

• helm repo add harbor https://helm.goharbor.io

• Download values.yaml

• Edit values.yaml file

  • Ingress section

  ingress:
    hosts:
      core: harbor.io
      notary: harbor.io
    className: "nginx"

  • externalURL

   externalURL: https://harbor.io

  • harborAdminPassword

   harborAdminPassword: "Harbor12345"

• Run helm cli

  helm template harbor harbor/harbor -f value/values.yaml | tee alochym.yaml

• Create harbor namespace

  kubectl create ns harbor

• Run kubectl cli

  kubectl -n harbor apply -f alochym.yaml

Configuration

• Ingress-nginx service

  NAME                               TYPE      CLUSTER-IP  EXTERNAL-IP PORT(S)                    AGE
  ingress-nginx-controller           NodePort  10.17.17.13 <none>      80:30271/TCP,443:31232/TCP 2d20h
  ingress-nginx-controller-admission ClusterIP 10.10.23.8  <none>      443/TCP                    2d20h

• Create harbor registry secret

  kubectl create -n harbor secret docker-registry regcred --docker-server=https://harbor.io --docker-username=admin --docker-password=Harbor12345 [email protected] --dry-run=client -oyaml | tee harbor-secret.yaml

• The harbor-secret.yaml content

  apiVersion: v1
  data:
    .dockerconfigjson: eyJhdXRocyI6eyJodHRwczovL2hhcmJvci5pbyI6eyJ1c2VybmFtZSI6ImFkbWluIiwicGFzc3dvcmQiOiJIYXJib3IxMjM0NSIsImVtYWlsIjoiYWRtaW5AaGFyYm9yLmlvIiwiYXV0aCI6IllXUnRhVzQ2U0dGeVltOXlNVEl6TkRVPSJ9fX0=
  kind: Secret
  metadata:
    name: regcred
    namespace: harbor
  type: kubernetes.io/dockerconfigjson

• Apply harbor-secret.yaml to kubernetes

  kubectl apply -f harbor-secret.yaml

• Config haproxy.cfg on Router server

  frontend harbor-ssl-passthrough
    bind *:443
    mode tcp
    option tcplog
    default_backend be-harbor
  
  backend be-harbor
    mode tcp
    server harbor 10.0.0.51:31232 # same as Ingress-nginx service

• Edit coredns to add harbor.io record. Can use host alias

  apiVersion: v1
  data:
    Corefile: |
      .:53 {
        errors
        health {
          lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
          ttl 30
        }
        hosts {
          42.118.242.163 harbor.io
          fallthrough
        }
        prometheus :9153
        forward . /etc/resolv.conf {
            max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
      }
  kind: ConfigMap
  metadata:
    name: coredns
    namespace: kube-system

• Reload all coredns pods in kubernetes cluster.

• Test harbor.io record

  kubectl exec -it dnsutils - nslookup harbor.io

• Create ingress nginx rule for harbor

  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: "harbor-ingress"
    labels:
      heritage: Helm
      release: harbor
      chart: harbor
      app: "harbor"
    annotations:
      ingress.kubernetes.io/proxy-body-size: "0"
      ingress.kubernetes.io/ssl-redirect: "true"
      nginx.ingress.kubernetes.io/proxy-body-size: "0"
      nginx.ingress.kubernetes.io/ssl-redirect: "true"
  spec:
    ingressClassName: nginx
    tls:
    - secretName: harbor-ingress
      hosts:
      - harbor.io
    rules:
    - http:
        paths:
        - path: /api/
          pathType: Prefix
          backend:
            service:
              name: harbor-core
              port:
                number: 80
        - path: /service/
          pathType: Prefix
          backend:
            service:
              name: harbor-core
              port:
                number: 80
        - path: /v2
          pathType: Prefix
          backend:
            service:
              name: harbor-core
              port:
                number: 80
        - path: /chartrepo/
          pathType: Prefix
          backend:
            service:
              name: harbor-core
              port:
                number: 80
        - path: /c/
          pathType: Prefix
          backend:
            service:
              name: harbor-core
              port:
                number: 80
        - path: /
          pathType: Prefix
          backend:
            service:
              name: harbor-portal
              port:
                number: 80
      host: harbor.io

• Download harbor registry certificate

  wget https://harbor.io/api/v2.0/systeminfo/getcert -O ca.crt

• Create harbor.io folder on all worker nodes and master node

  mkdir -p /etc/docker/certs.d/harbor.io/

• Copy harbor registry certificate to all worker nodes and master node

  scp ca.crt 10.0.0.200:/etc/docker/certs.d/harbor.io/
  scp ca.crt 10.0.0.50:/etc/docker/certs.d/harbor.io/
  scp ca.crt 10.0.0.51:/etc/docker/certs.d/harbor.io/
  scp ca.crt 10.0.0.52:/etc/docker/certs.d/harbor.io/

• Config /etc/hosts on all worker nodes and master nodes

  42.118.242.163 harbor.io

• Login Docker to harbor.io with username/password - admin/Harbor12345

  docker login https://harbor.io

Test

• Flask source code is store on elastic-03 - /home/hadn/flask-demo

  flask-demo/
  ├── app.py
  ├── Dockerfile
  └── requirements.txt

• Dockerfile

  FROM python:3.7
  RUN mkdir /catking
  WORKDIR /catking
  COPY ./requirements.txt /catking
  RUN pip install -r requirements.txt
  COPY . /catking
  CMD ["python", "app.py"]
  EXPOSE 5000

• app.py

  from flask import Flask
  app = Flask(__name__)
  @app.route('/')
  def hello_world():
      return 'Hello World'
  
  if __name__ == '__main__':
      app.run(host="0.0.0.0")

• requirements.txt

  click==8.1.3
  Flask==2.1.3
  importlib-metadata==4.12.0
  itsdangerous==2.1.2
  Jinja2==3.1.2
  MarkupSafe==2.1.1
  typing-extensions==4.3.0
  Werkzeug==2.2.0
  zipp==3.8.1

• Kaniko

  apiVersion: v1
  kind: Pod
  metadata:
    name: kaniko-secret
    namespace: harbor
  spec:
    containers:
    - name: kaniko
      image: gcr.io/kaniko-project/executor:debug
      args: ["--dockerfile=/demo/Dockerfile",
              "--context=/demo",
              "--insecure=true",
            "--skip-tls-verify=true",
              "--destination=harbor.io/library/flask-web:v1"]
      volumeMounts:
        - name: docker-config
          mountPath: /kaniko/.docker/
        - name: project-volume
          mountPath: /demo
    restartPolicy: Never
    volumes:
      - name: docker-config
        secret:
          secretName: regcred
          items:
            - key: .dockerconfigjson
              path: config.json
      - name: project-volume
        hostPath:
          path: /home/hadn/flask-demo
    nodeSelector:
      kubernetes.io/hostname: elastic-03

• Login harbor portal and check project library