Debezium GKE Setup

debezium-cm.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: debezium
  labels:
    app: debezium
  namespace: debezium
data:
  application.properties: |-
      debezium.sink.type=pubsub
      debezium.sink.pubsub.project.id=acko-poc
      debezium.sink.pubsub.ordering.enabled=true
      debezium.source.connector.class=io.debezium.connector.postgresql.PostgresConnector
      debezium.source.offset.storage.file.filename=data/offsets.dat
      debezium.source.offset.flush.interval.ms=0
      debezium.source.database.hostname=localhost #as we are connecting via Cloud Proxy
      debezium.source.database.port= <Postgres Port> #generally 5432
      debezium.source.database.user= <user created> # in our case - debezium_user 
      debezium.source.database.password=<password while creating new user> #in our case "admin"
      debezium.source.database.dbname=<Database> #OurCase postgres
      debezium.source.database.server.name=<Database Server name> #In Our Case debezium-db
      debezium.source.table.include.list=public.actor #Table which needs to captured
      debezium.source.plugin.name=pgoutput

debezium-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: debezium
  labels:
    name: debezium
  namespace: debezium
spec:
  type: ClusterIP
  ports:
    - port: 8080
      targetPort: 8080
      protocol: TCP
      name: http
  selector:
    name: debezium

debezium-statefulset.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: debezium-gke
  labels:
    name: debezium
  namespace: debezium
spec:
  replicas: 1
  serviceName: debezium
  selector:
    matchLabels:
      name: debezium
  template:
    metadata:
      labels:
        name: debezium
        version: v1
    spec:
      serviceAccountName: <Service Name Created while creating Cluster>
      nodeSelector:
        iam.gke.io/gke-metadata-server-enabled: "true"
      containers:
        - name: debezium
          image: debezium/server:1.7.0.Final
          volumeMounts:
            - name: debezium-config-volume
              mountPath: /debezium/conf
            - name: debezium-data-volume
              mountPath: /debezium/data
        - name: cloud-sql-proxy
          image: gcr.io/cloudsql-docker/gce-proxy:1.17
          command:
            - "/cloud_sql_proxy"
            - "-instances=<ConnectionName from CloudSQL Page>=tcp:<Port>"
          securityContext:
            runAsNonRoot: true
      volumes:
        - name: debezium-config-volume
          configMap:
            name: debezium
  volumeClaimTemplates:
    - metadata:
        name: debezium-data-volume
      spec:
        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 10Gb

Hey As the exception says- the Sink Type config is missing
Please use this debezium.sink.type=pubsub

Hi , i have already added this debezium.sink.type=pubsub in my config map file, but I'm still getting this error

Check if your configMap is updated and deployed. Also, the Pub Sub topic name needs to be in the proper format as mentioned

Hi that error is gone now, but I can this new error

Seems to be related to your configuration, Please validate

My PG SQL version is 11, do you think this may be causing the issue ?

And also i just want to ask one more thing in the debezium-statefulset.yaml file at line no-20, which service account do we need to pass? the one which is using by GKE VM's?