dasniko/_keycloak-cluster-config.md

Keycloak Cluster Configuration (How to)

This is a short and simple example on how to build a proper Keycloak cluster, using DNS_PING as discovery protocol and an NGINX server as reverse proxy.

If you prefer to use JDBC_PING, see @xgp's example gist here: https://gist.github.com/xgp/768eea11f92806b9c83f95902f7f8f80

Please see also my video about Keycloak Clustering: http://www.youtube.com/watch?v=P96VQkBBNxU
NOTE: The video covers JDBC_PING protocol and uses the legacy Keycloak Wildfly distribution!

keepthemomentum · 2022-10-25T15:55:32Z

Hi @dasniko

I need your help in configuring keycloak cluster. I'm using the custom stack for JDBC-PING to discover the instances, but its not working.
I have a shared Database and running two separate keycloak instances in docker in two different EC2 instances. Do i need to the IP address of atleast one instance for it to discover?
something like this or in the jdbc-custom stack. Any help on this would be much appreciated.

#IP address of this host, please make sure this IP can be accessed by the other Keycloak instances
JGROUPS_DISCOVERY_EXTERNAL_IP=172.21.48.39
#protocol
JGROUPS_DISCOVERY_PROTOCOL=JDBC_PING

Thank you!

mattiashem · 2022-11-30T20:39:34Z

command: start-dev -Djgroups.dns.query=keycloak

Cab you get it running in non dev mode ?

mattiashem · 2022-11-30T20:52:03Z

This is what i ended up with tp get it running in k8s in prod mode

     - args:
        - start
        - --proxy
        - edge
        - --hostname-strict=false
        env:
        - name: KC_CACHE
          value: ispn
        - name: KC_CACHE_STACK
          value: kubernetes
        - name: DB_VENDOR
          value: mariadb
        - name: DB_ADDR
          value: 
        - name: DB_PORT
          value: "3306"
        - name: DB_DATABASE
          value: 
        - name: JDBC_PARAMS
          value: connectTimeout=30000
        - name: JAVA_OPTS
          value: -Djboss.as.management.blocking.timeout=30000
        - name: PROXY_ADDRESS_FORWARDING
          value: "true"
        - name: KEYCLOAK_ADMIN
          value: 
        - name: KEYCLOAK_ADMIN_PASSWORD
          value: 
        - name: KEYCLOAK_WELCOME_THEME
          value: 
        - name: KEYCLOAK_DEFAULT_THEME
          value: 
        - name: KC_HOSTNAME
          value: 
        - name: KC_HTTP_ENABLED
          value: "true"
        - name: KC_HEALTH_ENABLED

arjunarisang · 2023-02-09T11:18:52Z

Hi @mattiashem, can you share full k8s yaml file? Do you use k8s ingress or dedicated nginx for reverse proxy?

mattiashem · 2023-02-09T11:47:26Z

`kind: StatefulSet
metadata:
  annotations:
  labels:
    app.kubernetes.io/instance: stackcore
    app.kubernetes.io/name: keycloak
  name: keycloak
  namespace: mantiser
spec:
  podManagementPolicy: Parallel
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: keycloak
      app.kubernetes.io/name: keycloak
  serviceName: keycloak-headless
  template:
    metadata:
      annotations:
      labels:
        app.kubernetes.io/instance: keycloak
        app.kubernetes.io/name: keycloak
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app.kubernetes.io/component
                  operator: NotIn
                  values:
                  - test
                matchLabels:
                  app.kubernetes.io/instance: keycloak
                  app.kubernetes.io/name: keycloak
              topologyKey: failure-domain.beta.kubernetes.io/zone
            weight: 100
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app.kubernetes.io/component
                operator: NotIn
                values:
                - test
              matchLabels:
                app.kubernetes.io/instance: keycloak
                app.kubernetes.io/name: keycloak
            topologyKey: kubernetes.io/hostname
      containers:
      - args:
        - start
        - --proxy
        - edge
        - --hostname-strict=false
        env:
        - name: jgroups.dns.query
          value: keycloak-headless.mantiser.svc.cluster.local
        - name: KC_CACHE
          value: ispn
        - name: KC_CACHE_STACK
          value: kubernetes
        - name: DB_VENDOR
          value: mariadb
        - name: DB_ADDR
          value: mysql-admin-pxc.mysql-admin.svc
        - name: DB_PORT
          value: "3306"
        - name: DB_DATABASE
          value: mantiser_keycloak
        - name: JDBC_PARAMS
          value: connectTimeout=30000
        - name: JAVA_OPTS
          value: -Djboss.as.management.blocking.timeout=30000
        - name: PROXY_ADDRESS_FORWARDING
          value: "true"
        - name: KEYCLOAK_ADMIN
          value: 
        - name: KEYCLOAK_ADMIN_PASSWORD
          value: 
        - name: KEYCLOAK_WELCOME_THEME
          value: mantiser
        - name: KEYCLOAK_DEFAULT_THEME
          value: mantiser
        - name: KC_HOSTNAME
          value: auth.mantiser.com
        - name: KC_HTTP_ENABLED
          value: "true"
        - name: KC_HEALTH_ENABLED
          value: "true"
        envFrom:
        - secretRef:
            name: keycloak-db
        image: mantiser/keycloak:latest
        imagePullPolicy: IfNotPresent
        name: keycloak
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        - containerPort: 8443
          name: https
          protocol: TCP
        - containerPort: 9990
          name: http-management
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /health/ready
            port: http
            scheme: HTTP
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        securityContext:
          runAsNonRoot: true
          runAsUser: 1000
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /opt/jboss/startup-scripts/keycloak.cli
          name: startup
          readOnly: true
          subPath: keycloak.cli
        - mountPath: /etc/tls
          name: tls
          readOnly: true
      dnsPolicy: ClusterFirst
      enableServiceLinks: true
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1000
      serviceAccount: keycloak
      serviceAccountName: keycloak
      terminationGracePeriodSeconds: 60
      volumes:
      - name: tls
        secret:
          defaultMode: 420
          optional: false
          secretName: api-mantiser-tls
      - configMap:
          defaultMode: 365
          items:
          - key: keycloak.cli
            path: keycloak.cli
          name: keycloak-startup
        name: startup
  updateStrategy:
    type: RollingUpdate

This is a describe on the stateful set. I use Traefik ingress as a reverse proxy

arjunarisang · 2023-02-09T12:27:31Z

Awesome! Thanks a lot @mattiashem

germanllop · 2023-03-30T05:27:14Z

Any workaound with Docker Swarm on v21?

mattiashem · 2023-03-31T08:19:14Z

I use this in docker compose maybe can help you in the right direction

mysql: image: mysql:5.7 volumes: - ./mysql:/var/lib/mysql environment: MYSQL_ROOT_PASSWORD: root MYSQL_DATABASE: keycloak MYSQL_USER: keycloak MYSQL_PASSWORD: password keycloak: image: quay.io/keycloak/keycloak environment: DB_VENDOR: MYSQL DB_ADDR: mysql DB_DATABASE: keycloak DB_USER: keycloak DB_PASSWORD: password KEYCLOAK_ADMIN: admin KEYCLOAK_ADMIN_PASSWORD: Pa55w0rd ports: - 8080:8080 - 9990:9990 depends_on: - mysql command: start-dev

germanllop · 2023-04-01T01:24:59Z

I use this in docker compose maybe can help you in the right direction

mysql: image: mysql:5.7 volumes: - ./mysql:/var/lib/mysql environment: MYSQL_ROOT_PASSWORD: root MYSQL_DATABASE: keycloak MYSQL_USER: keycloak MYSQL_PASSWORD: password keycloak: image: quay.io/keycloak/keycloak environment: DB_VENDOR: MYSQL DB_ADDR: mysql DB_DATABASE: keycloak DB_USER: keycloak DB_PASSWORD: password KEYCLOAK_ADMIN: admin KEYCLOAK_ADMIN_PASSWORD: Pa55w0rd ports: - 8080:8080 - 9990:9990 depends_on: - mysql command: start-dev

Thanks, I've already make it work on docker with a JDBC cluster.
I was wondering if there is any workaround to use the KC_CACHE_STACK: kubernetes on docker swarm, they don't discover each other, maybe a naming, I can't specify static IP because the replicas can spawn on different nodes, so there is the catch and the KC_CACHE_STACK: kubernetes is not finding the other keycloak replicas :(

bdoublet91 · 2023-04-26T12:57:11Z

Hi,
You can find what you want here:
Work for keycloak 20.0.3
keycloak/keycloak#10210
The main problem for docker swarm is node discovery with multiple ip interface container. Also a problem with healthcheck container but it's possible but harder than kubernetes :)

zhandospdf · 2023-05-24T21:33:13Z

Hello,
When using Jboss based Keycloak versions, it was possible to query the cluster size using jboss-cli.sh.
Does anyone know if it is possible to query (just to confirm) the cluster size in Quarkus based Keycloak ?

dasniko · 2023-05-25T07:06:44Z

Not by default. You'll have to provide your custom Infinispan XML config file with proper values - or, with placeholders to be replaced by environment variable values during runtime.

zhandospdf · 2023-05-25T07:33:43Z

@dasniko thanks for reply, but I am not sure if I got it right. Was your answer about the cache configuration and how we can set number of owners for a cache ?

My original question was more about how can we query the total number of Keycloak nodes present in the cluster at runtime. Any clues about that ?

P.S.
With Jboss I used to do it like this: /opt/jboss/keycloak/bin/jboss-cli.sh -c --command="/subsystem=jgroups/channel=ee:read-attribute(name=view)"

dasniko · 2023-05-25T07:40:52Z

@zhandospdf Ok, then I misunderstood your question.
AFAIK there's no ootb way to do this. Perhaps some custom java code could help here, but I don't have any details.

1capedbaldy · 2023-08-03T09:16:55Z

Hello. I am new to this. I tried this demo following your steps but I got a problem: nginx encountered this error and exited
nginx: [emerg] host not found in upstream "keycloak-1:8080" in /etc/nginx/conf.d/default.conf:2
Can you help me with this?

dasniko · 2023-08-03T11:10:19Z

Replace keycloak-1 (and keycloak-2) with the hostname of the Keycloak containers, as they are reachable from within the Docker network. Usually (default) this is <name-of-the-folder>-<servicename>-<running-number-of-replica>

PreeteshShettigar · 2023-10-06T20:24:02Z

I recently upgraded from Keycloak 18 to version 21.1.0. In my previous setup, Keycloak was running with cache stack as kubernates across multiple pods. However, after the upgrade, I encountered an issue where the jgroups were unable to locate multiple nodes

samorganist · 2023-10-08T14:31:12Z

Hello , anybody know how to share the cache between hosts that runs the same docker-compose like in this example?
i have this config:
Server 1 & 2 : running docker-compose
ExternalDatabse
loadBalancer: switch between the two servers

the cache is shared only on the instances running on the same server, so if you have a solution for this it will be so helpful for me.

vladano · 2024-12-24T18:02:04Z

Hi @dasniko
The name of the nging conf file should be nginx.conf not nginx.con.

On my Windows computer I created folder mykeycloak1 and put this code inside this folder.
Files docker-compose-cluster.yml and nginx.conf.
After that I run:
docker compose -f docker-compose-cluster.yml up -d
also I modified nginx.conf file:

upstream backend {
    ip_hash;
#    server keycloak-1:8080 fail_timeout=2s;
#    server keycloak-2:8080 fail_timeout=2s;
    server mykeycloak1-keycloak-1:8080 fail_timeout=5s;
    server mykeycloak1-keycloak-2:8080 fail_timeout=5s;
}

server {
    listen       8000;
    server_name  localhost;
    access_log   off;

    location / {
        proxy_set_header    X-Forwarded-For    $proxy_add_x_forwarded_for;
        proxy_set_header    X-Forwarded-Proto  $scheme;
        proxy_set_header    X-Forwarded-Host   $host;
        proxy_set_header    X-Forwarded-Port   $server_port;
        
        proxy_pass              http://backend;
        proxy_connect_timeout   2s;

        proxy_buffer_size          128k;
        proxy_buffers              4 256k;
        proxy_busy_buffers_size    256k;
    }
}

All containers running successfully.

When I try to access URL http://localhost:8000 it redirect my to URL https://localhost/admin/ with the message "Unable to connect"

Current latest version of keycloak (quay.io/keycloak/keycloak:latest) is 26.0.7.
Is this an issue with keycloak version 26.0.7 or this is an issue with nginx configuration?

Thanks in advance,
VladanO

	version: '3.8'

	services:
	postgres:
	image: postgres:latest
	environment:
	POSTGRES_USER: keycloak
	POSTGRES_PASSWORD: passw0rd
	volumes:
	- pg-data:/var/lib/postgresql/data

	keycloak:
	image: quay.io/keycloak/keycloak:latest
	command: start-dev -Djgroups.dns.query=keycloak
	environment:
	KC_CACHE: ispn
	KC_CACHE_STACK: kubernetes
	KC_DB: postgres
	KC_DB_URL: jdbc:postgresql://postgres:5432/keycloak
	KC_DB_USERNAME: keycloak
	KC_DB_PASSWORD: passw0rd
	KC_PROXY: edge
	KC_HOSTNAME: localhost
	KC_HOSTNAME_PORT: '8000'
	KEYCLOAK_ADMIN: admin
	KEYCLOAK_ADMIN_PASSWORD: admin
	deploy:
	replicas: 2
	endpoint_mode: dnsrr

	lb:
	image: nginx:alpine
	volumes:
	- ./nginx.conf:/etc/nginx/conf.d/default.conf
	ports:
	- "8000:8000"

	volumes:
	pg-data:
	name: keycloak-demo-cluster-data

dasniko/_keycloak-cluster-config.md

Keycloak Cluster Configuration (How to)

keepthemomentum commented Oct 25, 2022

mattiashem commented Nov 30, 2022

mattiashem commented Nov 30, 2022

arjunarisang commented Feb 9, 2023 •

edited

Loading

mattiashem commented Feb 9, 2023

arjunarisang commented Feb 9, 2023

germanllop commented Mar 30, 2023

mattiashem commented Mar 31, 2023

germanllop commented Apr 1, 2023

bdoublet91 commented Apr 26, 2023 •

edited

Loading

zhandospdf commented May 24, 2023

dasniko commented May 25, 2023

zhandospdf commented May 25, 2023 •

edited

Loading

dasniko commented May 25, 2023

1capedbaldy commented Aug 3, 2023

dasniko commented Aug 3, 2023

PreeteshShettigar commented Oct 6, 2023 •

edited

Loading

samorganist commented Oct 8, 2023

vladano commented Dec 24, 2024

	upstream backend {
	ip_hash;
	server keycloak-1:8080 fail_timeout=2s;
	server keycloak-2:8080 fail_timeout=2s;
	}

	server {
	listen 8000;
	server_name localhost;
	access_log off;

	location / {
	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	proxy_set_header X-Forwarded-Proto $scheme;
	proxy_set_header X-Forwarded-Host $host;
	proxy_set_header X-Forwarded-Port $server_port;

	proxy_pass http://backend;
	proxy_connect_timeout 2s;

	proxy_buffer_size 128k;
	proxy_buffers 4 256k;
	proxy_busy_buffers_size 256k;
	}
	}

dasniko/_keycloak-cluster-config.md

Keycloak Cluster Configuration (How to)

keepthemomentum commented Oct 25, 2022

mattiashem commented Nov 30, 2022

mattiashem commented Nov 30, 2022

arjunarisang commented Feb 9, 2023 • edited Loading

mattiashem commented Feb 9, 2023

arjunarisang commented Feb 9, 2023

germanllop commented Mar 30, 2023

mattiashem commented Mar 31, 2023

germanllop commented Apr 1, 2023

bdoublet91 commented Apr 26, 2023 • edited Loading

zhandospdf commented May 24, 2023

dasniko commented May 25, 2023

zhandospdf commented May 25, 2023 • edited Loading

dasniko commented May 25, 2023

1capedbaldy commented Aug 3, 2023

dasniko commented Aug 3, 2023

PreeteshShettigar commented Oct 6, 2023 • edited Loading

samorganist commented Oct 8, 2023

vladano commented Dec 24, 2024

arjunarisang commented Feb 9, 2023 •

edited

Loading

bdoublet91 commented Apr 26, 2023 •

edited

Loading

zhandospdf commented May 25, 2023 •

edited

Loading

PreeteshShettigar commented Oct 6, 2023 •

edited

Loading