azure-ad-k8s-idp.md

The following are steps to set up Azure AD as an Identity Provider for an on-prem Kubernetes cluster.

Starting point:

• Azure Subscription
• On-prem Kubernetes cluster (in my case RKE2 v1.27.12+rke2r1)

The scenario is pretty straightforward:

I want to connect from my Workstation (Windows/Linux/Mac) to an on-prem Kubernetes cluster and authenticate to is using Azure AD.

I could not find many useful resources for this scenario except the dbi blog, so i decided to expand on it a little.

• Create an App Registration in Azure
• Use kubelogin to authenticate to Azure AD through the App Registration and aquire a token
• Use that token to authenticate to Kubernetes API Server

This is the basic flow from the kubelogin repo:

Steps:

- On the Azure side

1. Create an App Registration and set its Redirect URI to http://localhost:8000 (Azure will only accept localhost with http, every other URI has to be https)

2. The App registration needs to have User.Read permissions to be able to sign users in and read their info.

3. Add a group claim for the tokens so that you can use group membership for Kubernetes RBAC. Keep in mind that there are limits to the number of group memberships a single token can hold, more on that: MSFT Docs.

NOTE: If you have a large AD with many groups you can assign specific groups to the App Registration so that only those groups are emitted in the token. MSFT Docs. In that case you should only select the 4th option:

4. Create a client secret to use when requesting tokens.

- On the Kubernetes Cluster side:

1. Add the following arguments to your Kubernetes API server.

  --oidc-client-id=<azure-app-registration-application-id>
  --oidc-issuer-url=https://sts.windows.net/<azure-tenant-id>/
  --oidc-username-claim=upn
  --oidc-username-prefix=oidc:
  --oidc-groups-claim=groups
  --oidc-groups-prefix=oidc:

These arguments are very well explained in the K8S Docs, i will just explain shortly what they mean to us practically in this example.

--oidc-username-claim - points the Kubernetes API to the fields in the JWT to use for the username. If you use upn you will be using the user principal name of the user for K8S role assignment. Other fields from the JWT can be used, for example oid - object id of the user.

--oidc-username-prefix - prefix prepended to the user claim. This is an arbitrary string.

--oidc-groups-claim - points the Kubernetes API to the fields in the JWT to use for the users group membership.

--oidc-groups-prefix - prefix prepended to group claims. This is also an arbitrary string.

- On the client (Workstation) side

1. Install kubelogin link

# Krew (macOS, Linux, Windows and ARM)
kubectl krew install oidc-login

2. Add user to your kubeconfig file, for example:
   

using kubectl:

```
kubectl config set-credentials <user_name> \
--exec-api-version=client.authentication.k8s.io/v1beta1 \
--exec-command=kubectl \
--exec-arg=oidc-login \
--exec-arg=get-token \
--exec-arg=--oidc-issuer-url=https://sts.windows.net/<azure-tenant-id>/ \
--exec-arg=--oidc-client-id=<azure-app-registration-application-id> \
--exec-arg=--oidc-client-secret=<azure-app-registration-client-secret> \
--kubeconfig <path-to-kubeconfig>
```

resulting in something like:
```
users:
- name: <user_name>
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - oidc-login
      - get-token
      - --oidc-issuer-url=https://sts.windows.net/<azure-tenant-id>/
      - --oidc-client-id=<azure-app-registration-application-id>
      - --oidc-client-secret=<azure-app-registration-client-secret>
      command: kubectl
      env: null
      interactiveMode: IfAvailable
      provideClusterInfo: false

```
NOTE: Don't forget to set the appropriate context (in the kubeconfig file) using the newly added user.

3. When connecting to the Kubernetes cluster (for example using kubectl) a browser will open for you to authenticate to Azure AD. After successful authentication a token will be aquired and cached and used to communicate further with the Kubernetes API Server. You can examine the cached token if necessary for troubleshooting (~/.kube/cache/oidc-login).

- Assigning RBAC roles

Now that the authentication is done, we need to assign RBAC roles to our user.

1. Define an example role:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: test-clusterrole
rules:
- apiGroups: [""]
  resources: ["nodes", "pods"]
  verbs: ["get", "watch", "list"]

2. Assigning role to a user:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: test-clusterrolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: test-clusterrole
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: "oidc:<user_principal_name>" # here our --oidc-username-prefix argument value comes into play 

3. Assigning role to a group:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: test-clusterrolebinding-group
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: test-clusterrole
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: Group
    name: "oidc:<group_object_id>" # here our --oidc-groups-prefix argument value comes into play

@vaspoz

Getting error :

kubectl --context=ev get --raw /.well-known/openid-configuration --v=10

I0306 20:21:06.128145 67247 loader.go:395] Config loaded from file: /Users/prathap.dasari/.kube/config
I0306 20:21:06.128377 67247 round_trippers.go:466] curl -v -XGET -H "Accept: application/json, /" -H "User-Agent: kubectl/v1.30.3 (darwin/arm64) kubernetes/6fc0a69" 'https://:6443/.well-known/openid-configuration'
I0306 20:21:06.173922 67247 round_trippers.go:510] HTTP Trace: Dial to tcp::6443 succeed
I0306 20:21:06.257599 67247 round_trippers.go:553] GET https://***********:6443/.well-known/openid-configuration 401 Unauthorized in 129 milliseconds
I0306 20:21:06.257632 67247 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 32 ms TLSHandshake 28 ms ServerProcessing 39 ms Duration 129 ms
I0306 20:21:06.257636 67247 round_trippers.go:577] Response Headers:
I0306 20:21:06.257642 67247 round_trippers.go:580] Audit-Id: 13f732be-2254-403e-bb06-c52be64341aa
I0306 20:21:06.257645 67247 round_trippers.go:580] Cache-Control: no-cache, private
I0306 20:21:06.257648 67247 round_trippers.go:580] Content-Type: application/json
I0306 20:21:06.257649 67247 round_trippers.go:580] Content-Length: 129
I0306 20:21:06.257651 67247 round_trippers.go:580] Date: Thu, 06 Mar 2025 19:21:06 GMT
I0306 20:21:06.257682 67247 request.go:1212] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
I0306 20:21:06.257901 67247 helpers.go:246] server response object: [{
"metadata": {},
"status": "Failure",
"message": "Unauthorized",
"reason": "Unauthorized",
"code": 401
}]
error: You must be logged in to the server (Unauthorized)

@vaspoz Getting correct token when retrieving the token with all details but when checking with the same token it does not work as expected ..

do we need to give any permissions here ?

@pr07pr07 , I had exactly the same issue couple of months ago lol. And it was an eeeeeasy fix, something that i just overlooked.
But i really don't remember, sorry man
maybe tomorrow with a fresh mind i would recall

@vaspoz That would be great if you could remember.

My suspicion is that something is missing in the K8s cluster itself. Anyhow, I can wait for some time; meanwhile, I will try to debug the issue.

Thank you in advance 👍

@vaspoz Also, may I know which version of K8s you implemented this on?

@pr07pr07
For me, it works on my current version of K8S - 1.30.8
Recheck you kubernetes api arguments maybe?

For example by getting the process on a master node
ps aux | grep kube-apiserver

or describing the kubernetes-api pod:
kubectl describe pod kube-apiserver-<node-name> -n kube-system

For me its like this:

		--oidc-groups-claim=groups 
		--oidc-groups-prefix=azuread: 
		--oidc-issuer-url=https://sts.windows.net/<tennant-id>/ 
		--oidc-username-claim=upn 
		--oidc-username-prefix=azuread: 

then because of the prefix i chose azuread: the role binding is as follows:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: azuread:<azure-group-object-id>

Maybe the prefixs dont match? By default it should be oidc:. If that's the case you should be seeing 401 unauthorized when trying to execute a command.

Also maybe check the content of token you are getting by executing:
kubectl oidc-login get-token --oidc-issuer-url=https://sts.windows.net/<azure-tenant-id>/ --oidc-client-id=<azure-app-registration-application-id> --oidc-client-secret=<azure-app-registration-client-secret>

@lukapetrovic-git

Yes, I am unable to see those arguments when describing kube-apiserver-. This could be the problem.

I have added them to /etc/kubernetes/manifests/kube-apiserver.yaml, but I still cannot see them when describing it.

@lukapetrovic-git

The location /etc/kubernetes/manifests/ is specifically designated for static pods. Any YAML file placed in this directory is automatically picked up by Kubernetes, which then creates the corresponding static pods.
During our backup process, a copy of the kube-apiserver.yaml file was placed in this directory.
As a result, Kubernetes automatically created an additional API server instance, leading to our changes not being reflected as expected.

Thank you !!

@pr07pr07 glad you figured it out! :)