Create domain joined k8s Windows worker AD groups and service account

build-windows-image.md

Build Windows 2019 Image

This example assumes you're going to run the dotnet-environment-viewer sample application. Create a Dockerfile at the root of the application based on the aspnet framework image.

FROM mcr.microsoft.com/dotnet/framework/aspnet:4.8

# The following installs and configured Windows Auth for the app (most apps won't need this)
RUN powershell.exe Add-WindowsFeature Web-Windows-Auth
RUN powershell.exe -NoProfile -Command Set-WebConfigurationProperty -filter /system.WebServer/security/authentication/AnonymousAuthentication -name enabled -value false -PSPath 'IIS:\'
RUN powershell.exe -NoProfile -Command Set-WebConfigurationProperty -filter /system.webServer/security/authentication/windowsAuthentication -name enabled -value true -PSPath 'IIS:\'

# The final instruction copies the site you published earlier into the container.
COPY ./ViewEnvironment/ /inetpub/wwwroot

From the directory where the Dockerfile is, build the image and then push it to Harbor.

PS> docker build -t harbor.run.haas.pez.example.com/library/dotnet-environment-viewer:latest .
PS> docker push harbor.run.haas.pez.example.com/library/dotnet-environment-viewer:latest

Now schedule a dotnet environment viewer pod and expose it on a load balancer. kubectl apply the following:

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: hellowindows
  name: hellowindows
spec:
  containers:
  - name: hellowindows
    image: shawnneal/dotnet-environment-viewer
  tolerations:
  - key: "windows"
    value: "2019"
    effect: NoExecute
  nodeSelector:
    kubernetes.io/os: windows
---
apiVersion: v1
kind: Service
metadata:
  labels:
    run: dotnet-environment-viewer
  name: dotnet-environment-viewer-svc
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: dotnet-environment-viewer
  sessionAffinity: None
  type: LoadBalancer

Take note of the load balancer IP and create a DNS entry to point to it that matches one of the SPNs you used when creating the gMSA.

gmsa-login.sql

CREATE LOGIN [ad\contoso$] FROM WINDOWS
sp_addsrvRolemember "ad\contoso$", "sysadmin"

sql-gmsa.ps1

# Create the security group
$sqlHostsGroup = New-ADGroup -Name 'SQL authorized hosts' -SamAccountName 'SQLHosts' -GroupScope DomainLocal -PassThru

# Add the SQL Server host(s) to the group, notice the computer name is suffixed with `$`
Add-ADGroupMember -Identity 'SQLHosts' -Members 'WIN-SQL01$'

# Create the gMSA
New-ADServiceAccount -Name 'SQL Service Account' `
  -PrincipalsAllowedToRetrieveManagedPassword 'SQLHosts' `
  -Enabled:$true `
  -DNSHostName sqlsa.ad.haas.pez.example.com `
  -SamAccountName 'sqlsa' `
  -ManagedPasswordIntervalInDays 1

# Now on the SQL Server add the gMSA, then reboot after running Install-ADServiceAccount:
Install-ADServiceAccount -Identity sqlsa
Test-ADServiceAccount -Identity sqlsa

tkgi-cluster.json

{
  "enable_gmsa": true,
  "domain_controller_ip_address": "dc1.ad.haas.pez.example.com",
  "domain_fqdn": "ad.haas.pez.example.com",
  "domain_user_username": "contosocluster",
  "domain_user_password": "CHANGEme",
  "domain_security_group": "CN=contoso authorized hosts,CN=Users,DC=ad,DC=haas,DC=pez,DC=example,DC=com",
  "domain_service_account": "contoso"
}

tkgi-gmsa.ps1

$appName = 'contoso'
$clusterUserName = 'contosocluster'
$clusterUserPassword = 'CHANGEme'
$appDomain = 'haas.pez.example.com'
$adDomain =  (Get-ADDomain).DNSRoot
$accountName = "${appName}Hosts"

$inheritedObjectTypeNoneGuid = '00000000-0000-0000-0000-000000000000'
$addRemoveSelfAsMemberGuid = 'bf9679c0-0de6-11d0-a285-00aa003049e2'
$computerObjectTypeGuid = 'bf967a86-0de6-11d0-a285-00aa003049e2'

# Create the security group
$hostsGroup = New-ADGroup -Name "$appName authorized hosts" -SamAccountName "$accountName" -GroupScope DomainLocal -PassThru

# Create the gMSA
$gmsaGroup = New-ADServiceAccount -Name "$appName" `
  -DnsHostName "${appName}.${appDomain}" `
  -ServicePrincipalNames "host/$appName", "host/${appName}.${appDomain}", "host/${appName}.${adDomain}" `
  -PrincipalsAllowedToRetrieveManagedPassword "$accountName" `
  -PassThru

# Create the cluster manager user
$clusterManagerCreds = ConvertTo-SecureString "$clusterUserPassword" -AsPlainText -force
$clusterUser = New-ADUser -Name "$clusterUserName" `
  -AccountPassword $clusterManagerCreds `
  -CannotChangePassword $true `
  -PasswordNeverExpires $true `
  -ChangePasswordAtLogon $false `
  -Enabled $true `
  -OtherAttributes @{'description'="TKGI cluster manager that can domain join Windows workers"} `
  -PassThru

# Give the cluster manager permissions to add workers to the app authorized hosts group
Set-ADGroup -ManagedBy $clusterUser -Identity $hostsGroup

$hostsGroupRule = New-Object System.DirectoryServices.ActiveDirectoryAccessRule(($clusterUser).SID, `
  ([System.DirectoryServices.ActiveDirectoryRights]::WriteProperty -bor[System.DirectoryServices.ActiveDirectoryRights]::ExtendedRight), `
  [System.Security.AccessControl.AccessControlType]::Allow, `
  $addRemoveSelfAsMemberGuid, `
  [System.DirectoryServices.ActiveDirectorySecurityInheritance]::None, `
  $inheritedObjectTypeNoneGuid)

$hostsGroupPath = "AD:\${hostsGroup}"
$hostsGroupAcl = Get-Acl $hostsGroupPath
$hostsGroupAcl.AddAccessRule($hostsGroupRule)
Set-Acl -AclObject $hostsGroupAcl -Path $hostsGroupPath

# Delegate computer account domain join perms to cluster manager
$domainJoinRule = New-Object System.DirectoryServices.ActiveDirectoryAccessRule(($clusterUser).SID, `
  [System.DirectoryServices.ActiveDirectoryRights]::CreateChild, `
  [System.Security.AccessControl.AccessControlType]::Allow, `
  $computerObjectTypeGuid, `
  [System.DirectoryServices.ActiveDirectorySecurityInheritance]::All, `
  $inheritedObjectTypeNoneGuid)

$computerContainer = (Get-ADDomain).ComputersContainer
$domainJoinPath = "AD:\${computerContainer}"
$domainJoinAcl = Get-Acl $domainJoinPath
$domainJoinAcl.AddAccessRule($domainJoinRule)
Set-Acl -AclObject $domainJoinAcl -Path $domainJoinPath

Powershell doesn't natively support these granular ACL settings so we have to use verbose .NET objects.