Skip to content

Instantly share code, notes, and snippets.

@anfernee

anfernee/00-windows-note.md

Last active April 1, 2021 04:39
Show Gist options
  • Save anfernee/0f51f742ab4b74661dfb0774f1c35800 to your computer and use it in GitHub Desktop.
Save anfernee/0f51f742ab4b74661dfb0774f1c35800 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
00-windows-note.md
  • Install updates in windows
Install-Module PSWindowsUpdate
Get-Command -module PSWindowsUpdate 
# Add-WUServiceManager -MicrosoftUpdate 
Install-WindowsUpdate -MicrosoftUpdate -AcceptAll -AutoReboot 

## Add SSH
Get-WindowsCapability -Online | ? Name -like 'OpenSSH*'
Add-WindowsCapability -Online -Name OpenSSH.Client~~~~0.0.1.0
Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0

Start-Service sshd
Set-Service -Name sshd -StartupType 'Automatic'
Get-NetFirewallRule -Name *ssh*
# There should be a firewall rule named "OpenSSH-Server-In-TCP", which should be enabled
# If the firewall does not exist, create one
New-NetFirewallRule -Name sshd -DisplayName 'OpenSSH Server (sshd)' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 22

Useful commands:

https://cloud.google.com/compute/docs/instances/windows/creating-managing-windows-instances#gcloud

# Create instance
gcloud compute instances create win-1 --image-project windows-cloud --image-family windows-server-2019-dc-core-for-containers-v20210212 --machine-type e2-highcpu-16 --metadata=windows-startup-script-url=https://storage.googleapis.com/ygui-gke-dev/prepare-antrea.ps1
  • Folders
    • K8S_DIR: C:\etc\kubernetes
    • NODE_DIR: C:\etc\kubernetes\node\bin
Raw
01-win-ver.md
Version Latest. Golden version Issue
Windows 10 version 1809. 17763.1790 YES
Windows 10 version 1903. 18362.1256
Windows 10 version 1909. 18363.1379
Windows 10 version 2004. 19041.844 19042.844
Windows 10 version 20H2. 19041.844 19042.844
GKE LTSC 17763.1577
GKE SAC 18363.1198
GCE windows-server-2019-dc-core-for-containers-v20210212 17763.1757 YES
GCE windows-server-2019-dc-core-for-containers-v20210319 17763.1817
windows-server-2019-dc-core-v20210112 17763.1697 No hnsCall failed: The parameter is incorrect. (0x57)

(up to 3/4/2021) From: https://support.microsoft.com/en-us/topic/windows-10-update-history-24ea91f4-36e7-d8fd-0ddb-d79d9d0cdbda

Raw
10-antrea-note.md

Init process on windows:

  • setupOVSBridge
    • ovsClient.Create()
    • prepareOVSBridge()
    • initInterfaceStore()
    • setupDefaultTunnelInterface()
    • setupGateway()
      • createHostGatewayInterface() antrea-gw0
      • setMTU()
      • configureGatewayInterface()
        • util.setLinkUp (retried)
        • allocateGatewayAddress()
          • util.ConfigureLinkAddress
            • util.ConfigureInterfaceAddress()
              • Net-NetIPAddress ...
Raw
20-toolchain.md

Prepare toolchain

curl -LO https://chocolatey.org/install.ps1
.\install.ps1
choco install git
choco install make

build antrea

# in antrea folder
make build-windows DOCKER_IMG_VERSION=v0.13.0
Raw
errors.md 
2021/04/01 04:23:40 GCEMetadataScripts: windows-startup-script-ps1: +   Register-ScheduledJob -Name PrepareAntreaAgent -Trigger $trigger -S ...
2021/04/01 04:23:40 GCEMetadataScripts: windows-startup-script-ps1: +   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021/04/01 04:23:40 GCEMetadataScripts: windows-startup-script-ps1: Kubernetes Windows node setup failed: An error occurred while registering scheduled job definition PrepareAntreaAgent to the Windows Task Scheduler.  The Task Scheduler error is: (13,8):UserId:.
2021/04/01 04:23:40 GCEMetadataScripts: windows-startup-script-ps1 exit status 1
Raw
linux-boot.sh
VERSION=v0.13.0
kubectl apply -f https://github.com/vmware-tanzu/antrea/releases/download/$VERSION/antrea.yml
kubectl apply -f https://github.com/vmware-tanzu/antrea/releases/download/$VERSION/antrea-windows.yml
kubectl apply -f - <<EOF
apiVersion: v1
data:
run-script.ps1: |-
$ErrorActionPreference = "Stop";
mkdir -force /host/var/lib/kube-proxy/var/run/secrets/kubernetes.io/serviceaccount
mkdir -force /host/k/kube-proxy
cp -force /k/kube-proxy/* /host/k/kube-proxy
cp -force /var/lib/kube-proxy/* /host/var/lib/kube-proxy
cp -force /var/run/secrets/kubernetes.io/serviceaccount/* /host/var/lib/kube-proxy/var/run/secrets/kubernetes.io/serviceaccount
wins cli process run --path /k/kube-proxy/kube-proxy.exe --args "--v=3 --config=/var/lib/kube-proxy/config.conf --proxy-mode=userspace --hostname-override=$env:NODE_NAME"
kind: ConfigMap
apiVersion: v1
metadata:
labels:
app: kube-proxy
name: kube-proxy-windows
namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
k8s-app: kube-proxy
name: kube-proxy-windows
namespace: kube-system
spec:
selector:
matchLabels:
k8s-app: kube-proxy-windows
template:
metadata:
labels:
k8s-app: kube-proxy-windows
spec:
hostNetwork: true
serviceAccountName: kube-proxy
containers:
- command:
- pwsh
args:
- -file
- /var/lib/kube-proxy-windows/run-script.ps1
env:
- name: NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
image: sigwindowstools/kube-proxy:v1.20.0-nanoserver
name: kube-proxy
volumeMounts:
- name: host
mountPath: /host
- name: wins
mountPath: \\.\pipe\rancher_wins
- mountPath: /var/lib/kube-proxy
name: kube-proxy
- mountPath: /var/lib/kube-proxy-windows
name: kube-proxy-windows
nodeSelector:
kubernetes.io/os: windows
tolerations:
- key: CriticalAddonsOnly
operator: Exists
- operator: Exists
volumes:
- configMap:
defaultMode: 420
name: kube-proxy-windows
name: kube-proxy-windows
- configMap:
name: kube-proxy
name: kube-proxy
- hostPath:
path: /
name: host
- name: wins
hostPath:
path: \\.\pipe\rancher_wins
type: null
updateStrategy:
type: RollingUpdate
EOF
Raw
prepare-antrea.ps1
# https://storage.googleapis.com/ygui-gke-dev/prepare-antrea.ps1
# windows metadata-key: windows-startup-script-url
function Enable-HyperV {
Install-WindowsFeature containers
Install-WindowsFeature Hyper-V-Powershell
dism /online /enable-feature /featurename:Microsoft-Hyper-V /all /NoRestart
dism /online /disable-feature /featurename:Microsoft-Hyper-V-Online /NoRestart
}
function Enable-OpenSSH {
Add-WindowsCapability -Online -Name OpenSSH.Client~~~~0.0.1.0
Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
# Change server start-up to Automatic
Set-Service -Name sshd -StartupType 'Automatic'
Start-Service sshd
}
function Install-OVS {
curl.exe -LO https://raw.githubusercontent.com/vmware-tanzu/antrea/main/hack/windows/Install-OVS.ps1
.\Install-OVS.ps1
}
function Prepare-Antrea {
mkdir c:\k\antrea
cd c:\k\antrea
curl.exe -LO https://raw.githubusercontent.com/vmware-tanzu/antrea/main/hack/windows/Clean-AntreaNetwork.ps1
curl.exe -LO https://raw.githubusercontent.com/vmware-tanzu/antrea/main/hack/windows/Prepare-ServiceInterface.ps1
curl.exe -LO https://raw.githubusercontent.com/vmware-tanzu/antrea/main/hack/windows/Prepare-AntreaAgent.ps1
.\Prepare-AntreaAgent.ps1
}
function Prepare-Kube {
curl.exe -LO https://github.com/kubernetes-sigs/sig-windows-tools/releases/latest/download/PrepareNode.ps1
.\PrepareNode.ps1 -KubernetesVersion v1.20.0
}
Enable-HyperV
Enable-OpenSSH
Bcdedit.exe -set TESTSIGNING ON
Restart-Computer
Install-OVS
Prepare-Kube
Prepare-Antrea
Raw
startup.md

Agent start script

    $ErrorActionPreference = "Stop"
    # wins will rename the binary when executing it. So we need to copy the binary everytime before running it.
    mkdir -force /host/k/antrea/bin
    cp /k/antrea/bin/* /host/k/antrea/bin/
    C:/k/antrea/utils/wins.exe cli process run --path /k/antrea/bin/antrea-agent.exe --args "--config=/k/antrea/etc/antrea-agent.conf --logtostderr=false --log_dir=/k/antrea/logs/ --alsologtostderr --log_file_max_size=100 --log_file_max_num=4" --envs "KUBERNETES_SERVICE_HOST=$env:KUBERNETES_SERVICE_HOST KUBERNETES_SERVICE_PORT=$env:KUBERNETES_SERVICE_PORT ANTREA_SERVICE_HOST=$env:ANTREA_SERVICE_HOST ANTREA_SERVICE_PORT=$env:ANTREA_SERVICE_PORT NODE_NAME=$env:NODE_NAME"

kubelet.exe

commandline : "C:\k\kubelet.exe" --network-plugin=cni --pod-infra-container-image=mcr.microsoft.com/oss/kubernetes/pause:1.4.0 --node-ip=21.0.88.220 --cert-dir=C:\var\lib\kubelet\pki --config=/var/lib/kubelet/config.yaml        
              --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --hostname-override=WIN-ULO7CKR86CU --pod-infra-container-image=mcr.microsoft.com/oss/kubernetes/pause:1.4.1  
              --enable-debugging-handlers --cgroups-per-qos=false --enforce-node-allocatable= --network-plugin=cni --resolv-conf= --log-dir=/var/log/kubelet --logtostderr=false --image-pull-progress-deadline=20m

kube-proxy

  run-script.ps1: |-
    $ErrorActionPreference = "Stop";
    mkdir -force /host/var/lib/kube-proxy/var/run/secrets/kubernetes.io/serviceaccount
    mkdir -force /host/k/kube-proxy

    cp -force /k/kube-proxy/* /host/k/kube-proxy
    cp -force /var/lib/kube-proxy/* /host/var/lib/kube-proxy
    cp -force /var/run/secrets/kubernetes.io/serviceaccount/* /host/var/lib/kube-proxy/var/run/secrets/kubernetes.io/serviceaccount

    wins cli process run --path /k/kube-proxy/kube-proxy.exe --args "--v=3 --config=/var/lib/kube-proxy/config.conf --proxy-mode=userspace --hostname-override=$env:NODE_NAME"

CNI binary path

PS C:\opt\cni\bin> ls


    Directory: C:\opt\cni\bin


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----        2/11/2021  10:36 PM       12300288 antrea.exe
-a----         6/5/2019   9:55 AM        3333632 host-local.exe


> cat antrea-agent.conf
> $env:NODE_NAME = "win-xxx"  # lowercase it
> antrea-agent.exe --config=antrea-agent.conf
trafficEncapMode: noEncap
clientConnection:
  kubeconfig: \kubeconfig
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment