kubernetes_in_khadas.md

说明

我买的Khadas型号是VIM1S。它的用户手册在官网产品的购买界面。这里可以找到自带的OOWOW嵌入式系统的文档。

这里介绍一下安装Kubernetes的过程。

Specification
Model	VIM1S	VIM3L	VIM3
SOC	Amlogic S905Y4 SoC	Amlogic S905D3 SoC	Amlogic A311D SoC
内存	LPDDR4 2GB	LPDDR4X 2GB	LPDDR4X 2GB
存储	eMMC 5.0 16GB	eMMC 5.1 16GB	eMMC 5.1 16GB
蓝牙	5.0	5.0	5.0
WIFI	2.4G/5G	2.4G/5G	2.4G/5G
电源适配器(最好)	5V/2A	12V/2A	12V/2A

修改登录密码

Khadas官方ubuntu镜像安装成功后，默认的用户名和密码都是khadas。第一次SSH登录时最好修改默认密码。

# ssh khadas@192.168.x.xx
~$ passwd
# Changing password for khadas.
# Current password: 
# New password: 
# Retype new password: 

安装MicroSD卡

我加了一个64GB的Micro SD卡，所以第一步是mount。参考Mount Micro SD Card Ubuntu

~$ lsblk
# => 其中mmcblk0是产品自带的那16G存储空间，当然只有14.6G可用。
# => 其中mmcblk1就是这个SD卡，mmcblk1p1指的是卡上的分区。

# NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
# mmcblk0      179:0    0  14.6G  0 disk 
# ├─mmcblk0p1  179:1    0   240M  0 part /boot
# └─mmcblk0p2  179:2    0  14.2G  0 part /
# mmcblk0boot0 179:32   0     4M  0 disk 
# mmcblk0boot1 179:64   0     4M  0 disk 
# mmcblk1      179:96   0  59.5G  0 disk 
# └─mmcblk1p1  179:97   0  59.5G  0 part 
# zram0        253:0    0     0B  0 disk 
# zram1        253:1    0 248.6M  0 disk [SWAP]
# zram2        253:2    0 248.6M  0 disk [SWAP]
# zram3        253:3    0 248.6M  0 disk [SWAP]
# zram4        253:4    0 248.6M  0 disk [SWAP]
# zram5        253:5    0     0B  0 disk 

# 查看分区格式
~$ lsblk -f
# NAME         FSTYPE FSVER LABEL  UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
# mmcblk1                                                                              
# ├─mmcblk1p1  vfat   FAT32 esp    492E-xxxx                                           
# ├─mmcblk1p2  vfat   FAT32 efi    4934-xxxx                                           
# ├─mmcblk1p3  btrfs        rootfs 304d3526-c99e-45e9-81b5-xxxxxxxxxxxx                
# ├─mmcblk1p4  ext4   1.0   var    17d46966-cb62-416d-8c9d-xxxxxxxxxxxx                
# └─mmcblk1p5  ext4   1.0   home   952128a7-3815-484e-9fe9-xxxxxxxxxxxx                
# ...                                                                     

# 建立分区，再格式化成exfat格式（别忘记备份资料），
~$ sudo apt update
~$ sudo apt install exfatprogs
~$ sudo fdisk /dev/mmcblk1
# step 1: Press `n` to create a new partition
# step 2: Press `p` for primary.
# step 3: Choose the partition number(e.g. 1).
# step 4: Accept the default start and end sectors to use the entire disk.
# step 5: Press `w` to write the changes and exit.
~$ sudo mkfs.exfat -n SanDisk /dev/mmcblk1p1      # 其中SanDisk只是一个名字，可以任意指定。

~$ sudo mkdir /media/microsd
# 1000 是用户ID(当前使用的khadas用户)，可以用`id -u`获得。
# umask=002，指的是用户ID为1000的用户和admin有写权限；其他人有只读权限。
~$ sudo mount -o uid=1000,umask=002 /dev/mmcblk1p1 /media/microsd

# 查看分区的文件系统
~$ df -T
# =>
# Filesystem     Type  1K-blocks    Used Available Use% Mounted on
# tmpfs          tmpfs    203656   14392    189264   8% /run
# /dev/mmcblk0p2 ext4   14555288 4533428   9811992  32% /
# tmpfs          tmpfs   1018272       0   1018272   0% /dev/shm
# tmpfs          tmpfs      5120       4      5116   1% /run/lock
# tmpfs          tmpfs   1018272       0   1018272   0% /tmp
# /dev/mmcblk0p1 ext4     221664   41116    170724  20% /boot
# tmpfs          tmpfs    203652      68    203584   1% /run/user/1000
# /dev/mmcblk1p1 exfat  62334976   11136  62323840   1% /media/microsd

WiFi network

NetworkManager是一个网络管理服务，用于简化计算机系统的网络连接管理。可以使用nmcli命令和NetworkManager进行交互。

~$ nmcli d 
# => 
# DEVICE         TYPE      STATE         CONNECTION         
# eth0           ethernet  connected     Wired connection 1 
# wlan0          wifi      disconnected  --                 
# wlan1          wifi      disconnected  --                 
# dummy0         dummy     unmanaged     --                 
# ip6tnl0        iptunnel  unmanaged     --                 
# lo             loopback  unmanaged     --                 
# ip_vti0        vti       unmanaged     --                 
# p2p-dev-wlan0  wifi-p2p  unmanaged     --                 
# p2p-dev-wlan1  wifi-p2p  unmanaged     --                 

# check WiFi radio status
~$ nmcli r wifi

# 显示可用的WiFi网络
~$ nmcli d wifi list
# =>
# IN-USE  BSSID              SSID  MODE   CHAN  RATE        SIGNAL  BARS  SECURITY 
#         B4:xx:xx:xx:xx:7C  X&Y   Infra  149   270 Mbit/s  72      ▂▄▆_  WPA2     
#         B4:xx:xx:xx:xx:78  X&Y   Infra  11    270 Mbit/s  62      ▂▄▆_  WPA2     

# 连接到WiFi X&Y (X&Y是我家WiFi的名字，可以使用单引号来转译特殊字符)
~$ sudo nmcli d wifi connect 'X&Y' password '<password>'
# => 
# Device 'wlan0' successfully activated with 'ff2579be-81e6-xxxx-xxxx-xxxxxxxxxxxx'.

更改hostname，以便后续在路由器上区分。

~$ hostname          # => check current hostname
~$ sudo hostnamectl set-hostname khadas3     # => update hostname to khadas3

SELinux (has problem)

系统应该是默认安装了SELinux，因为/sys/fs/selinux和/etc/selinux两个目录确实存在。具体可以参见官方selinux-notebook文档。

/sys/fs/selinux: The SELinux filesystem that interfaces with the kernel based security server. The new location has been available since Fedora 17.

/etc/selinux: The SELinux configuration directory that holds the sub-system configuration files and policies.

/var/lib/selinux/<SELINUXTYPE>: The SELinux policy store that holds policy modules and configuration details. The new location has been available since Fedora 23.

可以使用getenforce或sestatus命令来查看SELinux是否处于开启状态。但是系统默认并没有安装这两个命令。下面是安装过程和简单的使用方式。

# check SELinux status
~$ sudo apt update
~$ sudo apt install selinux-utils
~$ getenforce
# =>
# Disabled
#
~$ sudo apt install policycoreutils
~$ sestatus
# 
# 修改/etc/selinux/config
# 将
# SELINUX=enforcing
# 修改为
# SELINUX=disabled

～$ reboot

安装MicroK8s

首先安装snap包管理系统。snap相比于apt的优势是它是自包含的(self-contained)，即运行所需的所有库文件都存放在安装目录中，独立于系统存在。由此可以避免当两个应用程序依赖于同一个系统文件时导致的版本兼容问题。

~$ sudo apt install snapd
~$ snap version

其次，使用microk8s构建一个小型的Kubernetes系统。可以参见Ubuntu microk8s的官方文档。

~$ sudo snap install microk8s --classic
~$ microk8s version

# 查看micro8s的运行状态，这里显示默认状态下dns, ha-cluster, helm, helm3插件是启动的。
~$ sudo microk8s status --wait-ready
# =>
# 2024/01/28 08:00:40.554644 cmd_run.go:1046: WARNING: cannot create user data directory: failed to verify SELinux context of /root/snap: exec: "matchpathcon": # executable file not found in $PATH
# microk8s is running
# high-availability: no
#   datastore master nodes: 127.0.0.1:19001
#   datastore standby nodes: none
# addons:
#   enabled:
#     dns                  # (core) CoreDNS
#     ha-cluster           # (core) Configure high availability on the current node
#     helm                 # (core) Helm - the package manager for Kubernetes
#     helm3                # (core) Helm 3 - the package manager for Kubernetes
#   disabled:
#     cert-manager         # (core) Cloud native certificate management
#     cis-hardening        # (core) Apply CIS K8s hardening
#     community            # (core) The community addons repository
#     dashboard            # (core) The Kubernetes dashboard
#     host-access          # (core) Allow Pods connecting to Host services smoothly
#     hostpath-storage     # (core) Storage class; allocates storage from host directory
#     ingress              # (core) Ingress controller for external access
#     kube-ovn             # (core) An advanced network fabric for Kubernetes
#     mayastor             # (core) OpenEBS MayaStor
#     metallb              # (core) Loadbalancer for your Kubernetes cluster
#     metrics-server       # (core) K8s Metrics Server for API access to service metrics
#     minio                # (core) MinIO object storage
#     observability        # (core) A lightweight observability stack for logs, traces and metrics
#     prometheus           # (core) Prometheus operator for monitoring and logging
#     rbac                 # (core) Role-Based Access Control for authorisation
#     registry             # (core) Private image registry exposed on localhost:32000
#     rook-ceph            # (core) Distributed Ceph storage using Rook
#     storage              # (core) Alias to hostpath-storage add-on, deprecated

# 将khadas用户加入到管理microk8s的用户组中，这样就不需要每次使用`microk8s`命令时加上`sudo`了。
# 需要退出shell并重新连接使其生效。
~$ sudo usermod -a -G microk8s khadas
# 开启dashboard插件。
~$ microk8s enable dashboard
# 启动dashboard
~$ microk8s dashboard-proxy
# => 
# 2024/01/28 08:26:35.117767 cmd_run.go:1046: WARNING: cannot create user data directory: failed to verify SELinux context of /home/khadas/snap: exec: "matchpathcon": executable file not found in $PATH
# Checking if Dashboard is running.
# Infer repository core for addon dashboard
# Waiting for Dashboard to come up.
# Trying to get token from microk8s-dashboard-token
# Waiting for secret token (attempt 0)
# Dashboard will be available at https://127.0.0.1:10443
# Use the following token to login:
# xxxx

按照提示，你就可以用另一台局域网内的电脑，通过网络访问到Kubernetes，并使用提供的token登录UI。

最后，安装并使用kubectl来管理kubernetes。

# 在microk8s的机器上打输出config的内容
~$ microk8s config
# 将输出的内容复制并拷贝到另一台机器的.kube/config中
# 修改config里面的IP，用免费的域名替代

~$ kubectl config set-cluster artra_kubernetes --server=https://192.168.3.73:16443 --insecure-skip-tls-verify=true
~$ kubectl config get-clusters
# =>
# NAME
# artra_kubernetes

# 看能否成功获取pod信息，以此来判断kubectl是否连接正确。
~$ kubectl get pod

配置集群cluster

MicroK8s的官方文档在建立多个节点的集群这块已经说的很好了，可以参见Create a MicroK8s cluster。

# 使用如下命令查看集群节点具体运行的service
~$ kubectl get pods -A -o wide
# NAMESPACE     NAME                                         READY   STATUS    RESTARTS         AGE     IP              NODE      NOMINATED NODE   READINESS GATES
# ingress       nginx-ingress-microk8s-controller-7k4ns      1/1     Running   0                3m26s   10.1.120.44     khadas2   <none>           <none>
# ingress       nginx-ingress-microk8s-controller-zvklh      1/1     Running   0                3m26s   10.1.126.193    khadas1   <none>           <none>
# kube-system   calico-kube-controllers-77bd7c5b-sxqrs       1/1     Running   2065 (58m ago)   197d    10.1.120.52     khadas2   <none>           <none>
# kube-system   calico-node-sqcwq                            1/1     Running   1 (58m ago)      60m     192.168.3.190   khadas2   <none>           <none>
# kube-system   calico-node-xkhvc                            0/1     Running   0                49m     192.168.3.189   khadas1   <none>           <none>
# kube-system   coredns-864597b5fd-w6ssl                     1/1     Running   1888 (58m ago)   197d    10.1.120.42     khadas2   <none>           <none>
# kube-system   dashboard-metrics-scraper-5657497c4c-kq9hl   1/1     Running   3 (58m ago)      17h     10.1.120.48     khadas2   <none>           <none>
# kube-system   kubernetes-dashboard-54b48fbf9-pj6t7         1/1     Running   3 (58m ago)      17h     10.1.120.41     khadas2   <none>           <none>
# kube-system   metrics-server-6d484c6d7d-qmgbj              1/1     Running   3 (58m ago)      17h     10.1.120.39     khadas2   <none>           <none>

Certificate的问题

具体参见stack overflow

# 将免费域名加入到microk8s的配置中
# 编辑/var/snap/microk8s/current/certs/csr.conf.template文件，加入所需的DNS

[ alt_names ]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = kubernetes.default.svc
DNS.4 = kubernetes.default.svc.cluster
DNS.5 = kubernetes.default.svc.cluster.local
DNS.6 = your.dns.com
IP.1 = 127.0.0.1
IP.2 = ...

DDNS和域名

1. 华为路由器的端口映射。
2. 注册dynu，获取免费域名。
3. 配置kubectl。

# 拷贝配置文件到本地
~$ microk8s config > config

# 在另外一台电脑：
# 修改config里面的IP，用免费的域名替代
# 这个要弄清证书信任的问题。
~$ kubectl cluster-info --insecure-skip-tls-verify

Notes

1. Boot OOWOW - hold FUNCTION and short press RESET.
2. 在OOWOW中还是连不上Huawei wifi 6；需要用到手机的热点。后面发现连不上的原因是我的Wifi名字里面有特殊字符，比如X&Y，&就是特殊字符。我家里的东芝的洗衣机也有这个问题。
3. 当Kubernetes搭建好后，会有以下端口。
   • kubectl的管理端口：16443。
   • UI界面的端口：10443.
4. Batocera v33有游戏保存跨设备同步功能，用syncthing实现的；可以考虑在建好的k8s中设置这个服务。

建立第一个Hello World服务

建立第一个deployment和对应的service。这里使用hashicorp/http-echo这个docker image。当你通过HTTP访问它的默认端口5678时，可以在浏览器上看到Hello World的字样。

生成http-echo.yaml文件如下

apiVersion: apps/v1
kind: Deployment
metadata:
  name: http-echo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: http-echo
  template:
    metadata:
      labels:
        app: http-echo
    spec:
      containers:
      - name: http-echo
        image: hashicorp/http-echo
        args: ["-text=Hello World!"]
---
apiVersion: v1
kind: Service
metadata:
  name: http-echo-service
spec:
  selector:
    app: http-echo
  ports:
    - protocol: TCP
      port: 5678
      targetPort: 5678

然后执行：

~$ kubectl apply -f http-echo.yaml

配置Ingress

生成ingress-rule.yaml文件如下：

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-routes
spec:
  rules:
  - http:
      paths:
      - path: /helloworld
        pathType: Prefix
        backend:
          service:
            name: http-echo-service
            port:
              number: 5678

然后执行：

~$ kubectl apply -f ingress-rule.yaml

这样你就可以通过浏览器访问MicroK8s cluster所在的IP。

~$ curl 192.168.x.x/helloworld
# => Hello World!

也可以通过浏览器打开xxx.freeddns.org/helloworld

动态DNS解析

https://www.dynu.com/

写一个小程序，每10分钟运行一次：

1. 查看自己的wan ip，参考 getting machine external IP.
2. 使用dynu提供的API来更新WAN口的IP地址。

建立Rust工程

# `Rust`版本信息.
~$ rustup --version
# rustup 1.27.1 (54dd3d00f 2024-04-24)
# info: This is the version for the rustup toolchain manager, not the rustc compiler.
# info: The currently active `rustc` version is `rustc 1.84.0 (9fc6b4312 2025-01-07)`

# initialize project
~$ cargo new dns_updater
~$ cd dns_updater

# 使用下面的代码覆盖`dns_updater/src/main.rs`。

# resolve dependency
dns_updater ~$ cargo add reqwest --features json
dns_updater ~$ cargo add tokio --feature full
dns_updater ~$ cargo add futures
dns_updater ~$ cargo add serde --feature derive
dns_updater ~$ cargo add serde_json

代码.

use reqwest;
use serde::{Deserialize, Serialize};
use serde_json::json;
use std::env;

static DDNS_DOMAIN: &str = "xxxxxx.freeddns.org";

#[derive(Deserialize, Serialize, Debug)]
struct DomainListResponse {
    #[serde(rename = "statusCode")]
    status_code: u16,
    domains: Vec<Domain>,
}

#[derive(Deserialize, Serialize, Debug)]
struct Domain {
    id: u64,
    name: String,
    #[serde(rename = "ipv4Address")]
    ipv4_address: String,
}

#[derive(Deserialize, Serialize, Debug)]
struct DomainResponse {
    #[serde(rename = "statusCode")]
    status_code: u16,
    id: u64,
    name: String,
    #[serde(rename = "ipv4Address")]
    ipv4_address: String,
}

#[tokio::main]
async fn main() {
    let api_key = env::var("API_KEY").unwrap_or_else(|_| {
        eprintln!("API_KEY must be set in environment variable");
        std::process::exit(1);
    });
    let current_ip = get_current_ip().await.trim().to_string();
    println!("current ip is {}", current_ip);
    let entry_id = get_entry_id(api_key.to_string()).await;
    if let None = entry_id {
        println!("cannot find entry whose name is {}", DDNS_DOMAIN);
    } else {
        let ddns_ip = get_ddns_ip(api_key.to_string(), entry_id.unwrap()).await;
        println!("ddns ip is {}", ddns_ip);
        if current_ip != ddns_ip {
            println!("Update action needed as Current IP and DDNS IP are different.");
            update_ddns_ip(api_key.to_string(), entry_id.unwrap(), current_ip).await;
        } else {
            println!("No action needed. Current IP and DDNS IP are the same.");
        }
    }
}

async fn get_current_ip() -> String {
    reqwest::get("https://checkip.amazonaws.com")
        .await
        .unwrap()
        .text()
        .await
        .unwrap()
}

async fn get_entry_id(api_key: String) -> Option<u64> {
    let url = "https://api.dynu.com/v2/dns";
    let client = reqwest::Client::new();
    let response = client
        .get(url)
        .header("Content-Type", "application/json")
        .header("API-Key", api_key)
        .send()
        .await
        .unwrap()
        .text()
        .await
        .unwrap();
    let response: DomainListResponse = serde_json::from_str(&response).unwrap();
    if let Some(domain) = response.domains.iter().find(|d| d.name == DDNS_DOMAIN) {
        Some(domain.id)
    } else {
        None
    }
}

async fn get_ddns_ip(api_key: String, entry_id: u64) -> String {
    let url = format!("https://api.dynu.com/v2/dns/{}", entry_id);
    let client = reqwest::Client::new();
    let response = client
        .get(url)
        .header("Content-Type", "application/json")
        .header("API-Key", api_key)
        .send()
        .await
        .unwrap()
        .text()
        .await
        .unwrap();
    let domain_response: DomainResponse = serde_json::from_str(&response).unwrap();
    domain_response.ipv4_address
}

async fn update_ddns_ip(api_key: String, entry_id: u64, ipv4: String) {
    let url = format!("https://api.dynu.com/v2/dns/{}", entry_id);
    let payload = json!({
        "name": DDNS_DOMAIN,
        "group": "",
        "ipv4Address": ipv4,
        "ipv6Address": null,
        "ttl": 90,
        "ipv4": true,
        "ipv6": true,
        "ipv4WildcardAlias": true,
        "ipv6WildcardAlias": true
    });
    let client = reqwest::Client::new();
    let response = client
        .post(url)
        .header("Content-Type", "application/json")
        .header("API-Key", api_key)
        .json(&payload)
        .send()
        .await;
    println!("{:?}", response);
}

Dockerfile

FROM rust:1.84 AS build

COPY ./ ./
RUN cargo build --release

# ----- -----
# Can NOT use debian:bookworm-slim
# As it will cause the following error:
# dns_updater: error while loading shared libraries: libssl.so.3
FROM rust:1.84-slim-bookworm
COPY --from=build /target/release/dns_updater /usr/local/bin/dns_updater
CMD ["dns_updater"]

运行.

~$ docker build -t dns_updater .
~$ docker run --env-file .env  --rm dns_updater

Cronjob in k8s

~$ kubectl create secret docker-registry 'docker-hub' --docker-username='keegoo' --docker-password='docker hub token' --insecure-skip-tls-verify
~$ kubectl get secrets --insecure-skip-tls-verify

使用kubectl管理kubernetes

由于下面两个原因，我可以使用kubectl通过外网访问到我家里的Kubernetes。

1. 我已经在路由器端，将Kubernetes的管理端口: 16443，和UI端口: 10443配置了端口映射
2. 将路由器的WAN口IP绑定了Dynu上的免费Domain

下面是一些基本的信息。

~$ kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}'
# => https://xxxxxx.freeddns.org:16443

~$ kubectl config get-contexts
# CURRENT   NAME       CLUSTER            AUTHINFO   NAMESPACE
# *         microk8s   microk8s-cluster   admin      

~$ kubectl get nodes
# NAME      STATUS   ROLES    AGE    VERSION
# khadas1   Ready    <none>   209d   v1.28.15
# khadas2   Ready    <none>   406d   v1.29.15

~$ kubectl get namespaces
# default           Active   406d
# ingress           Active   209d
# kube-node-lease   Active   406d
# kube-public       Active   406d
# kube-system       Active   406d

搭建Jupyter Notebook

~$ kubectl create namespace jupyter-notebook

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jupyter-notebook
  namespace: jupyter-notebook
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jupyter-notebook
  template:
    metadata:
      labels:
        app: jupyter-notebook
    spec:
      containers:
      - name: jupyter-notebook
        image: jupyter/base-notebook:latest
        ports:
        - containerPort: 8888
        env:
        - name: JUPYTER_TOKEN
          value: "xuer1231"