k0s Lab: loki, grafana, promtail

README.md

TLDR;

Setup Colima for running docker

colima start   # by default create 2CPU, 2GiB, 100GiB
colima delete
colima start --cpu 1 --memory 2 --disk 10
colima stop
colima delete
colima start --cpu 4 --memory 4 --disk 40 --vm-type=vz --vz-rosetta
docker ps
docker run --rm -it busybox

Setup k0s cluster on docker

docker run -d --name k0s --hostname k0s --privileged -v /Users/reed/work/k0s/k0s.yaml:/etc/k0s/k0s.yaml -v /var/lib/k0s -p 6443:6443 --cgroupns=host docker.io/k0sproject/k0s:v1.31.2-k0s.0 -- k0s controller --enable-worker -c /etc/k0s/k0s.yaml

token=$(docker exec -t -i k0s k0s token create --role=worker)

docker run -d --name k0s-wk1 --hostname k0s-wk1 --privileged -v /var/lib/k0s --cgroupns=host docker.io/k0sproject/k0s:v1.31.2-k0s.0 k0s worker $token

docker run -d --name k0s-wk2 --hostname k0s-wk2 --privileged -v /var/lib/k0s --cgroupns=host docker.io/k0sproject/k0s:v1.31.2-k0s.0 k0s worker $token

docker exec k0s k0s kubectl get all -A

alias k='docker exec k0s k0s kubectl'

k get nodes -A

Install Helm

sudo mkdir /usr/local/bin
sudo bash < <(curl -s -L https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3)

Setup kubeconfig

Then, set KUBECONFIG from k0s

docker exec k0s cat /var/lib/k0s/pki/admin.conf > ~/kubeconfig-k0s.yml
export KUBECONFIG=/Users/reed/kubeconfig-k0s.yml
kubectl cluster-info

Add grafana helm repo

helm repo list
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

Create persistent volume (PV)

kubectl apply -f /Users/reed/work/k0s/pv01.yaml
kubectl apply -f /Users/reed/work/k0s/pv02.yaml
kubectl apply -f /Users/reed/work/k0s/pv03.yaml

Install Loki, promtail, grafana

This loki helm chart will also install minio for its storage as part of loki installation.

helm install --values /Users/reed/work/k0s/loki-values.yaml loki grafana/loki

Install promtail

helm install promtail grafana/promtail

Install grafana

helm install grafana grafana/grafana

Loki gateway url is: http://loki-gateway.default.svc.cluster.local/

Verify if promtail installed correctly

kubectl port-forward daemonset/promtail 3101
curl localhost:3101/metrics

Verify if grafana installed correctly

kubectl get secret --namespace default grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo # show admin-password password

kubectl port-forward services/grafana 8000:80
# then login via browser to http://localhost:8000/login

Within the cluster, grafana hostname is grafana.default.svc.cluster.local.

To change/update loki setting, edit file /Users/reed/work/k0s/loki-values.yaml then do

helm upgrade --values /Users/reed/work/k0s/loki-values.yaml loki grafana/loki

k0s.yaml

apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
metadata:
  name: k0s
spec:
  api:
    address: 172.17.0.2
    k0sApiPort: 9443
    port: 6443
    sans:
    - 172.17.0.2
  controllerManager: {}
  extensions:
    helm:
      concurrencyLevel: 5
      repositories:
      - name: stable
        url: https://charts.helm.sh/stable
  installConfig:
    users:
      etcdUser: etcd
      kineUser: kube-apiserver
      konnectivityUser: konnectivity-server
      kubeAPIserverUser: kube-apiserver
      kubeSchedulerUser: kube-scheduler
  konnectivity:
    adminPort: 8133
    agentPort: 8132
  network:
    clusterDomain: cluster.local
    dualStack:
      enabled: false
    kubeProxy:
      iptables:
        minSyncPeriod: 0s
        syncPeriod: 0s
      ipvs:
        minSyncPeriod: 0s
        syncPeriod: 0s
        tcpFinTimeout: 0s
        tcpTimeout: 0s
        udpTimeout: 0s
      metricsBindAddress: 0.0.0.0:10249
      mode: iptables
      nftables:
        minSyncPeriod: 0s
        syncPeriod: 0s
    kuberouter:
      autoMTU: true
      hairpin: Enabled
      metricsPort: 8080
    nodeLocalLoadBalancing:
      enabled: false
      envoyProxy:
        apiServerBindPort: 7443
        konnectivityServerBindPort: 7132
      type: EnvoyProxy
    podCIDR: 10.244.0.0/16
    provider: kuberouter
    serviceCIDR: 10.96.0.0/12
  scheduler: {}
  storage:
    etcd:
      peerAddress: 172.17.0.2
    type: etcd
  telemetry:
    enabled: true

loki-values.yaml

loki:
  commonConfig:
    replication_factor: 1
    path_prefix: /tmp/loki
  schemaConfig:
    configs:
      - from: "2024-04-01"
        store: tsdb # boltdb
        object_store: filesystem
        schema: v13
        index:
          prefix: loki_index_
          period: 24h
  ingester:
    chunk_encoding: snappy
  tracing:
    enabled: false
  pattern_ingester:
      enabled: true
  limits_config:
    allow_structured_metadata: false
    volume_enabled: true
    retention_period: 30m # 672h # 28 days retention
  compactor:
    retention_enabled: true
    delete_request_store: s3
  ruler:
    enable_api: true
  storage_config:
    filesystem:
      directory: /tmp/loki
    boltdb_shipper:
      active_index_directory: /tmp/loki-boltdb-index
      cache_location: /tmp/loki-boltdb-cache
      cache_ttl: 24h
    tsdb_shipper:
      active_index_directory: /tmp/loki-tsdb-index
      cache_location: /tmp/loki-tsdb-cache
      cache_ttl: 24h


chunksCache:
  # -- Specifies whether memcached based chunks-cache should be enabled
  enabled: false
  # -- Amount of memory allocated to chunks-cache for object storage (in MB).
  #allocatedMemory: 512

minio:
  enabled: true
  persistence:
    storageClass: manual

deploymentMode: SingleBinary

singleBinary:
  replicas: 1
  persistence:
    storageClass: manual
  resources:
    limits:
      cpu: 1
      memory: 1Gi
    requests:
      cpu: 1
      memory: 100Mi

# Zero out replica counts of other deployment modes
backend:
  replicas: 0
read:
  replicas: 0
write:
  replicas: 0

ingester:
  replicas: 0
querier:
  replicas: 0
queryFrontend:
  replicas: 0
queryScheduler:
  replicas: 0
distributor:
  replicas: 0
compactor:
  replicas: 0
indexGateway:
  replicas: 0
bloomCompactor:
  replicas: 0
bloomGateway:
  replicas: 0

pv01.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-01
spec:
  capacity:
    storage: 20Gi  # Size of the volume
  accessModes:
    - ReadWriteOnce  # Access modes: ReadWriteOnce, ReadOnlyMany, or ReadWriteMany
  persistentVolumeReclaimPolicy: Delete  # Retain, Recycle, or Delete
  storageClassName: manual
  hostPath:
    path: /tmp/data/pv01  # Path to storage on the host (use only for testing)

pv02.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-02
spec:
  capacity:
    storage: 20Gi  # Size of the volume
  accessModes:
    - ReadWriteOnce  # Access modes: ReadWriteOnce, ReadOnlyMany, or ReadWriteMany
  persistentVolumeReclaimPolicy: Delete  # Retain, Recycle, or Delete
  storageClassName: manual
  hostPath:
    path: /tmp/data/pv02  # Path to storage on the host (use only for testing)

pv03.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-03
spec:
  capacity:
    storage: 20Gi  # Size of the volume
  accessModes:
    - ReadWriteOnce  # Access modes: ReadWriteOnce, ReadOnlyMany, or ReadWriteMany
  persistentVolumeReclaimPolicy: Delete  # Retain, Recycle, or Delete
  storageClassName: manual
  hostPath:
    path: /tmp/data/pv03  # Path to storage on the host (use only for testing)