Minecraft on Raspberry Pi cluster with metrics

minecraft-on-raspberry-pi.md

Minecraft on Raspberry Pi cluster with metrics

Ever wanted to put your Rapsberry Pi cluster to great use? Our team is working remotely, so we started to play Minecraft. I decided I would host the Minecraft server on my Raspberry Pi cluster. This gist will guide you through the steps I took to get a k3s cluster up with k3sup and later installed Minecraft as well as metrics exporter and Prometheus Operator

Why?

Quoniam Possumus - Because we can

You'll need

• Raspberry Pis to run kubernetes on. I used 3 Raspberry Pi 3 and one Raspberry Pi 4.

• A copy of the game Minecraft Java Edition so you can play on your server.

• Helm

Will it cluster?

Read about how you can install kubernetes to your Raspberry Pi in 15 minutes

Or visit https://k3sup.dev/ to setup your cluster.

Note, If you are using Raspberry Pi 4, consider using a 64 bit kernel to utilize more than 2 GB memory per process.

Taint and label the Minecraft node

To give the node where Minecraft is running access to the whole CPU and RAM you'll need to taint it so it only tolerates the Minecraft Server

kubectl taint nodes minecraft-node app=minecraft:NoSchedule

And to make sure that the minecraft installation will select the node it tolerates you'll also need to label the node.

kubectl label nodes minecraft-node role=minecraft

Install Minecraft Server

This is the helmchart that we will use https://github.com/helm/charts/tree/master/stable/minecraft

Minecraft Server comes in many flavours.

• To install Minecraft and later install Minecraft Prometheus Export plugin we need to use the SPIGOT distribution of Minecraft Server, you can also run PAPER. This is also running more lightweight than VANILLA Minecraft which is a perfect fit for Raspberry Pi.

• We need to make sure we are using the correct Minecraft Server image itzg/minecraft-server:multiarch

• Further more Minecraft needs to have the tolerations that we tainted the minecraft-node with and the nodeSelector

• Minecraft on Raspberry Pi takes longer time to start than on modern servers, the readyness and liveness probe is turned up a lot. If you have problems with restarts, try to increase the probes even more.

All of these settings are represented in the values file that will be used with the helm chart for Minecraft.

Create the values.yaml below to use with your helm installation.

kubectl create namespace minecraft
helm install --namespace minecraft minecraft -f values.yaml stable/minecraft

Checkpoint

Make sure Minecraft has started and is in Ready state

kubectl get pods -n minecraft

Lookup the IP address for your Minecraft Server.

kubectl get service -n minecraft

Use the IP of the loadbalancer to start playing right away. Or continue with metrics.

---

We're not done yet, metrics metrics metrics

Now we will install

• One plugin to expose metrics that can be scraped by Prometheus
• Prometheus Operator to get a full fledged monitoring system

Minecraft Prometheus Exporter

To get metrics scraped we need to install a Bukkit plugin for just this.

The plugin we are going to install can be found here https://github.com/sladkoff/minecraft-prometheus-exporter

This step requires manual labour inside the minecraft containers.

Get the latest release from minecraft-prometheus-exporter, it is the .jar file that you want to use. Copy that into the Minecraft pod's mounted volume, where the plugins should go.

kubectl cp minecraft-prometheus-exporter-<VERSION>.jar minecraft/minecraft-minecraft-<POD-ID>:/data/plugins

Next restart the pod by deleteing the pod, and it will restart again

We need to make one more change, make sure the exporter listens to all traffic. Exec into the pod and change the config.yml of the minecraft-prometheus-exporter, as described on the plugins README.md file.

# Note that the HTTP server binds to localhost by default.
# If your Prometheus runs on another host or inside a Kubernetes cluster 
# set this to any reachable IP or 0.0.0.0 to listen on all interfaces.
host: localhost
port: 9225

Connect to your pod:

kubectl exec -it -n minecraft minecraft-minecraft-<POD-ID> -- /bin/bash

Edit the config file as per our needs set host to 0.0.0.0 instead of localhost

nano plugins/PrometheusExporter/config.yml

host: 0.0.0.0
port: 9225
...

and exit the pod

exit

Now we need to expose the 9225 port from the deployment as well

kubectl patch deployment -n minecraft minecraft-minecraft --type='json' -p='[{"op":"add", "path": "/spec/template/spec/containers/0/ports/-", "value":{"containerPort":9225, "name":"metrics"}}]'

This adds a port entry with containerPort: 9225 and name: metrics.

The end result will look like this

        
        ...e: minecraft-minecraft
        ports:
        - containerPort: 25565
          name: minecraft
          protocol: TCP
        - containerPort: 9225
          name: metrics
          protocol: TCP
        readine...

Scrape Minecraft

Install the Prometheus Operator. To do this an ARM (Raspberry Pi architecture) I used https://github.com/uocxp/prometheus-operator-chart-arm.

Prometheus is using ServiceMonitors to scrape endpoints. We need to create a ServiceMonitor that let us scrape a port that we now produce metrics on.

Create a servicemonitor.yaml below.

kubectl apply -f servicemonitor.yaml

Note that the namespace of the service monitor should be the same as where you installed prometheus operator

• The label release: prometheus-operator needs to be there because the operator only listens to ServiceMonitors with that label by default.

• The targetPort is the one that we just created in the previous step and the app: minecraft-minecraft comes from the helm chart.

• The relabeling is to get the node as server_name on the dashboards.

Now you are ready to open up Grafana and install the two dashboards that can be found on the minecraft-prometheus-exporter repository.

kubectl port-forward -n monitor svc/prometheus-operator-grafana 3000:80

Open a browser to http://localhost:3000 and login with the default username and password for prometheus-operator admin prom-operator

Hit the + and paste the Server and the Player dashboards.

Enjoy!

servicemonitor.yaml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: minecraft-servicemonitor
  namespace: monitor
  labels:
    release: prometheus-operator
spec:
  endpoints:
    - interval: 10s
      path: /metrics
      relabelings:
      - sourceLabels: 
        - __meta_kubernetes_pod_node_name
        targetLabel: server_name
      targetPort: metrics
  namespaceSelector:
    matchNames:
      - minecraft
  selector:
    matchLabels:
      app: minecraft-minecraft

values.yaml

imageTag: multiarch

resources:
  requests: {}

tolerations:
  - key: "app"
    operator: "Equal"
    value: "minecraft"
    effect: "NoSchedule"

nodeSelector:
  role: minecraft

livenessProbe: 
  initialDelaySeconds: 120
  periodSeconds: 60

readinessProbe:
  initialDelaySeconds: 120
  periodSeconds: 60

minecraftServer:
  # This must be overridden, since we can't accept this for the user.
  eula: "TRUE"

  # One of: LATEST, SNAPSHOT, or a specific version (ie: "1.7.9").
  version: "1.15.2"
  # This can be one of "VANILLA", "FORGE", "SPIGOT", "BUKKIT", "PAPER", "FTB", "SPONGEVANILLA"
  type: "SPIGOT"

  # Max view distance (in chunks).
  viewDistance: 7

  # Message of the Day
  motd: "Welcome to Minecraft on Kubernetes on Rapsberry Pi!"

  # If you adjust this, you may need to adjust resources.requests above to match.
  # If you are running on raspberry pi 4 you can utilize a lot more
  memory: 512M

persistence:
  ## minecraft data Persistent Volume Storage Class
  ## If defined, storageClassName: <storageClass>
  ## If set to "-", storageClassName: "", which disables dynamic provisioning
  ## If undefined (the default) or set to null, no storageClassName spec is
  ##   set, choosing the default provisioner.  (gp2 on AWS, standard on
  ##   GKE, AWS & OpenStack)
  ##
  # storageClass: "-"
  dataDir:
    # Set this to false if you don't care to persist state between restarts.
    enabled: true
    Size: 10Gi

What im asking is if all the available memory will be used. Becuase from what i understand, minecraft wont use more than a specific amount becuase of the software being used.

The memory from one Raspberry will be used, as the process will only be running on one machine. And to make things even worse, if you are running a 32 bit operating system on your Raspberry (like raspbian lite) one process can only allocate 2 GB memory.

So to be able to utilize at least 7GB memory you will have to use a 64-bit operating system on the raspberry. The

minecraftServer: 
    memory: 512M

needs to change.

Max,
You did not respond to my last question. Related to pods that I have running but have no idea how to view the stats.

Do you have any idea?
Lawrence

@islamp I think we need more interactive way of communicating.

First you can portforward to your minecraft pod on port 9225 and visit the metrics page /metrics

Then you need to check if that is scraped by prometheus, but portforwarding to your prometheus-monitoring-kube... pod on port 9090 (usually). There you can see targets and service discovery to see if everything is setup correctly. Give me shout on twitter or similar and we can see if we can get everything sorted!

@gabrielsson I 100% agree. Do you have a preferred method of communication.

I am based in the Netherlands so in the same timezone.

below is my config file.

When I try to run the following. I get a duplication error.
kubectl patch deployment -n minecraft minecraft-minecraft --type='json' -p='[{"op":"add", "path": "/spec/template/spec/containers/0/ports/-", "value":{"containerPort":9225, "name":"metrics"}}]'

The Deployment "minecraft-minecraft" is invalid: spec.template.spec.containers[0].ports[2].name: Duplicate value: "metrics"

Lawrence

been experimenting with the bedrock-edition and helm but it seems that the java version also does not scale . :-( If i set replicas to 3 nothing happens . But 1 node than is running high cpu .. I think the pi-edition does support this.. Can you confirm ?

No, the minecraft process does not scale horizontally. It is merely a fun hosting challange than useful 😆 I am in the midst of updating my cluster... I will give this gist a re-wamp in that case.... this is very outdated now.

yeah I see that it's been quiet here for the last 3 years but it's still a great project ! That's the way I also see it , a nice challenge to host all of this . I managed to install longhorn and use my synology NAS for the PVC. As far as the metrics I want to use kube-state-metics and elastic-agent to monitor the cluster . combining work-stuff and hosting a custom bedrock map for my son is a great hobby .

Hello.
Please forgive my poor English.
How do I change it to make it in Forge version of Minecraft?