Skip to content

Instantly share code, notes, and snippets.

@gabrielsson
Last active September 29, 2024 15:03
Show Gist options
  • Save gabrielsson/2d110bb3f43b46597831f4a0e4065265 to your computer and use it in GitHub Desktop.
Save gabrielsson/2d110bb3f43b46597831f4a0e4065265 to your computer and use it in GitHub Desktop.
Minecraft on Raspberry Pi cluster with metrics

alt

Minecraft on Raspberry Pi cluster with metrics

Ever wanted to put your Rapsberry Pi cluster to great use? Our team is working remotely, so we started to play Minecraft. I decided I would host the Minecraft server on my Raspberry Pi cluster. This gist will guide you through the steps I took to get a k3s cluster up with k3sup and later installed Minecraft as well as metrics exporter and Prometheus Operator

Why?

Quoniam Possumus - Because we can

You'll need

  • Raspberry Pis to run kubernetes on. I used 3 Raspberry Pi 3 and one Raspberry Pi 4.

  • A copy of the game Minecraft Java Edition so you can play on your server.

  • Helm

Will it cluster?

Read about how you can install kubernetes to your Raspberry Pi in 15 minutes

Or visit https://k3sup.dev/ to setup your cluster.

Note, If you are using Raspberry Pi 4, consider using a 64 bit kernel to utilize more than 2 GB memory per process.

Taint and label the Minecraft node

To give the node where Minecraft is running access to the whole CPU and RAM you'll need to taint it so it only tolerates the Minecraft Server

kubectl taint nodes minecraft-node app=minecraft:NoSchedule

And to make sure that the minecraft installation will select the node it tolerates you'll also need to label the node.

kubectl label nodes minecraft-node role=minecraft

Install Minecraft Server

This is the helmchart that we will use https://github.com/helm/charts/tree/master/stable/minecraft

Minecraft Server comes in many flavours.

  • To install Minecraft and later install Minecraft Prometheus Export plugin we need to use the SPIGOT distribution of Minecraft Server, you can also run PAPER. This is also running more lightweight than VANILLA Minecraft which is a perfect fit for Raspberry Pi.

  • We need to make sure we are using the correct Minecraft Server image itzg/minecraft-server:multiarch

  • Further more Minecraft needs to have the tolerations that we tainted the minecraft-node with and the nodeSelector

  • Minecraft on Raspberry Pi takes longer time to start than on modern servers, the readyness and liveness probe is turned up a lot. If you have problems with restarts, try to increase the probes even more.

All of these settings are represented in the values file that will be used with the helm chart for Minecraft.

Create the values.yaml below to use with your helm installation.

kubectl create namespace minecraft
helm install --namespace minecraft minecraft -f values.yaml stable/minecraft

Checkpoint

Make sure Minecraft has started and is in Ready state

kubectl get pods -n minecraft

Lookup the IP address for your Minecraft Server.

kubectl get service -n minecraft

Use the IP of the loadbalancer to start playing right away. Or continue with metrics.


We're not done yet, metrics metrics metrics

Now we will install

  • One plugin to expose metrics that can be scraped by Prometheus
  • Prometheus Operator to get a full fledged monitoring system

To get metrics scraped we need to install a Bukkit plugin for just this.

The plugin we are going to install can be found here https://github.com/sladkoff/minecraft-prometheus-exporter

This step requires manual labour inside the minecraft containers.

Get the latest release from minecraft-prometheus-exporter, it is the .jar file that you want to use. Copy that into the Minecraft pod's mounted volume, where the plugins should go.

kubectl cp minecraft-prometheus-exporter-<VERSION>.jar minecraft/minecraft-minecraft-<POD-ID>:/data/plugins

Next restart the pod by deleteing the pod, and it will restart again

We need to make one more change, make sure the exporter listens to all traffic. Exec into the pod and change the config.yml of the minecraft-prometheus-exporter, as described on the plugins README.md file.

# Note that the HTTP server binds to localhost by default.
# If your Prometheus runs on another host or inside a Kubernetes cluster 
# set this to any reachable IP or 0.0.0.0 to listen on all interfaces.
host: localhost
port: 9225

Connect to your pod:

kubectl exec -it -n minecraft minecraft-minecraft-<POD-ID> -- /bin/bash

Edit the config file as per our needs set host to 0.0.0.0 instead of localhost

nano plugins/PrometheusExporter/config.yml

host: 0.0.0.0
port: 9225
...

and exit the pod

exit

Now we need to expose the 9225 port from the deployment as well

kubectl patch deployment -n minecraft minecraft-minecraft --type='json' -p='[{"op":"add", "path": "/spec/template/spec/containers/0/ports/-", "value":{"containerPort":9225, "name":"metrics"}}]'

This adds a port entry with containerPort: 9225 and name: metrics.

The end result will look like this

        
        ...e: minecraft-minecraft
        ports:
        - containerPort: 25565
          name: minecraft
          protocol: TCP
        - containerPort: 9225
          name: metrics
          protocol: TCP
        readine...

Scrape Minecraft

Install the Prometheus Operator. To do this an ARM (Raspberry Pi architecture) I used https://github.com/uocxp/prometheus-operator-chart-arm.

Prometheus is using ServiceMonitors to scrape endpoints. We need to create a ServiceMonitor that let us scrape a port that we now produce metrics on.

Create a servicemonitor.yaml below.

kubectl apply -f servicemonitor.yaml

Note that the namespace of the service monitor should be the same as where you installed prometheus operator

  • The label release: prometheus-operator needs to be there because the operator only listens to ServiceMonitors with that label by default.

  • The targetPort is the one that we just created in the previous step and the app: minecraft-minecraft comes from the helm chart.

  • The relabeling is to get the node as server_name on the dashboards.

Now you are ready to open up Grafana and install the two dashboards that can be found on the minecraft-prometheus-exporter repository.

kubectl port-forward -n monitor svc/prometheus-operator-grafana 3000:80

Open a browser to http://localhost:3000 and login with the default username and password for prometheus-operator admin prom-operator

Hit the + and paste the Server and the Player dashboards.

grafana.png Enjoy!

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: minecraft-servicemonitor
namespace: monitor
labels:
release: prometheus-operator
spec:
endpoints:
- interval: 10s
path: /metrics
relabelings:
- sourceLabels:
- __meta_kubernetes_pod_node_name
targetLabel: server_name
targetPort: metrics
namespaceSelector:
matchNames:
- minecraft
selector:
matchLabels:
app: minecraft-minecraft
imageTag: multiarch
resources:
requests: {}
tolerations:
- key: "app"
operator: "Equal"
value: "minecraft"
effect: "NoSchedule"
nodeSelector:
role: minecraft
livenessProbe:
initialDelaySeconds: 120
periodSeconds: 60
readinessProbe:
initialDelaySeconds: 120
periodSeconds: 60
minecraftServer:
# This must be overridden, since we can't accept this for the user.
eula: "TRUE"
# One of: LATEST, SNAPSHOT, or a specific version (ie: "1.7.9").
version: "1.15.2"
# This can be one of "VANILLA", "FORGE", "SPIGOT", "BUKKIT", "PAPER", "FTB", "SPONGEVANILLA"
type: "SPIGOT"
# Max view distance (in chunks).
viewDistance: 7
# Message of the Day
motd: "Welcome to Minecraft on Kubernetes on Rapsberry Pi!"
# If you adjust this, you may need to adjust resources.requests above to match.
# If you are running on raspberry pi 4 you can utilize a lot more
memory: 512M
persistence:
## minecraft data Persistent Volume Storage Class
## If defined, storageClassName: <storageClass>
## If set to "-", storageClassName: "", which disables dynamic provisioning
## If undefined (the default) or set to null, no storageClassName spec is
## set, choosing the default provisioner. (gp2 on AWS, standard on
## GKE, AWS & OpenStack)
##
# storageClass: "-"
dataDir:
# Set this to false if you don't care to persist state between restarts.
enabled: true
Size: 10Gi
@gabrielsson
Copy link
Author

Ah or... you have the node selector...

nodeSelector:
role: minecraft

You can remove that part from the values.yaml file. If the none of the nodes have this role.

@lslamp
Copy link

lslamp commented Apr 4, 2021

Sorry I do not understand. I would alike all nodes to serve the minecraft app.

@gabrielsson
Copy link
Author

gabrielsson commented Apr 4, 2021

@islamp sorry to dissapoint. Minecraft server does not scale horizontally. The purpose of this write up is to use familiar tools like rpi, prometheus, grafana together with minecraft.

@lslamp
Copy link

lslamp commented Apr 4, 2021

Max, maybe my wording is wrong and my understanding of clustering.
I have now deleted the namesapce minecraft. so now when I go into k9s, I cannot see the namespace.
If I run the following command I do see respawned nodes.
kubectl top pod --all-namespaces
NAMESPACE NAME CPU(cores) MEMORY(bytes)
kube-system coredns-66c464876b-gwtmw 8m 5Mi
kube-system local-path-provisioner-7ff9579c6-6h45p 4m 5Mi
kube-system metrics-server-7b4f8b595-blfjs 3m 10Mi
kube-system svclb-traefik-26xvs 0m 1Mi
kube-system svclb-traefik-8ppgf 0m 0Mi
kube-system svclb-traefik-tmq2v 0m 0Mi
kube-system traefik-5dd496474-x7d7r 6m 9Mi
kubernetes-dashboard dashboard-metrics-scraper-74db988864-fcdn7 1m 3Mi
kubernetes-dashboard kubernetes-dashboard-7bbb9b5fc6-dkdln 2m 10Mi
minecraft svclb-minecraft-minecraft-59rl6 0m 0Mi
minecraft svclb-minecraft-minecraft-9w225 0m 0Mi
minecraft svclb-minecraft-minecraft-l9pn7 0m 0Mi

@lslamp
Copy link

lslamp commented Apr 4, 2021

Ok now I have deleted and re-created minecraft.
kubectl delete ns minecraft

kubectl create namespace minecraft

helm install --namespace minecraft minecraft -f values.yaml stable/minecraft

kubectl get svc --namespace minecraft minecraft-minecraft

I had to restart the minecraft server twice, but now it is working. Thanks for your input and advice.

Can you please clarify to me how this now works. Does this mean that the cluster is working, and minecraft only works on one of the pi's in the cluster. Is it not using the cluster to use all the CPU and RAM as a single entity?

Thanks
Larwence

@gabrielsson
Copy link
Author

I am glad it works! So now you have accomplished two things. You have a kubernetes cluster up and running. The cluster has 3 raspberrys or "nodes" available for declarative instructions, such as "let there be one minecraft installation".

The minecraft installation installed by helm have done just that, described how the minecraft installation should work. It consists of the deployment which describes how the application should start. The pod, which is the container running minecraft. A service which is the loadbalanced IP address that serves minecraft outside of the cluster. Persistant Volume which makes storage available. Persistant Volume Claim, which minecraft allocated from the volume. But minecraft as an application does not make use of more than a single raspberry, as the actual java application is not distributed, it runs in a single java context (JVM).

Now you can install more things on your home cluster. I am running ad-blocking DNS, home automation, blogs etc all on my raspberry pi cluster. The main purpose of my cluster is for me to educate myself, learning different networking strategies etc when working with kubernetes.

@lslamp
Copy link

lslamp commented Apr 4, 2021

Thanks again for your support and suggestions. It is very much appreciated.
One last question, earlier I asked if it was possible to use influxdb with graphana?

Thanks
Lawrece

@lslamp
Copy link

lslamp commented Apr 7, 2021

Max, I am back with more questions.
I did not get an answer to using influxdb rather than prometheus. I assume this is not possible.
So I was reading details on how to implement prometheus.

To clarify, I have a linux server running grafana, I want to present the minecraft stats onto that page as well. So part of this should be done on the node2 where minecraft is deployed, and the other to my grafana server.

Is this correct logic?

According to your description above, I should run the following command:
kubectl exec -n minecraft-minecraft-7668c5d866-h4gvg -- /bin/bash
error: pod, type/name or --filename must be specified
I also tried the following, but that also failed.

kubectl cp minecraft-prometheus-exporter-2.4.0.jar minecraft/minecraft-minecraft-7668c5d866-h4gvg:/data/plugins
W0407 21:03:21.764754 740167 loader.go:223] Config not found: /home/llamprec/scripts/kubeconfig
Error from server (NotFound): the server could not find the requested resource (get pods minecraft-minecraft-7668c5d866-h4gvg)

kubectl cp minecraft-prometheus-exporter-2.4.0.jar minecraft-minecraft-7668c5d866-h4gvg:/data/plugins
W0407 21:04:49.166165 740536 loader.go:223] Config not found: /home/llamprec/scripts/kubeconfig
Error from server (NotFound): the server could not find the requested resource (get pods minecraft-minecraft-7668c5d866-h4gvg)

Any ideas what I am doing wrong.
Lawrence

@lslamp
Copy link

lslamp commented Apr 9, 2021

Hey Max,
Back to hassle you again ... I managed to guess my way through to copying of the jar file to the plugin dir.

Now I am stuck on exposing port 9225 from deployment.

kubectl patch deployment -n minecraft minecraft-minecraft-7668c5d866-s22wr --type='json' -p='[{"op":"add", "path": "/spec/template/spec/containers/0/ports/-", "value":{"containerPort":9225, "name":"metrics"}}]'

I get the following error.
Error from server (NotFound): deployments.apps "minecraft-minecraft-7668c5d866-s22wr" not found

Now looking I do not have a deployment container. I have read that I can create one but with what config file?
What should the deployment.ymal file look like?

Can you please help me.
Thanks
Lawrence

@gabrielsson
Copy link
Author

Hey Lawrence,

You are using the pods name to patch the deployment. It should be just as I wrote.

kubectl patch deployment -n minecraft minecraft-minecraft --type='json' -p='[{"op":"add", "path": "/spec/template/spec/containers/0/ports/-", "value":{"containerPort":9225, "name":"metrics"}}]'

You can see this first by doing

kubectl get deployment -n minecraft minceraft-minecraft

You should consider using a graphical UI for kubernetes like either https://k8slens.dev/ or https://k9scli.io/ as mentioned before.

The influxdb with grafana can absolutely work, I have no experience with influx though.

@lslamp
Copy link

lslamp commented Apr 10, 2021

Max,

it seems that I am missing something here. when I run the command
kubectl get deployment -n minecraft minceraft-minecraft

I see the below error.
Error from server (NotFound): deployments.apps "minceraft-minecraft" not found

I did follow your advice and used k9s but I also do not see a deployment namespace.
image

I also thoguht I would try to create a deployment namespace with the following.
kubectl create namespace deployment

but even after this I did not see it listed, don't understand why though.

It seems that I do not have the required pre-requisits to run that command

I am very sorry but I am a complete novice with this.
Lawrence

@gabrielsson
Copy link
Author

No worries.

Kubectl get deployment -A

@gabrielsson
Copy link
Author

gabrielsson commented Apr 10, 2021

You ser, you have a deployment there in k9s

It's name is minecraft-minecraft.

In namespace minecraft

Creating a namespace does not do anything for you. (To learn the glossary you can refer to www.kubernetesbyexample.com)

You can actually just edit the deployment you see in k9s and add the metric port.

Then it restarts the pod, you can then port forward into the pod and access the metrics page.

@lslamp
Copy link

lslamp commented Apr 10, 2021

Hey Max,
Again thanks for your patience and tolerance.
It seems I take a step forward and 3 steps backwards.

using k9s I edited the deployment and added the necessary port. The minecraft pod restarted as you said without errors.

I then tried to forward the port using the following.
kubectl apply -f servicemonitor.yaml
servicemonitor.monitoring.coreos.com/minecraft-servicemonitor created

kubectl port-forward -n monitor svc/prometheus-operator-grafana 3000:80
Error from server (NotFound): services "prometheus-operator-grafana" not found

Seems that I am missing something.

I also tried the following which also failed.

helm install prometheus-operator stable/prometheus-operator --namespace monitor --values prometheus-operator.values --set prometheusOperator.tlsProxy.enabled=false --set prometheusOperator.admissionWebhooks.enabled=false
WARNING: This chart is deprecated
W0410 18:40:32.269845 671423 warnings.go:70] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0410 18:40:32.315540 671423 warnings.go:70] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0410 18:40:32.620645 671423 warnings.go:70] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0410 18:40:32.654711 671423 warnings.go:70] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0410 18:40:32.702113 671423 warnings.go:70] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0410 18:40:32.988529 671423 warnings.go:70] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
coalesce.go:200: warning: cannot overwrite table with non table for image (map[repository:kiwigrid/k8s-sidecar tag:0.1.151])
coalesce.go:200: warning: cannot overwrite table with non table for image (map[repository:kiwigrid/k8s-sidecar tag:0.1.151])
coalesce.go:200: warning: cannot overwrite table with non table for image (map[repository:kiwigrid/k8s-sidecar tag:0.1.151])
Error: template: prometheus-operator/charts/grafana/templates/deployment.yaml:43:10: executing "prometheus-operator/charts/grafana/templates/deployment.yaml" at <include "grafana.pod" .>: error calling include: template: prometheus-operator/charts/grafana/templates/_pod.tpl:62:22: executing "grafana.pod" at <.Values.sidecar.image.repository>: can't evaluate field repository in type interface {}

Thanks
Lawrence

@gabrielsson
Copy link
Author

Seems like the instructions are too old

Maybe try something like https://rpi4cluster.com/monitoring/k3s-prometheus-oper/

@gabrielsson
Copy link
Author

That is a longshot... I actually don't know. But there should be some resources online for installing prometheus and grafana on raspberry pi kubernetes cluster.

I am just happy if I can help. But things get stale after a year with this technology :)

@lslamp
Copy link

lslamp commented Apr 10, 2021

Max,
The frustrating thing is I have tried to follow your instructions, and yes, given things change very much over time, but .... after looking at the different sites each has a different naming convention and logic. Reading yours you use monitor, the one you sent me to says monitoring. So with my lack of understanding, it is very confusing when I am asked to do certain things.

I will keep plugging. I will delete all monitoring and see if that helps me.
Lawrence

@gabrielsson
Copy link
Author

I get your frustration.

I looked at the new helm chart for prometheus-kube-stack and that is now supporting raspberry pi it seems (prometheus-community/helm-charts#373)

So you should be able to just run with the standard instructions from
https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack

@lslamp
Copy link

lslamp commented Apr 11, 2021

Hey Max,
Holy schmoly, I have been on a little ride.
I first deleted the pod monitor that I created with my last task. I then followed the instructions on the lower link that you gave me.
Before I started I have some 10 pods running on my cluster. After I carried out my tasks, even though they did not fully work, I ended up with about 25 pods. Many were running but 5 pf the prometheus pods were pending ... could not see why though. What this did was break the minecraft deployment. We could not connect to the server. Many of the pods were in a CaptureLoop state.
I then decided to try to delete all the prometheus pods and see if I could recover the minecraft server.
Using k9s I managed to delete 10 of the pods, but I could not delete a number of them. I then tried to kill the minecraft pod, but that kept re-spawning without errors but I still could not connect to the server. I eventually tailed the log and noticed that the respawn had moved to one of my other nodes. Originally it was on Node2 - 192.168.1.243, but then I saw it had moved to node3 - 192.168.1.247
I then changed my port forwarding and everything worked.
Now I have the following pods.

image

As you can see there are a few with prometheus services, but I have not clue what that means. Is it possible that they are gathering metric data from the minecraft server?

Lawrence

@LytesUp
Copy link

LytesUp commented May 10, 2021

Quick question. Don't go crazy on me because I'm almost completely new with all this stuff and am starting to get into it. My question is what is the max amount of ram your able to allocate to a single server if you had say 32 gb of physical ram in the cluster. (4 8gb raspberry pi 4s)

@lslamp
Copy link

lslamp commented May 10, 2021

@LytesUp
I am also new to this but I don't really understand your question. From what I can gather, If you are using 4 x 8gb RASPIs then all memory will be shared across the cluster. I don't know of any config where you can allocate or assign specific amounts of memory per node.
Lawrence

@LytesUp
Copy link

LytesUp commented May 10, 2021

What im asking is if all the available memory will be used. Becuase from what i understand, minecraft wont use more than a specific amount becuase of the software being used.

@gabrielsson
Copy link
Author

The memory from one Raspberry will be used, as the process will only be running on one machine. And to make things even worse, if you are running a 32 bit operating system on your Raspberry (like raspbian lite) one process can only allocate 2 GB memory.

So to be able to utilize at least 7GB memory you will have to use a 64-bit operating system on the raspberry. The

minecraftServer: 
    memory: 512M

needs to change.

@lslamp
Copy link

lslamp commented May 10, 2021

Max,
You did not respond to my last question. Related to pods that I have running but have no idea how to view the stats.

Do you have any idea?
Lawrence

@gabrielsson
Copy link
Author

@islamp I think we need more interactive way of communicating.

First you can portforward to your minecraft pod on port 9225 and visit the metrics page /metrics

Then you need to check if that is scraped by prometheus, but portforwarding to your prometheus-monitoring-kube... pod on port 9090 (usually). There you can see targets and service discovery to see if everything is setup correctly. Give me shout on twitter or similar and we can see if we can get everything sorted!

@lslamp
Copy link

lslamp commented May 10, 2021

@gabrielsson I 100% agree. Do you have a preferred method of communication.

I am based in the Netherlands so in the same timezone.

below is my config file.
image

When I try to run the following. I get a duplication error.
kubectl patch deployment -n minecraft minecraft-minecraft --type='json' -p='[{"op":"add", "path": "/spec/template/spec/containers/0/ports/-", "value":{"containerPort":9225, "name":"metrics"}}]'

The Deployment "minecraft-minecraft" is invalid: spec.template.spec.containers[0].ports[2].name: Duplicate value: "metrics"

Lawrence

@nilsrenes
Copy link

been experimenting with the bedrock-edition and helm but it seems that the java version also does not scale . :-( If i set replicas to 3 nothing happens . But 1 node than is running high cpu .. I think the pi-edition does support this.. Can you confirm ?

@gabrielsson
Copy link
Author

gabrielsson commented Feb 8, 2024

No, the minecraft process does not scale horizontally. It is merely a fun hosting challange than useful 😆 I am in the midst of updating my cluster... I will give this gist a re-wamp in that case.... this is very outdated now.

@nilsrenes
Copy link

yeah I see that it's been quiet here for the last 3 years but it's still a great project ! That's the way I also see it , a nice challenge to host all of this . I managed to install longhorn and use my synology NAS for the PVC. As far as the metrics I want to use kube-state-metics and elastic-agent to monitor the cluster . combining work-stuff and hosting a custom bedrock map for my son is a great hobby .

@k-yasub
Copy link

k-yasub commented Mar 24, 2024

Hello.
Please forgive my poor English.
How do I change it to make it in Forge version of Minecraft?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment