However, there is an alternative way of running docker, based on systemd-nspawn now available for testing!
Last active
October 30, 2024 20:38
-
-
Save tprelog/7988dc6b196775f33929beb19f0090d7 to your computer and use it in GitHub Desktop.
Use docker-compose on TrueNAS SCALE 22.12 (Bluefin) without Kubernetes
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env bash | |
| # | |
| # Enable docker and docker-compose on TrueNAS SCALE (no Kubernetes) | |
| # | |
| # This script is a hack! Use it at your own risk!! | |
| # Using this script to enable Docker is NOT SUPPORTED by ix-systems! | |
| # You CANNOT use SCALE Apps while using this script! | |
| # | |
| # 1 Create a dedicated Docker dataset in one of your zpools | |
| # 2 Save this script somewhere else on your zpool, not in the Docker dataset | |
| # 3 Edit line 20 of the script, set a path to the Docker dataset you created | |
| # 4 You can now start Docker by running the script from the SCALE console | |
| # | |
| # For these changes to persist after SCALE reboots and upgrades, run the script at start-up | |
| # Schedule this script to run via System Settings -> Advanced -> Init/Shutdown Scripts | |
| # Click Add -> Type: Script and choose this script -> When: choose to run as Post Init | |
| ## Set a path to your docker dataset | |
| docker_dataset='/mnt/tank/docker' | |
| ## HEREDOC: docker/daemon.json | |
| read -r -d '' JSON << END_JSON | |
| { | |
| "data-root": "${docker_dataset}", | |
| "storage-driver": "overlay2", | |
| "exec-opts": [ | |
| "native.cgroupdriver=cgroupfs" | |
| ] | |
| } | |
| END_JSON | |
| ## path to docker daemon file | |
| docker_daemon='/etc/docker/daemon.json' | |
| if [ ${EUID} -ne 0 ]; then | |
| echo "Please run this script as root or using sudo" | |
| elif [ "$(systemctl is-enabled k3s)" == "enabled" ]; then | |
| echo "You can not use this script while k3s is enabled" | |
| elif [ "$(systemctl is-active k3s)" == "active" ]; then | |
| echo "You can not use this script while k3s is active" | |
| elif ! which docker &> /dev/null; then | |
| echo "Docker executable not found" | |
| elif ! chmod +x /usr/bin/docker-compose &> /dev/null; then | |
| echo "Failed to make docker-compose executable" | |
| elif ! install -d -m 755 -- /etc/docker &> /dev/null; then | |
| echo "Failed to install directory: /etc/docker" | |
| elif ! zfs list "${docker_dataset}" &> /dev/null; then | |
| echo "Dataset not found: ${docker_dataset}" | |
| else | |
| echo "Checking file: ${docker_daemon}" | |
| if test "${JSON}" != "$(cat ${docker_daemon} 2> /dev/null)"; then | |
| echo "Updating file: ${docker_daemon}" | |
| jq -n "${JSON}" > ${docker_daemon} | |
| if [ "$(systemctl is-active docker)" == "active" ]; then | |
| echo "Restarting Docker" | |
| systemctl restart docker | |
| elif [ "$(systemctl is-enabled docker)" != "enabled" ]; then | |
| echo "Enable and starting Docker" | |
| systemctl enable --now docker | |
| fi | |
| fi | |
| fi |
It seems this no longer works with TrueNAS SCALE 22.12.1. Anyone has any tips on how to make it work?
It seems this no longer works with TrueNAS SCALE 22.12.1. Anyone has any tips on how to make it work?
Create empty /etc/docker.env file.
People also experiencing issues with SMB shared host paths breaking plex arrs etc. Welp I guess we didn’t have to wait til Cobia for them to butcher it and guess i’m never updating lol.
People also experiencing issues with SMB shared host paths breaking plex arrs etc. Welp I guess we didn’t have to wait til Cobia for them to butcher it and guess i’m never updating lol.
There is nothing new being butchered. Bluefin hostPath validation existed since bluefin launch.
People also experiencing issues with SMB shared host paths breaking plex arrs etc. Welp I guess we didn’t have to wait til Cobia for them to butcher it and guess i’m never updating lol.
There is nothing new being butchered.
Bluefin hostPath validation existed since bluefin launch.
Ok let me be more clear: a hostpath volume under a shared dataset would pass validation under 22.12.0, this “bug” was “fixed” in 22.12.1. This breaks 99% of people’s plex/emby/jellyfin configurations.
People also experiencing issues with SMB shared host paths breaking plex arrs etc. Welp I guess we didn’t have to wait til Cobia for them to butcher it and guess i’m never updating lol.
There is nothing new being butchered.
Bluefin hostPath validation existed since bluefin launch.Ok let me be more clear: a hostpath volume under a shared dataset would pass validation under 22.12.0, this “bug” was “fixed” in 22.12.1. This breaks 99% of people’s plex/emby/jellyfin configurations.
This was warned against by me on SO MANY places, it was in fact a bug and never intended to work. It was fixed a week or so after release of Bluefin even.
Ok i’ll keep my buggy 22.12.0 and warn others to do the same, thanks.
Ok i’ll keep my buggy 22.12.0 and warn others to do the same, thanks.
It's one of the reasons I build NFS support into truecharts. Works flawlessly for media libraries for most of our users.
Docker inside a jail made with jailmaker survived the update to 22.12.1 without any issues. I've also never ran into host path validation. I have nextcloud use the same local path I use for my SMB shares.
People also experiencing issues with SMB shared host paths breaking plex arrs etc. Welp I guess we didn’t have to wait til Cobia for them to butcher it and guess i’m never updating lol.
There is nothing new being butchered.
Bluefin hostPath validation existed since bluefin launch.Ok let me be more clear: a hostpath volume under a shared dataset would pass validation under 22.12.0, this “bug” was “fixed” in 22.12.1. This breaks 99% of people’s plex/emby/jellyfin configurations.
Hostpath validation shouldn’t matter for folks using this or other docker workarounds. 22.12.1 works fine with Jip-Hops Jailmaker. I personally use Plex, Emby, and several arrs with this setup.
It seems this no longer works with TrueNAS SCALE 22.12.1. Anyone has any tips on how to make it work?
Create empty /etc/docker.env file.
You're a genius! Thank you!
Do you mind explaining why this works? Noob here.
It seems this no longer works with TrueNAS SCALE 22.12.1. Anyone has any tips on how to make it work?
Create empty /etc/docker.env file.
You're a genius! Thank you!
Do you mind explaining why this works? Noob here.
Sure.
Take a you look in /etc/systemd/system/docker.service.d/override.conf and you'll see this
EnvironmentFile=/etc/docker.env
Docker service will fail to start if the file is missing with the following error
docker.service: Failed to load environment files: No such file or directory
You can see the log with command journalctl -u docker.service
It seems this no longer works with TrueNAS SCALE 22.12.1. Anyone has any tips on how to make it work?
Create empty /etc/docker.env file.
Do the contents of /etc get reset on the next upgrade?
This should probably be added to the enable-docker.sh script:
touch /etc/docker.env
It seems this no longer works with TrueNAS SCALE 22.12.1. Anyone has any tips on how to make it work?
Create empty /etc/docker.env file.
Do the contents of /etc get reset on the next upgrade?
This should probably be added to the enable-docker.sh script:
touch /etc/docker.env
I believe this is the case.
It seems this no longer works with TrueNAS SCALE 22.12.1. Anyone has any tips on how to make it work?
Create empty /etc/docker.env file.
You're a genius! Thank you!
Do you mind explaining why this works? Noob here.Sure.
Take a you look in
/etc/systemd/system/docker.service.d/override.confand you'll see thisEnvironmentFile=/etc/docker.envDocker service will fail to start if the file is missing with the following error
docker.service: Failed to load environment files: No such file or directoryYou can see the log with command
journalctl -u docker.service
Thank you.
When I checked /etc/systemd/system/docker.service.d/override.conf the only thing inside was LimitCORE=1. The journalclt info I confess that it's hard for me to understand much but I did see docker.service: Failed to load environment files: No such file or directory just didn't have a clue how to solve that. I guess I learned something new today.
I have a similar though more simple script to make this all work and I'm also thinking about adding the touch /etc/docker.env bit to it.
It seems this no longer works with TrueNAS SCALE 22.12.1. Anyone has any tips on how to make it work?
Create empty /etc/docker.env file.
You're a genius! Thank you!
Do you mind explaining why this works? Noob here.Sure.
Take a you look in/etc/systemd/system/docker.service.d/override.confand you'll see thisEnvironmentFile=/etc/docker.env
Docker service will fail to start if the file is missing with the following errordocker.service: Failed to load environment files: No such file or directory
You can see the log with commandjournalctl -u docker.serviceThank you.
When I checked
/etc/systemd/system/docker.service.d/override.confthe only thing inside wasLimitCORE=1. The journalclt info I confess that it's hard for me to understand much but I did seedocker.service: Failed to load environment files: No such file or directoryjust didn't have a clue how to solve that. I guess I learned something new today.I have a similar though more simple script to make this all work and I'm also thinking about adding the
touch /etc/docker.envbit to it.
Did this work for you?
I'm getting an error when trying to run a container that uses a local volume. I've set-up the script with /mnt/spins3/appdata/docker for docker_dataset and I can see that all the docker files are there.
`root@truenas# docker volume create portainer_data portainer_data
root@truenas[~]# docker volume ls DRIVER VOLUME NAME local portainer_data
root@truenas[~]# docker run -d -p 8000:8000 -p 9443:9443 --name portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce:latest docker: Error response from daemon: invalid volume specification: 'portainer_data:/data': portainer_data path not allowed to be mounted. See 'docker run --help'.`
Anyone else getting this issue?
Did you managed to get around this? A year later I cannot install Portainer agent on the TrueNAS Scale host.
I used to use portainer on TrueNAS SCALE before I switched to jailmaker. Worked for me when I used bind mounts instead of docker volumes. Just create a directory where you want to keep your portainer data and mount it inside the portainer container instead of using the volume. No need to create a volume in this case either. But beware this method of running docker will stop working in October.
I used to use portainer on TrueNAS SCALE before I switched to jailmaker. Worked for me when I used bind mounts instead of docker volumes. Just create a directory where you want to keep your portainer data and mount it inside the portainer container instead of using the volume. No need to create a volume in this case either. But beware this method of running docker will stop working in October.
Thanks for replying. Having read all kinds of discussions about docker implementation on TrueNAS Scale, I'm thinking of moving out my Docker containers entirely from it and using it for what it was intended to do in the first place - storage.
@scotrod Did you try my guide on the subject? https://forum.level1techs.com/t/truenas-scale-native-docker-vm-access-to-host-guide/190882
@scotrod Did you try my guide on the subject? https://forum.level1techs.com/t/truenas-scale-native-docker-vm-access-to-host-guide/190882
Thank you! Definetely saving this for future references. That said, I really don't like TrueNAS' integrated Docker solution. I will look up if I can hook up everything via NFS shares, so the containers access the data via the network. It's going to hit the performance for sure, but for my purposes I think it should be fine.
@scotrod Did you try my guide on the subject? https://forum.level1techs.com/t/truenas-scale-native-docker-vm-access-to-host-guide/190882
Thank you! Definetely saving this for future references. That said, I really don't like TrueNAS' integrated Docker solution. I will look up if I can hook up everything via NFS shares, so the containers access the data via the network. It's going to hit the performance for sure, but for my purposes I think it should be fine.
We've all been down this same path for the same reasons and you should really look at Jip-Hops Jailmaker script.
Considering they will remove Docker in the next update of Scale, it won't be their solution, but a regular Docker install on a Debian server.
Considering they will remove Docker in the next update of Scale, it won't be their solution, but a regular Docker install on a Debian server.
They are removing Docker now!?
Considering they will remove Docker in the next update of Scale, it won't be their solution, but a regular Docker install on a Debian server.
They are removing Docker now!?
Yeah, it was discussed here, in 23.10 they will remove Docker from TrueNAS, but my script should bring it back on the next reboot right after the update. (Still not tested, of course…) They'd give us more of a headache if they removed apt, but that would still be solvable. This will always be a Linux server with a network connection, so almost everything is possible.
IMO the best way to run docker, if you don't want to run it inside a 'jail' with jailmaker, is not to enable the package manager and install docker. Instead, download and run the binaries. You can set it up in a portable way so your docker install will be preserved on upgrades. And you minimize the interaction with the TrueNAS host (not enabling and installing from the package manager with possibly unintended side effects). I think this way it should even be possible to get it to run alongside TrueNAS Apps. If someone would create a script for this I recommend to take a look at what I'm doing with ubernerd. It uses nerdctl (which is a mostly docker compatible tool to run containers) which has the additional benefit of being able to run 'jails'. This docker install script could be inspiring too. It can install the latest docker binaries, but it's missing the part to make the installation contained to a single directory (to keep it preserved after upgrade and limit the amount of interaction with the host rootfs).
And here we come a year later. Today's announcement is that TrueNAS Scale apps are moving from Kubernetes to Docker :)
And here we come a year later. Today's announcement is that TrueNAS Scale apps are moving from Kubernetes to Docker :)
There was no need to necro this thread a year later, by posting about an announcement made 6(!) months ago.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
To be clear: The performance things you see is not likely related to the docker setup you did.
Also: Cobia is not going to be "butchered" or "crippled", they just removed something they are not using anymore for the kubernetes stack.
Just because that might or might not affect you for hacks like these, doesn't mean it was intended for fuck with people using docker.
That being said: With docker removed, it should be safer now to run docker by installing it via apt-get :)
Why? Because iX isn't using it anymore, so less chance of potential conflicts!