Skip to content

Instantly share code, notes, and snippets.

@averyfreeman

averyfreeman/mongodb-in-systemd-container.md

Last active January 22, 2025 05:25
Show Gist options
  • Save averyfreeman/47b25d01f7b57b1e02d33706f9d4454c to your computer and use it in GitHub Desktop.
Save averyfreeman/47b25d01f7b57b1e02d33706f9d4454c to your computer and use it in GitHub Desktop.
Download ZIP
mongodb in a systemd container (systemd-nspawn/machinectl)
Raw
mongodb-in-systemd-container.md
title author date
mongodb in systemd-container
averyfreeman
2022-03-01

happy robot

mongodb in systemd-container

(which make awesome dev containers)

What's a Systemd-Container?:

If you're here, you probably already know some of the benefits of systemd-containers compared to more conventionally-packaged containers (e.g. OCI or "docker" containers), but just in case you're not familiar, I'll tick through some differences real quick:

  • pros:
    • flexible, easy to manage
    • can be RW or RO
    • can actually run systemd inside (were initially created for testing systemd on a computer already running systemd)
    • appear to filesystem like any other directory (tar.xz -based containers, not .img, .qcow2)
  • cons:
    • linux-only (compat layer on WSL, FreeBSD or Illumos maybe possible, but unlikely)
    • sparse documentation, small community, not a lot of helper tools (any?)
    • no built-in consumer-friendly packaging mechanism (hence: tar.xz, .img, .qcow2)
    • the set-up is all your own (there's no "machinectl repository" with a bunch of packaged apps")

There's not a whole lot of information about them online, or community around them, but there's a man page, which is a good place to start.

Ubuntu version (might contain vendor-specific changes): https://manpages.ubuntu.com/manpages/hirsute/man1/machinectl.1.html

Freedesktop version (orig / universal): https://www.freedesktop.org/software/systemd/man/systemd-nspawn.html

I've also curated some related further reading at the end of the gist

If you're on Ubuntu, the package is systemd-container - here's the rundown:

$ apt show systemd-container
Package: systemd-container
Version: 249.10-0ubuntu1
Priority: optional
Section: admin
Source: systemd
Origin: Ubuntu
Maintainer: Ubuntu Developers <[email protected]>
Original-Maintainer: Debian systemd Maintainers <[email protected]>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 1,398 kB
Depends: libacl1 (>= 2.2.23), libbz2-1.0, libc6 (>= 2.34), libcurl3-gnutls (>= 7.16.3), libgcrypt20 (>= 1.9.0), liblzma5 (>= 5.1.1alpha+20120614), libseccomp2 (>= 2.3.1), libselinux1 (>= 3.1~), systemd (= 249.10-0ubuntu1), zlib1g (>= 1:1.1.4), default-dbus-system-bus | dbus-system-bus
Recommends: libnss-mymachines
Homepage: https://www.freedesktop.org/wiki/Software/systemd
Download-Size: 387 kB
APT-Manual-Installed: no
APT-Sources: http://us.archive.ubuntu.com/ubuntu jammy/main amd64 Packages
Description: systemd container/nspawn tools
 This package provides systemd's tools for nspawn and container/VM management:
  * systemd-nspawn
  * systemd-machined and machinectl
  * systemd-importd
  * systemd-portabled and portablectl

to install ...:

$ sudo apt update && sudo apt install -y systemd-container

Housekeeping --- $env tested:

$ lsb_release -a && uname -a  && zfs --version && systemctl --version
Distributor ID: Ubuntu
Description:    Ubuntu Jammy Jellyfish (development branch)
Release:        22.04
Codename:       jammy
Linux endpoint-bob 5.15.25-xanmod1 #0~git20220223.c7a9be0 SMP Wed Feb 23 16:21:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
zfs-2.1.2-1ubuntu3
zfs-kmod-2.1.2-1ubuntu3
systemd 249 (249.10-0ubuntu1)
+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS -OPENSSL +ACL +BLKID +CURL +ELFUTILS -FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP -LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified

Quick note:

I am breaking these out into smaller sections since this gist keeps growing. The first section probably doesn't apply to that many people, so I decided to separate it first, but if you'd like to see it, I moved it here:

tl;dr for zfs users: machinectl unsupported fs

btrfs zvol for use with systemd-container

Let's get started:

  • pull ubuntu image
    • last version of ubuntu officially compatible with mongod 4.2 is 18.04 (bionic)
    • reminder: machinectl known bug re: renaming on systemd <249. If experienced, wget <imageurl>, then import-tar as workaround
    • below works on 22.04 zfs, but only if supported fs zvol mounted /var/lib/machines (btrfs, xfs, ext4 - tested btrfs). zfs naming issue persistent (zfs not supported machinectl), zfs users, see above tl;dr link for stable workaround
 # machinectl pull-tar https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64-root.tar.xz bionic --verify=no

  • --verify=no (?) : checksumming in machinectl perpetually broken

  • clone pulled or imported image to retain an original:

# machinectl clone bionic bionicmongo

I plan on making another section just for basic host<--->container networking, but for now we'll just set it up as easily as possible to be able to download mongodb and use apt while you're inside.

  • navigate /var/lib/machines/bionicmongo/etc - this is /etc folder of your container.
  • delete resolv.conf, which is a symlink from systemd-resolved

note: you cannot configure systemd-resolved in the container unless you start it with -b, as process 1|PID1 is not available (so, no systemctl for services).

# cd /var/lib/machines/bionicmongo/etc && rm -f resolv.conf
# echo 'nameserver $yourdnsip' > resolv.conf
  • go back to the root of the folder, and we'll delete some garbage
  • then start the container using systemd-nspawn from the root of container /var/lib/machines/bionicmongo):
    • important note: when running container for mongod to run, ---capability=CAP_IPC_LOCK is to avoid mlock failure. Discussion here
# cd ../
# systemd-nspawn --capability=CAP_IPC_LOCK
  • now that you're inside the container, remove useless garbage:
    • additionally remove ufw if don't want to have to configure firewall ports from inside the container.
      • I don't have any info here on using ufw, so if you're not familiar how to open ports with ufw (most notably 27017 for mongod), I recommend googling "ufw open port" if you decide to keep ufw (there's lots of resources).
# apt remove -y lxc lxcfs lxd-client lxd open-iscsi plymouth plymouth-theme-ubuntu-text snapd ubuntu-release-upgrader-core unattended-upgrades ubuntu-advantage-tools ubuntu-release-upgrader-core [ufw]
apt autoremove --purge -y
apt update && apt upgrade -y

There's probably a lot more you can remove, that's what I got from a quick scan. I'll be back to add more later.

  • make user (or at least give root a passwd):
ability to login as root with -b flag:
# passwd root
<enterpasswd x2>

user acct - no login:
# useradd  -d /dev/null -s /bin/false -u 1001 -g 1001 mongouser

or with login:
# mkdir -p /home/mongouser
# useradd  -d /home/mongouser -s /bin/bash -u 1001 -g 1001 mongouser
# passwd mongouser
<enterpasswd x2>
  • download the appropriate .deb for mongodb. I plan on using robo3t, but it's only compatible with v4.2. the latest stable patch version of v4.2 is currently 4.2.18. if you get it from the site, make sure says "server", the REPL .deb is in the same list. 
# cd /opt && wget https://repo.mongodb.org/apt/ubuntu/dists/bionic/mongodb-org/4.2/multiverse/binary-amd64/mongodb-org-server_4.2.18_amd64.deb
# apt install -y ./mongodb-org-shell_4.2.18_amd64.deb
# mkdir -p /data/db
# mongod &

should have running mongod using same loopback address as the host, so the database will be available at localhost@27017.

if want to have mongod communicate a separate network interface, there's a bunch more we have to do. stay tuned for my addendum I'm working on where I'll detail all that.

for now, though, you should have a working copy of mongod available inside a container you can get into and make changes to easily. if you'd like to package it up for distribution, you can make it into an OCI ("docker") image later. It's a good start!

if you need systemd services, login with -b flag

  • important: must always use --capability=CAP_IPC_LOCK, otherwise inevitable mongod segfault mlock failure. Full invocation example (-b shown as optional):
[ root@/var/lib/containers/bionicmongo ] # systemd-nspawn --capability=CAP_IPC_LOCK [-b]
  • or start from somewhere else:
[ root@/some/other/folder ] # systemd-nspawn --capability=CAP_IPC_LOCK [-b] -D /var/lib/containers/bionicmongo
  • start command script:
# vim start-mongodb-systemd-container.sh
  • paste:
#!/bin/bash
systemd-nspawn --capability=CAP_IPC_LOCK -b -D /var/lib/containers/bionicmongo
  • save, make executable:
# chmod +x start-mongodb-systemd-container.sh
  • must run privileged:
$ sudo ./start-mongodb-systemd-container.sh
  • if need to know where mongo-core-server files installed (while inside container):
# dpkg -L mongodb-org-server

e.g.:

/etc/mongod.conf
/lib/systemd/system/mongod.service
/usr/bin/mongod
/usr/share/doc/mongodb-org-server/LICENSE-Community.txt.gz
/usr/share/doc/mongodb-org-server/MPL-2.gz
/usr/share/doc/mongodb-org-server/README
/usr/share/doc/mongodb-org-server/THIRD-PARTY-NOTICES.gotools.gz
/usr/share/doc/mongodb-org-server/THIRD-PARTY-NOTICES.gz
/usr/share/doc/mongodb-org-server/changelog.gz
/usr/share/doc/mongodb-org-server/copyright
/usr/share/lintian
/usr/share/lintian/overrides
/usr/share/lintian/overrides/mongodb-org-server
/usr/share/man/man1/mongod.1.gz

Packaging as OCI (for docker, containerd, kubernetes, etc.)

% TODO - part deux: packaging a systemd-container into OCI (and possibly systemd units)---

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment