Skip to content

Instantly share code, notes, and snippets.

@wmantly

wmantly/lxc.md

Last active September 8, 2020 20:23
Show Gist options
  • Save wmantly/fbe170f1a0f311965e54a6340536c4f5 to your computer and use it in GitHub Desktop.
Save wmantly/fbe170f1a0f311965e54a6340536c4f5 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
lxc.md

LXC Unprivileged setup

This will help you set up LXC containers to be used by non-root users. This guide has been tested and updated based on Ubuntu 16.04/18.04.

Update the system

Update your system to the newest packages and reboot. You may need to do this several times:

sudo apt update && sudo apt upgrade -y sudo reboot

Add a container user

If you are going to use LXC for production containers on a server, I recommend adding a user just for running LXC containers. This should SHOULD NOT have sudo access! Make the password something strong, and remember it.

sudo adduser virt

If you are running LXC localy on your desktop for development/testingm skipp this step.

Installing packages

Now you can install the packages we need:

sudo apt install lxc lxctl lxc-templates

Configure network access for the containers

Give the lxc user's network access:

echo "$USER veth lxcbr0 1024" | sudo tee -a /etc/lxc/lxc-usernet

Run this command, replacing $USER with the the username of each user you want to run LXC containers, ie virt.

Configure local LXC setting

Lets set up the config file for each user. Run the lines below for each user that will have LXC containers

The lines below will add the proper config file:

mkdir -p ~/.config/lxc
echo "lxc.idmap = u 0 `grep -oP "^$USER:\K\d+" /etc/subuid` `grep -oP "^$USER:\d+:\K\d+" /etc/subuid`" > ~/.config/lxc/default.conf
echo "lxc.idmap = g 0 `grep -oP "^$USER:\K\d+" /etc/subgid` `grep -oP "^$USER:\d+:\K\d+" /etc/subgid`" >> ~/.config/lxc/default.conf
echo "lxc.net.0.type = veth" >> ~/.config/lxc/default.conf
echo "lxc.net.0.link = lxcbr0" >> ~/.config/lxc/default.conf

Run this command as each user you want to run LXC containers.

Its safer at this point to reboot the system, exit back to the privileged user and sudo reboot

SSH or log directly into the user!!! this will not work if you use su to get into the user! Desktop users must be logged in direcrlt via GUI login.

Change the IP subnet for lxcbr0

If you are using the WG VPN in your network, the defualt subnet for lxcbr0 is 10.0.3.x/24. This can cuase overlap issue withe WG VPN. This is pretty simple to change:

  • Pick a new subnet that will not overlap with the VPN, or your LAN. Here are some options

    • 172.16.3.0/24
    • 192.168.3.0/24
    • a subet net within your VPN subet, if your VPN subenet is 10.2.0.0/16, you can use 10.2.3.0/24

  • Once you have selecred a new subet edit /etc/default/lxc to reflect your new choice replace all 10.0.3.X refecences

      ```
 LXC_BRIDGE="lxcbr0"
 LXC_ADDR="10.0.3.1"
 LXC_NETMASK="255.255.255.0"
 LXC_NETWORK="10.0.3.0/24"
 LXC_DHCP_RANGE="10.0.3.2,10.0.3.254"
 LXC_DHCP_MAX="253"
  ```

  • Restart the lxc-net service using service lxc-net restart

  • Restart any container

Ethernet Bridge

Setting up a Ethernet bridge will allow containers have normal LAN IP's and send/receive raw LAN traffic. You may want this if you have to open inbound ports for the container such as game servers or torrent clients. And also for containers that need to use broadcasts on your LAN, like a start home controller.

Required packages

Please make sure you have the bridge-utils package. This is installed with lxc, so you should already have it.

Get your current interface name

Unless you are using a very old kernel version, your interface will not be eth0. Run the ifconfig command and find the interface with the servers IP. In this example, the interface is enp4s0. Remember your interface for next steps.

william@zdars:~$ ifconfig 
enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.119  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::4b11:928a:a3d4:9019  prefixlen 64  scopeid 0x20<link>
        ether 40:8d:5c:74:e8:2a  txqueuelen 1000  (Ethernet)
        RX packets 179563164  bytes 209188458099 (209.1 GB)
        RX errors 0  dropped 68  overruns 0  frame 0
        TX packets 82616723  bytes 78057486074 (78.0 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 1781039  bytes 356584718 (356.5 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1781039  bytes 356584718 (356.5 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Netplan/ Ubuntu 17.10+

The newer versions of Ubuntu use A new way to configure networking called Netplan. Netplan uses yaml files for its configuration.

Remove the default configuration

By default, Ubuntu used cloud-init to manage the networking and we do not want this. Disable cloud-init:

echo "network: {config: disabled}" > /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg

And remove the existing configuration file:

rm /etc/netplan/50-cloud-init.yaml

Configure the Bridge

The following is a example configures a Bridge using a static IP 192.168.1.41 for interface enp4s0. You can use DHCP, but i do not recommend it for a server.

network:  
  version: 2  
  renderer: networkd
  ethernets:
    enp4s0:
      dhcp4: no
      dhcp6: no
  bridges:  
    br0:  
      interfaces: [enp4s0]  
      dhcp4: no  
      dhcp6: no 
      addresses: [192.168.1.41/24]  
      gateway4: 192.168.1.1  
      nameservers:  
        addresses: [192.168.1.1,8.8.8.8,8.8.4.4]  
      parameters:  
        stp: false  
        forward-delay: 0

Make the changes for this template that apply to your system. Things to change are: * All changed are under br0

  • DHCP4. If you want to use DHCP, set this to true and remove br0 -> address, br0 -> gateway4 and br0 -> namservers.
  • Interface. Both under ethernets and the interfaces array under br0
  • IP address. Under br0 -> address. This is an array of static IP's in CIDR notation.
  • Gateway. Under br0 -> gateway4 This should be the address of your router.

Sticky DHCP

If you use sticky DHCP, have bound the the IP to the server from the DHCP server, you will need to set the macaddress field under br0 to the interfaces MAC address.

To get the MAC address, run this command replacing with the ethernet interface name:

cat /sys/class/net/<INTERFACE>/address

Chage your configur to look like this:

...  
  bridges:  
    br0:  
      interfaces: [enp4s0]
      macaddress: 40:8d:5c:74:e8:2a
      dhcp4: yes  
      dhcp6: no  
      parameters:  
        stp: false  
        forward-delay: 0

source

Once you have configured the this template, save it to /etc/netplan/80-bridge.yaml

Bring the bridge up

To apply the bridge run the netplan generate and apply commands

sudo netplan generate
sudo netplan apply

The server will need to be rebooted for the network to come up after this change, if you workinf remote, make you apply like so:

sudo netplan apply; sleep 10; sudo reboot

Now when you run ifconfig you should see your br0!

ifupdown / Ubuntu 17.04 or lower

Older version of Ubuntu use the legacy network configuration.

Configure the bridge

The following is a example configured Bridge using a static IP 192.168.1.41 for interface enp4s0. You can use DHCP, but i do not recommend it for a server.

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto enp4s0
iface enp4s0 inet manual

auto br0
iface br0 inet static
  address 192.168.1.41
  network 192.168.1.0
  netmask 255.255.255.0
  broadcast 192.168.1.255
  gateway 192.168.1.1
  dns-nameservers 192.168.1.1
  bridge_ports enp4s0
  • Interface. Both under auto and the interfaces array under br0
  • IP address. Under br0 -> address. T
  • Gateway. Under br0 -> gateway This should be the address of your router.

Bring the bridge up

To apply the br0, restart the networking service:

sudo service networking restart`

The server will need to be rebooted for the network to come up after this change, if you workinf remote, make you service like so:

sudo service networking restart; sleep 10; sudo reboot

Now when you run ifconfig you should see your br0!

LXC usage

Creating a container

The lxc-create command makes new containers. -n {name} specifies the name of the container and -t download tells LXC to base the container from a downloaded template, it will present you with a list of available templates.

lxc-create -n test-ubuntu -t download

Starting

lxc-start will start a container and keep it running in the background. -n {name} tells LXC which container you want to start. If all goes well, nothing will be returned.

lxc-start -n test-ubuntu

Attaching

To actually get into the container, we use the lxc-attach command. Its like SSHing into the container. Again, -n {name} specifies which contianer we want to get into. When you are done with the container, you can just use the exit command to leave.

lxc-attach -n test-ubuntu

Information

We can get a list of all containers, and some basic information with the lxc-ls command. use the --fancy flag to get useful stats on each container.

lxc-ls --fancy

Stopping

You can use the reboot and shutdown command within the container just as if it was a real computer. You can also use the lxc-stop command from outside to turn off the container. Buy this point, we should know what -n does.

lxc-stop -n test-ubuntu

Destroying

If you no longer want the container, you can destroy it with the lxc-destroy command. -f will force it be destroyed even if its running.

lxc-destroy -n test-ubuntu

Auto start on boot

For containers to start on boot, lxc-autostart needs to be called by each user that has containers to be booted. To do this, add the following line to each users crontab using crontab -e

@reboot sleep 30 ; /usr/bin/lxc-autostart

Container Configuration

When the container is created, a default condfig file is created for it in the ~/.local/share/lxc/<CONTAINER NAME>/config file. The defaults are mostly fine, but there are some common things you may want to change.

Auto start on boot

If the container houses a service, you probably want it to be started when the server boots. To do this, add these lines to the containers config file:

# Auto start
lxc.start.auto = 1
lxc.start.delay = 10
lxc.start.order = 2
  • lxc.start.auto = 1 Tells the auto start command to start this. Setting 0 would disable this.
  • lxc.start.delay = 10 Is the delay in seconds of starting up the container. You want this if the container needs on other services or containers to function.
  • lxc.start.order = 2 When more then one container is listed to be auto started, there are started in no real order, this lets you dictate the order containers are brought up.

Networking

By default, the containers will be on the lxcbr0 network, receive a IP address in the 10.0.3.x range and NOT be inbound accessible out side of the server. If you need the container to have a normal LAN IP, you must change it to be on the br0 interface. Look for these settings in your config file, add them if needed under the # Network configuration heading.

  • lxc.net.0.type = veth Sets the interface type for the container. Do not change or remove this.
  • lxc.net.0.link = lxcbr0 Sets the bride network for the container. Change this to br0 if you would the container to sit on the LAN.
  • lxc.net.0.flags = up Sets if the interface is up or down.
  • lxc.net.0.ipv4.address = 10.0.3.10/24 Sets the static IP for the container.
  • lxc.net.0.ipv4.gateway = 10.0.3.1 Sets the gateway for the interface.

Mount folders from the Host

With LXC you can mount folders from the host to use on the container. Since we are using unprivileged containers, there are 2 things to keep in mind:

  1. We must use absolute paths from the host system in all configuration files.
  2. Reads and writes happen as the container user!

To make a mount, start with creating a mount point in the container. This example will be /media/stuff

mkdir /media/stuff

Now shutdown the container. It needs to be booted from a cold boot for the mount to take affect. You can shutdown the container from inside with shutdown -h now or from the host with lxc-stop -n <name>

Now we have have to edit the configuration file to include the mount. Here is a sample line:

lxc.mount.entry = /stuffpool/stuff     /home/virt/.local/share/lxc/speedtest/rootfs/media/stuff none bind,follow_symlinks 0 0

In this line, the user virt owns the container speedtest and is mounting /stuffpool/stuff from the host to the container /media/stuff. Notice how the the mount point is the absolute path for the host to the containers root directory. For unprivileged containers THIS MUST BE SET LIKE THIS!!! The root FS will be /home/$USER/.local/share/lxc/<container name>/rootfs/. The last portion is normal mount options.

When this is configured, you can bring up your container using lxc-start -n <name> and you should have access to the mount.

More

You can see other options valid for your version by running man 5 lxc.container.conf

ToDo

  • Add section for LAN bridge interface.
  • Add section on configuring containers.
  • Add section for auto starting containers.
  • Add section on resource limiting.
  • Add section on remote HTTP access.

https://www.kubos.cz/2018/07/20/ubuntu-unprivileged-lxc

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment