Linux Kernel Firmware Packages

kernel-firmware-pkg.md

Linux Kernel Firmware Packages

Motivation

Kernel firmware packages consume a lot of disk space on a Linux system, both on the installed system and in an installation environment (inst-sys). In some cases (e.g. RAM disk for a minimal NET ISO), that also means a lot of RAM usage.

The installed kernel-firmware packages on a Leap 15.5 (screenshot of qdirstat pkg:/kernel-firmware)

Together, they consume more disk space than most of the largest packages of the system:

All installed packages on a Leap 15.5 development system

Yet most of those packages are largely unused; only those for which the hardware is actually present are really used. But there are caveats; see below.

Firmware Packages Content

In general, those firmware packages contain binary data that either needs to be uploaded to some piece of hardware, or (mostly hardware-specific) binary code that is not available as Open Source code.

In a few cases, it might also be binary-only codecs that cannot be distributed as source code.

Directory Structure

Firmware binary files can be in any of those directories:

On SLE-15 SPx / Leap 15.x:

• /lib/firmware (directly without a subdirectory)
• /lib/firmware/<hardware-vendor-name> (most common)
• /lib/firmware/<hardware-vendor-name>/<hardware-name>

On newer versions (Tumbleweed, ALP) after the usr-merge:

• /usr/lib/firmware (directly without a subdirectory)
• /usr/lib/firmware/<hardware-vendor-name> (most common)
• /usr/lib/firmware/<hardware-vendor-name>/<hardware-name>

(The old /lib/firmware paths are still valid because /lib is a symlink to /usr/lib)

Addendum by Takashi:

/lib/firmware is treated as the root directory, and kernel tries to load from the /lib/firmware/updates/$VERSION, /lib/firmware/updates, and /lib/firmware/$VERSION directories before /lib/firmware.

Compression

The firmware binary files are already compressed in the .xz format.

Addendum by Takashi:

• Some files are intentionally left uncompressed; e.g. AMD CPU ucode or some small files that don't need compression.
• The all-in-one uncompressed package is planned to be renamed in future to kernel-firmware-compressed, so that it won't be used for OBS builds. Once after this rename, kernel-firmware-all will be used as kernel-firmware alias instead. The installer and other images can use the compressed / split packages as well. Tracked in bsc#1214789.

Exception: Legacy kernel-firmware

Package kernel-firmware is the monolithic legacy version before the package split, and before file compression was added to the kernel modules. That is why this package has an installed size of 874.2 (!) MB.

This package is not installed by default, but it is used in the YaST inst-sys.

To check with Steffen: Is this really true? Could we also use the split and compressed ones to save disk space? Would it even save disk space, or is the inst-sys filesystem compressed anyway?

Versions / Hardware Revisions

In many cases, there is a firmware binary file in many different versions for the same hardware; but even hardware comes in different versions or revisions, and they may require different firmware binaries.

Dependencies

Each kernel-firmware package has a number of provides for its firmware binaries, e.g.

• firmware(RTL8192E/main.img)
• firmware(rt3090.bin)
• firmware(rtl_nic/rtl8153a-4.fw)

They have supplements for the hardware IDs the firmware supports, e.g.

• modalias(pci:v000010ECd00002502svsdbcsci*)
• modalias(pci:v000010ECd00008168svsdbcsci*)
• modalias(pci:v000010ECd00008173svsdbcsci*)

Addendum by Takashi:

Each kernel-firmware-* subpackage contains the hardware supplements, which are retrieved from the corresponding kernel modules, so that they can be installed automatically when the hardware is present. Some other firmware packages have also supplements, too (usually specified in the spec manually).

They typically do not have any meaningful requires.

There is a "bracket" package kernel-firmware-all that pulls in most (almost all) of the other kernel-firmware-* packages.

Do not confuse this package with the legacy kernel-firmware package that is the one before the package split, containing all the firmware binary files in uncompressed format.

Other Firmware Packages

Notice that besides kernel-firmware packages, there are also other firmware packages (try zypper search firmware), e.g.

• adaptec-firmware
• alsa-firmware
• arm-trusted-firmware
• arm-trusted-firmware-a80x0_mcbin
• arm-trusted-firmware-imx8mm
• ...
• atmel-firmware
• b43legacy-firmware
• bcm20702a1-firmware
• bcm43xx-firmware
• bladeRF-fpga-firmware
• bladeRF-fx3-firmware
• bluez-firmware
• gnome-firmware
• raspberrypi-firmware
• raspberrypi-firmware-dt
• raspberrypi-firmware-extra
• rtl8761b-firmware
• sof-firmware
• ...
• ...

To name just a few. None of those gets installed by default, though.

Kernel Modules and Firmware

Typically, it's the kernel driver (module) that loads the firmware from those files and uses it. Check the journal (dmesg or sudo journalctl).

Addendum by Takashi:

The load of a firmware isn't always shown in the kernel log.

It's visible only when the debugging is enabled on the kernel firmware loader, e.g. passing the firmware_class.dyndbg=+p boot option.

Virtual Machines

Virtual machines (QEMU / LibVirt, VirtualBox, VMWare) do not need any kernel-firmware packages: The virtualization layer takes care of that.

Addendum by Takashi:

Usually no need of firmware for a VM. Due to that reason, kernel-default-base, which is a reduced version of the kernel-default package for VM and co, doesn't have recommends of kernel-firmware unlike the standard kernel-default package.

Hence, for example, SUSE Micro doesn't install firmware files as default, and it caused problems when you install it on bare metal as a side effect.

Minimizing the Footprint

Are They All Needed?

Currently, we install all of those kernel-firmware packages; but that is mostly for historic reasons. They used to be all in one single huge kernel-firmware package which, at the time, was the largest package by far that you could install.

Then we asked the maintainer of that package, Takashi Iwai [email protected], if it was possible to split that huge package into smaller ones, and to compress the binary files, and he agreed and did all the (considerable) work. But so far, there has not been a good concept how and when to install only those of the smaller kernel-firmware packages that are actually needed.

The Bare Essentials Approach

Use the result of hardware probing during installation and identify what hardware is present and install only the kernel-firmware package for that hardware.

This really minimizes the number of firmware packages to install at the risk of not having enough of them when the hardware changes ever so slightly (see section Hardware Replacement below).

If this approach is chosen, there should also be an easy way for the user to trigger this again at some later time: Do the hardware probing again, install the missing kernel-firmware packages and ask the user if those that are now not needed anymore should be deinstalled.

Pluggable Hardware

USB hardware, for example, cannot be detected that way; it might not be plugged in, or not switched on. But admittedly, that type of hardware does not frequently need firmware binaries.

To verify with Takashi

Hardware Replacement, Addition or Upgrade

When a piece of hardware dies, it is typically easy to replace; either individually like a a graphics card, a RAID controller, or a network card, or, more commonly, the whole mainboard with the onboard components.

But that will most often mean replacing it with hardware from a different vendor, or at least with a different hardware revision. And that means that the previously installed kernel firmware package may no longer be the right one for that hardware.

The same is true, of coure, if a new piece of hardware is added to the system, or if one is upgraded to a newer, better one (a new graphics card, for example): The matching kernel-firmware might not be installed yet.

Adding Firmware on Demand

So, if the installation only installs the bare minimals of kernel-firmware packages, just the ones that were needed at the time of installation, we need to be able to add more kernel-firmware packages whenever needed. And firstly, of course, the system must still be able to boot and to allow user interaction (display output, accept input) and to install those packages (i.e. access the network and storage).

The Chicken-Egg Problem

To install those additional firmware packages, all the hardware that is needed in that process needs to work; and that means having the firmware for that hardware available: Fetching kernel-firmware packages from a remote repository means a working network connection. Unpacking the RPMs to /lib/firmware means a working storage controller and working disks. Issuing the commands to start it means a working keyboard, probably also a working mouse or touchpad, and a working video card. And to even get that far after replacing or adding hardware means booting, which again means a working storage controller and working disks.

That does not leave that much hardware that is not needed during that process.

Essential Hardware

• Booting:

  • Mainboard
  • CPU
  • Storage controllers
  • Disks (including SSDs)

• User Interaction:

  • Graphics card
  • Display
  • Input devices (keyboard, mouse, touchpad)
  • Alternatively: Serial console

• Connectivity:

  • Netword cards

Firmware in the Installation Environment

During system installation, all the hardware that is involved in the installation needs to be working; see section Essential Hardware above. Not all that hardware needs to work in the best possible mode of operation, though; to some extent, degraded modes are acceptable.

YaST inst-sys

etc/config in the installation-images GitHub source repo adds those firmare packages to the inst-sys for the x86_64 (PC) architecture:

• kernel-firmware
• adaptec-firmware
• ipw-firmware
• iwl4965-ucode
• iwl5000-ucode
• atmel-firmware
• ralink-firmware

i.e. beyond the legacy kernel-firmware package (containing uncompressed firmware files) also the firmware for very common Adaptec RAID controllers and for a number of WiFi adapters.

YaST inst-sys /lib/firmware

(Checked on a Leap 15.5 ISO)

This is much less than expected; just 179.3 MB in 1126 (uncompressed!) files, far from the complete legacy kernel-firmware package's 874.2 MB. This appears to be a hand-curated list of firmware binaries that are actually needed during installation.

To be checked with Steffen

Agama Live Media

@tiwai, @wfeldt: FYI; still heavily WIP. Comments welcome.

This is a temporary location. Mid-term, this will become part of a GitHub repo like agama, and this will become an alternate location. But it's much more convenient to edit here.

Thanks for a nice article!
Some additional notes from my side.

Directory Structure:

• It's /usr/lib/firmware on Tumbleweed (which is symlinked to /lib/firmware).

• /lib/firmware is treated as the root directory, and kernel tries to load from /lib/firmware/updates/$VERSION, /lib/firmware/updates, and /lib/firmware/$VERSION directories before /lib/firmware.

Compression:

• Some files are intentionally left as uncompressed; e.g. AMD CPU ucode or some small files that don't need
  compressions.

• The all-in-one uncompressed package is planned to be renamed in future to kernel-firmware-compressed, so that it won't be used for OBS builds. Once after this rename, kernel-firmware-all will be used as kernel-firmware alias instead.
  The installer and other images can use the compressed / split packages as well. Tracked in bsc#1214789.

Versioning:

• The kernel firmware should be in principle backward-compatible;
  if the driver requires a new firmware ABI, usually it has a different firmware file name with some versioning.

Dependencies:

• Each kernel-firmware-* subpackage contains the hardware Supplements, which are retrieved from the corresponding kernel modules, so that they can be installed automatically when the hardware is present.
  Some other firmware packages have also Supplements, too (usually specified in the spec manually).

Kernel Modules and Firmware:

• The load of a firmware isn't always shown in the kernel log.
  It's visible only when the debugging is enabled on the kernel firmware loader, e.g. passing firmware_class.dyndbg=+p boot
  option.

Virtual Machines:

• Usually no need of firmware for a VM. Due to that reason, kernel-default-base, which is a reduced version of kernel-default package for VM and co, doesn't have Recommends of kernel-firmware unlike the standard kernel-default package.
  (Hence, for example, SUSE Micro doesn't install firmware files as default, and it caused problems when you install it on a bare
  metal as a side-effect.)

@wfeldt writes about the inst-sys firmware:

The current set is https://github.com/openSUSE/installation-images/blob/master/etc/config#L46-L50 - individual packages might no longer exist, though. Like these ucode packages.

There is also a planned move away from kernel-firmware, see https://bugzilla.suse.com/show_bug.cgi?id=1214789 that's what Takashi meant in his comment in your gist.

In general, the inst-sys checks with modinfo and only adds firmware blobs that are required by modules. Hence the list of actually installed firmware files is shortened.

https://github.com/openSUSE/installation-images/blob/master/etc/config#L46-L50

; extra firmware packages
[Firmware]
default = kernel-firmware,adaptec-firmware
i386    = kernel-firmware,adaptec-firmware,ipw-firmware,iwl4965-ucode,iwl5000-ucode,atmel-firmware,ralink-firmware
x86_64  = kernel-firmware,adaptec-firmware,ipw-firmware,iwl4965-ucode,iwl5000-ucode,atmel-firmware,ralink-firmware

Some Comments

that also means a lot of RAM usage

I do not think so, if the firmware file is not used (because the system does not contain appropriate hardware) then it stays in the Live image unused. Live ISO uses transparent compression and the files are not read until really needed. (Unless you use the rd.live.ram=1 dracut boot option which loads the whole image into RAM, see the documentation.)

Would it even save disk space, or is the inst-sys filesystem compressed anyway?

The inst-sys is already compressed using SquashFS. I do not know which particular compression we use but SquashFS also supports the XZ compression. So very likely there will be no difference between using the normal firmware and the XZ compressed files.

I think the only problem is that the ISO image gets too large. Currently it is ~1GB which is quite a lot just for a "simple" installer. Especially if in most cases you do not need any firmware.

Needed FW

The question is also which firmware we really need? E.g. some graphical cards can work in VESA framebuffer compatibility mode, the UI could be a bit slow but it should work. Similar for some storage devices, IIRC many SATA controllers can work in IDE compatibility mode.

If some device can work without firmware in some backward compatible (although slower) mode then we do not strictly need the firmware. But if it is needed also in the compatibility mode, then it won't help.

Solutions?

I think there are basically two solutions, load the firmware from external sources when needed or build multiple ISO flavors.

Load on Demand

Because the installer needs an access to installation repository and the Live ISO uses a writable overlay (so the zypper install command works as expected) we could possibly download and install the firmware packages on the fly.

However, there are some problems:

• Chicked egg problem for network firmware, if the network device needs a firmware it must be already present in the system
• How do we detect that a firmware is needed? How do we find out which package we should download and install?
• After installing a firmware package, how to tell the kernel that the needed firmware has been just installed? Do we need to notify the kernel or does it work automagically?
• It would not work in isolated systems (no network) unless the ISO contains the installation RPMs as well

Build Several Images

Alternatively we could build several ISO images:

• Small one without firmware or with just the very basic firmware set (like ucode-amd/ucode-intel ??), for VMs and standard bare metal
• Bigger one with some additional less but still commonly used firmware
• Full with everything, so it would work even with rare HW

(And when working on that maybe we could build a really small image without X and Firefox for remote installations.)

Or 3rd solution: carefully go through the all included firmware and check if we could remove some not needed ones. Not sure how much that would help... (Like sound cards are not needed for installation so we could possibly remove alsa-firmware?)

Some notes from a brainstorm session

Generally the problem affects two scopes:

• The final installed system
• The int-sys (ie. the system in which Agama runs in order to deploy the final system)

We will focus on the latter, since the former is a general topic that should be handled at distribution level (it has many implications beyond the installation itself).

The main goal is to keep the inst-sys as small as possible. Despite the obvious problem of needing to download 1GiB of image only to start the installation (the repositories are not included there), the size of the inst-sys has many implications in scenarios like PXE, network deployment, loading the image into RAM and so on.

It makes sense to provide several versions of the int-sys:

• One with no firmware. That's actually enough to install in most virtual machines.
• One with a reasonable set of firmware that makes installation possible on most systems.
• A full-loaded one including all firmware available in the distribution.

Installation-images already does a great job minimizing the footprint of the packages included in the inst-sys. It even loads dynamically some parts on the int-sys on demand during execution.

Regarding the firmware packages, installation-images contains a certain logic to filter only the firmware files that are needed to load the kernel modules. Although that logic is not something we can trivially duplicate in Kiwi, we can use the resulting list of files as an input to tweak the current Agama-Live images.

So the first step would likely be to rely on that to generate a flavor of the Agama-Live ISO which only contains a curated list of firmware packages. Just to see the difference in size and decide what are the next steps.