This morning, my laptop got stuck for a minute during boot in the typical waiting for... message coming from systemd.
Of course, the easiest and funniest approach here would be blaming systemd and writing an angry tweet about how it sucks, but let's see if we can do better.
First, once the system booted, let's find out which services failed:
[localhost ~]$ systemctl --failed
UNIT LOAD ACTIVE SUB DESCRIPTION
● systemd-udev-settle.service loaded failed failed udev Wait for Complete Device Initialization
● vboxdrv.service loaded failed failed VirtualBox Linux kernel module
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
2 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
vboxdrv is expected to fail, as that's just a leftover that I'm too lazy to remove, but systemd-udev-settle shouldn't have failed. Let's take a closer look:
[localhost ~]$ systemctl status systemd-udev-settle.service
● systemd-udev-settle.service - udev Wait for Complete Device Initialization
Loaded: loaded (/usr/lib/systemd/system/systemd-udev-settle.service; static; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2017-08-29 08:11:18 CEST; 16min ago
Docs: man:udev(7)
man:systemd-udevd.service(8)
Process: 2362 ExecStart=/usr/bin/udevadm settle (code=exited, status=1/FAILURE)
Main PID: 2362 (code=exited, status=1/FAILURE)
Aug 29 08:09:17 localhost.localdomain systemd[1]: Starting udev Wait for Complete Device Initialization...
Aug 29 08:11:18 localhost.localdomain systemd[1]: systemd-udev-settle.service: Main process exited, code=exited, status=1/FAILURE
Aug 29 08:11:18 localhost.localdomain systemd[1]: Failed to start udev Wait for Complete Device Initialization.
Aug 29 08:11:18 localhost.localdomain systemd[1]: systemd-udev-settle.service: Unit entered failed state.
Aug 29 08:11:18 localhost.localdomain systemd[1]: systemd-udev-settle.service: Failed with result 'exit-code'.
So this service failed, aproximately two minutes after it started udevadm settle. A quick look at udevadm manpage shows that, by default, it'll wait for two minutes for udev's queue to be empty.
This means udevadm settle is timing out, because there's a udev operation stuck. Let's see if we can find more:
[localhost ~]$ journalctl --unit systemd-udevd --since "2017-08-29 08:09:00"
(...)
Aug 29 08:10:18 localhost.localdomain systemd-udevd[2355]: seq 2461 '/devices/platform/i8042/serio1/serio2/input/input6' is taking a long time
(...)
Aug 29 08:12:18 localhost.localdomain systemd-udevd[2355]: seq 2461 '/devices/platform/i8042/serio1/serio2/input/input6' killed
Aug 29 08:12:18 localhost.localdomain systemd-udevd[2355]: worker [2377] terminated by signal 9 (KILL)
Aug 29 08:12:18 localhost.localdomain systemd-udevd[2355]: worker [2377] failed while handling '/devices/platform/i8042/serio1/serio2/input/input6'
Aug 29 08:12:18 localhost.localdomain systemd-udevd[2375]: Failed to write 'sensitivity' attribute for '(null)': No such device or address
So udev was trying to write the sensitivity attribute for a i8042 device (probably this laptop's trackpad), and the latter was misbehaving.
In the end, this "systemd issue" has turned to be a hardware issue. As usual ;-)