Use old phone as a second display: USB GUD with postmarketOS

GUD.md

Demo

Xiaomi Redmi 2 with resolution 1280x720 (GUD RGB565 with compression)

9.mp4

10.mp4

Intro

GUD is an open USB display protocol. As it is USB, there is a host and a device. The device acts as a display for the host.

The host should have a driver for this protocol and the device should support this protocol.

For linux host, the driver has already been mainlined since v5.13. The device would be added as a drm device (e.g. /dev/dri/card1) then Xorg/Wayland would use this as their screen.

The device can be either a dedicated hardware like STM32 and Rpi Pico implementing this protocol; or another linux box acting as USB Device where the kernel have GUD device functionality. In Linux term, this is called USB Gadget. USB GUD Gadget has not been mainlined.

For phones running postmarketOS, they use linux kernel, they have USB and they have a screen. They are quite suitable for this usage.

Also, phones have touchscreen, audio and mic, which can also be driven by USB Gadget and serve the host. With proper configuration, this display can be a touchscreen display! One possible application is this touchscreen display to be a customizable keyboard with custom functions; Or you may mirror your display to the phone and use the phone to touch it (Gaming?).

One more note, with usbip, the whole thing can be wireless. The host uses vhci-hcd and the device uses usbip-vudc, then it is IP world: you can go over wifi, cellular etc.

Setup

Host

The host is Arch Linux with the lastest Linux kernel (5.16.1-arch1-1). Note that Host GUD driver hitted mainline at v5.13, so Arch Linux with linux-lts (5.15.15-1-lts) will also work.

Check that CONFIG_DRM_GUD is m or y.

$ zcat /proc/config.gz | grep GUD
CONFIG_DRM_GUD=m

Then the host side is done. You can just plug-and-play*.

* (Click to expand) There are issues with Xorg, Nvidia and DRM hotplug, which I haven't looked into yet.

On my setup, When I was using Intel graphics card, GUD works fine (as demoed above). Only when I plug in/out USB GUD device, Xorg would stuck for a while or just crash. After crash I have to restart my Xorg server.

Process 730 (Xorg) of user 0 dumped core.

When I was using Nvidia and modesetting (nvidia-drm.modeset=1, Identifier "intel" Driver "modesetting"), things became non-trivial and kernel panicked for a while like:

...many warnings
[  943.637857] WARNING: CPU: 11 PID: 300 at drivers/gpu/drm/drm_gem_shmem_helper.c:299 drm_gem_shmem_vmap+0x18c/0x1a0
...kernel backtrace
[  944.074808] WARNING: CPU: 4 PID: 140 at drivers/gpu/drm/drm_gem_shmem_helper.c:299 drm_gem_shmem_vmap+0x18c/0x1a0
...kernel backtrace
[  944.124816] WARNING: CPU: 4 PID: 140 at drivers/gpu/drm/drm_gem_shmem_helper.c:299 drm_gem_shmem_vmap+0x18c/0x1a0
...kernel backtrace
[  946.372404] kernel BUG at drivers/dma-buf/dma-buf.c:67!
[  946.372432] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
[  946.372445] CPU: 6 PID: 891 Comm: Xorg:disk$0 Tainted: P        W  OE     5.16.1-arch1-1 #1 49bbb8d20d0329f70e47963ef5feb4a66c3cd442
[  946.394975] Fixing recursive fault but reboot is needed!

Device

For device I used a second hand $10 Xiaomi Redmi 2. Any phone with UDC and a screen would satisfy the need; Because you need to compile and replace the kernel, some phones with little mainlining support may not use the feature.

Booting up postmarketOS for this device is trivial. One thing to note is that you need to setup a second access method like ssh over WIFI or serial port, because when configuring USB GUD, USB Networking/USB Serial Port would be disabled.

Replace Kernel

1. Steps below are done on the host.
2. For this device, the kernel src is at here.
3. After cloning, you switch to a tag corresponding to your current running kernel (in my case, v5.15-msm8916).
4. Then you need to apply patches from here and (here, if >5.10).
5. Compile the kernel as guided at here, note the step below.
6. After generating the default config, you should use make menuconfig to enable USB_CONFIGFS_F_GUD.
7. Then flash the kernel to your phone.

Configure USB GUD Gadget

postmarketOS mostly have USB Networking Gadget enabled, which conflicts* with USB GUD Gadget.

*(I know you can set them both up, but we have to disable USB Networking to spare the UDC then enable it again with two functions like GUD and Networking, which means USB Networking, namely your connection have to stop for a while)

USB Gadget are configured through a interface called configfs, usually located at /sys/kernel/config.

How to config USB Gadget using configfs is introduced in gud script and this article.

Also, how to configure USB GUD Gadget is introduced in that script.

Note: g1 gadget are used for USB Networking after boot and its function path is different. So you can not just use gud stop to stop USB Networking; You need to manually cleanup.

1. Stop tinydm. (I did so for drm device to become available)
2. Stop USB Networking Gadget
3. Start USB GUD Gadget (remember to read dmesg if you fail to enable)
4. Plug the phone to the host
5. On the host, you may have to use xrandr or arandr to enable this screen.
6. Play it!

Future works

(of course, can not be done by only one person)

1. USB GUD Gadget has not been mainlined, which needs further involvement (cleanup).
2. USB GUD Gadget profiling. Currently it consumes only around 30% of one core. Want to know the critical path. Higher framerate/lower response time wanted. With USB 2.0 bandwidth and this CPU, 60fps can be achieved.
3. More advanced GUD protocol. Phones have quite strong CPU/GPU, may use codec (kernel space codec?). Compression methods can also be explored.
4. DRM hotplugging (why kernel panic)
5. Nvidia modesetting stuff (why kernel panic).
6. Xorg stucking/crashing should be handled.
7. Drive touchscreen (USB HID Gadget, maybe?) and proper configuration (Xorg.conf coord map?)
8. Drive audio (USB DAC Gadget, maybe?)
9. Drive mic
10. Packaging scripts for postmarketOS/pmbootstrap

UserspaceAndFFS.md

Beside f_gud.ko, there is f_fs.ko

Motivation:

1. f_gud inside kernel space: codec is hard, directly accessing drm interferes with Xserver/other programs
2. gud client in user space: use existing codec, can use compositor features (e.g. use a small window to show another screen)

Implementation:

FunctionFS can deliver USB traffic to the userspace using filesystem API. Then we run a daemon reading and writing to this filesystem. Then the daemon decode the received data using userspace APIs (e.g. gpu) and let the compositor display it.