Last active
August 14, 2026 12:01
-
-
Save cfriedt/7c8d8d59d62345caee8127d71e9fe86b to your computer and use it in GitHub Desktop.
zephyr-process-support.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # POSIX_MULTI_PROCESS support for Zephyr (posix-next) | |
| ## Implementation status (branch `posix-multi-process`) | |
| **Naming/layering rules (Chris, 2026-08-14, apply to all process work):** | |
| - `k_`/`K_` prefixes: APIs and constants that do not allocate or free at runtime. | |
| The kernel layer never refers to `sys_` or POSIX symbols. | |
| - `sys_`/`SYS_` prefixes: APIs and constants involved in runtime allocation or | |
| deallocation (lib/os). May use `k_`; never refers to POSIX. | |
| - POSIX declarations derive their constants from the `K_`/`SYS_` counterparts and | |
| wrap the `k_`/`sys_` functions. Existing precedent: `k_spawn()` takes a | |
| caller-provided thread + stack (no allocation, correctly `k_`); the | |
| pool-allocating spawn belongs at `sys_spawn()`. There is no `k_fork()`: POSIX | |
| `fork()` wraps `k_clone()` with the COW flags directly (fork-as-a-name lives in | |
| libc/POSIX, as on Linux). | |
| **HISTORY LINEARIZED 2026-08-12** (force-pushed; pre-rewrite tip preserved as | |
| `backup-prelinearize-29863b116`). The 16 milestone-ordered commits were rebuilt | |
| into 11 layer-ordered commits on `520f32e1a`, per Chris's direction (native | |
| impl → native tests/docs → POSIX impl → POSIX tests → POSIX docs; fixes to | |
| main-owned patches first, folded into the patch that introduced the issue): | |
| | Commit | Layer | What it is | | |
| |--------|-------|------------| | |
| | `e53c7adce` | fix-for-main | Both k-signal fixes **folded into `k-signal.patch` itself** (purge dying thread's pending signals + skip delivery under `k_is_in_user_syscall()`); standalone fix patches deleted; context-only regens of k-thread-cancel/k-timer-clock/k-timer-signal/k-msgq-notify/sys-aio. Fold candidate for main's `4978872c2` on next main rewrite. | | |
| | `c86dad7be` | native kernel | `k-process-core.patch`: full k_process API surface (now introduces the **low-bit `K_PID_PGRP()` encoding** directly; waitpid-tag fix patch folded in). | | |
| | `c0ff4f768` | native kernel | `k-process-exit-wait.patch`: lifecycle, k_spawn, exit/wait, init start, `tests/kernel/process`. | | |
| | `32aca11de` | native kernel | `k-process-sigchld.patch`: SIGCHLD on child zombify. | | |
| | `9d2bd0a79` | native kernel | `k-process-pgrp-session.patch`: k_setpgid/k_kill_pgrp/k_kill + process_sig_target. | | |
| | `2bccdcbbc` | native libc | `libc-multi-process-headers.patch`: sys/wait.h shims, CLD_*. | | |
| | `ba6bdb650` | native libc | `libc-common-exit.patch`: common exit()/atexit(). | | |
| | `2b4d42fda` | native libc | `libc-common-under-exit.patch`: `_exit()` → `k_exit()`. | | |
| | `d1566d030` | POSIX impl | All module impl (headers, one-file-per-function, final kill()/atfork shape). | | |
| | `b3aa3c177` | POSIX tests | tests/posix/multi_process (+exit app), signals/threads_base updates. | | |
| | `5927fea93` | POSIX docs | doc yaml data (sys/wait.h mapping, typo-alias removal). | | |
| Verified: applied zephyr tree **byte-identical** to the pre-rewrite verified | |
| tree; 119-patch series applies from pristine v4.4.1; patches.yml sha256s | |
| consistent at every commit; smoke twister 32/32 (kernel process+signal, posix | |
| multi_process+signals on qemu_cortex_m3 + qemu_riscv32). Follow-up flagged: | |
| native layer still has no Zephyr-tree docs page (doc/kernel/services/) — add | |
| during the fine-tuning pass. Series count is now 119 (three standalone fix | |
| patches folded away). | |
| Pre-rewrite milestone history (superseded, kept for context): | |
| | Commit | Milestone | What landed | | |
| |--------|-----------|-------------| | |
| | `d99a4a165` | M0a | Kernel process API surface + `<sys/wait.h>`/`sys/types.h` headers (declarations, stubs). Now also carries the syscall-gating folded in during M1c. | | |
| | `5faa513a8` | M0b | Defined single-process fallback behaviour for the whole option group; real `pthread_atfork` (elastipool registry); 23 ZTESTs; exit console app; `muti_process` typo fix. | | |
| | `f32f5e43c` | (libc) | `exit()`/`atexit()` promoted to common libc (`COMMON_LIBC_EXIT`), user-mode-safe (atomics + `Z_LIBC_DATA`), bounded CAS registry. Fixes picolibc stub-atexit. | | |
| | `6739f0706` | M1 (a+b) | `k_process` lifecycle: thread-group membership, static-process start, `k_exit`/`k_waitpid`, **`k_spawn`**; reserved swapper/init/kthreadd; `tests/kernel/process`. | | |
| | `4ffcc97a2` | M1c | POSIX layer backed by real `k_process` (`POSIX_MULTI_PROCESS` selects `PROCESS`); identity/wait wrappers over `k_*`; process-aware `kill`/`sigqueue`; process syscalls gated `CONFIG_PROCESS`; `CLD_*` in libc shims. | | |
| | `49beb4afc` | M1d+CI | k_pid_of/k_process_by_pid syscalls (userspace fix). | | |
| | `c61f32e83` | M1e | `_exit()` → `k_exit()` (`COMMON_LIBC_UNDERSCORE_EXIT`); a process's libc `exit()` terminates that process alone. | | |
| | `79f07c6fa` | M1f | init process (pid 1) started at boot from a SYS_INIT static process. | | |
| | `39b4322cd` | **M2** | **Process groups + sessions**: new `k_setpgid`/`k_kill_pgrp` kernel syscalls (`k-process-pgrp-session.patch`); `setpgid()` wrapper; `kill(pid<=0)` group fan-out; kernel `test_setpgid`/`test_kill_pgrp` + POSIX `test_setpgid` + `kill(0,0)` signals check. | | |
| **M2 COMPLETE** (2026-08-11). Process-group/session control done: `k_setpgid` moves a | |
| process (self or child) into a group under the same-session / not-a-session-leader rules; | |
| `k_kill_pgrp` fans a signal out to every member's group leader (CONFIG_SIGNAL-gated, | |
| `-ENOSYS` without). Module `kill()` now handles `pid<=0` (pid 0 = caller's group, pid<-1 = | |
| group `-pid`); **kill() is process-directed, pthread_kill() thread-directed** (user | |
| guidance). Verified green on the full runci matrix *except* qemu_riscv32, which fails the | |
| **pre-existing M1 spawn/waitpid breakage** (confirmed at the M1f baseline with M2 stashed — | |
| not an M2 regression; riscv32 was never in M1's verified set). Series now **118 patches**. | |
| **M2.1 (review refinements, 2026-08-11)** — folded into the M2 commits (now `cf83dcb53` | |
| M2 + `57168beea` purge; series **120 patches**), per Chris's feedback that this belongs | |
| below the line: | |
| - **`k_kill(pid, signo)` kernel syscall** owns the full POSIX pid encoding; module | |
| `kill()` is a thin wrapper. `kill()` is process-directed, `pthread_kill()` thread-directed. | |
| - **Process-directed dispatch to the first *unblocked* thread** (`process_sig_target`), | |
| used by `k_kill`/`k_kill_pgrp`; delivered to an eligible member, not always the leader. | |
| - **`z_sig_purge_thread()`** frees a dying thread's queued-but-undelivered signals from the | |
| halt path (new `k-signal-purge-thread-on-exit.patch`); `k_sig_lock` made a leaf (raise | |
| after unlock) so the purge is safe under the scheduler lock. | |
| Verified on the runci matrix (all green except pre-existing qemu_riscv32; the one | |
| qemu_x86_64 kernel.process.init "fail" is an SMP harness flake — the suite passes 13/13). | |
| **M2.2 (root-cause fix, 2026-08-11)** — commit `571e6620c`, series **121 patches**: | |
| - **Nested-syscall signal-delivery fault ROOT-CAUSED AND FIXED** | |
| (`k-signal-nested-syscall-delivery.patch`). The generated `z_<syscall>` wrapper runs | |
| `z_syscall_sig_check()` on its kernel-direct branch; when `k_kill()` (a user syscall) | |
| calls the `k_sig_queue()` wrapper in kernel context, that ran signal delivery | |
| (`z_sig_trampoline`→`k_sig_return`→`arch_sig_post`) against the outer syscall frame with | |
| no matching `arch_sig_pre()`, corrupting it. Fix: `z_syscall_sig_check()` returns early | |
| under `k_is_in_user_syscall()` (the arch userspace-return path delivers instead). | |
| - **`kill()` is now FULLY THIN** over `k_kill()` for all pid encodings — the M2 self-delivery | |
| workaround is removed. `tests/posix/signals` passes on qemu_x86_64 (SMP userspace). | |
| **Known follow-ups:** | |
| - **SIGCHLD to an ignoring/never-consuming parent** leaks a slab slot (distinct from the | |
| thread-exit purge); the guidance is "kernel-mode parents must explicitly unmask the signals | |
| they care about" (test-side). See [[sigchld-ignoring-parent-leak]]. | |
| - **threads_base.userspace atfork fault** — RESOLVED (`b624d7ebf`): the M0b elastipool atfork | |
| registry was kernel .bss touched from user mode; moved to the libc app-memory partition | |
| (`static Z_LIBC_DATA` array + atomic, atexit-style). | |
| - **qemu_riscv32 spawn/waitpid** — RESOLVED (`846cda184`): K_PID_PGRP selector used sign discrimination; process pointers at 0x80000000 are negative on 32-bit. Now low-bit tagged (`k-process-waitpid-pgrp-tag.patch`). | |
| **Note**: branch hashes changed (CI fixes folded into M0a/M0b, syscall fix bundled on top). Full-matrix twister via CI surfaced three issues now fixed: `<sys/wait.h>` W* redefinition (#undef+redefine), exec/fork/setsid tests unsafe on host libc (skip under NATIVE_LIBC), and `k_pid_of`/`k_process_by_pid` hanging in user mode on riscv64 PMP (made syscalls). | |
| **Verified**: `tests/kernel/process` 11/11 and `tests/posix/multi_process` 23/23 on | |
| qemu_cortex_m3 (kernel-mode, userspace, minimal libc), qemu_x86, qemu_riscv64; signals | |
| 19/19, realtime_signals green; process syscalls absent when `CONFIG_PROCESS=n`; series | |
| applies from pristine (**117 patches**). | |
| **M1 COMPLETE** (M1a–M1f). Prior **Remaining** note (now done): M1e — SIGCHLD emission on child zombify (needs `k_sig_info` | |
| `si_pid`/`si_status` extension via a new `k-process-signal` patch, plus halt-path-safe | |
| deferral: set per-process `sigchld_pending` under the scheduler lock, submit a shared | |
| scan-work whose handler re-validates `ZOMBIE` state under `process_lock` to avoid | |
| slot-reuse races). **M1f** — init-process start (a `SYS_INIT`-launched static pid-1 process). | |
| **Housekeeping debt**: M1c and M1d bundle their kernel-patch regenerations into the | |
| feature commit rather than folding into the introducing commits (autosquash conflicted | |
| on wholesale patch regeneration + entangled `patches.yml`); a pre-upstream cleanup | |
| rebase should fold them. History was rewritten (M0b/M1), so the open PR needs a | |
| force-push. | |
| **Next milestones** (see roadmap): M2 `k_spawn`+`posix_spawn`, M3 exec via LLEXT, | |
| M4 COW fork on MMU, M5 groups/sessions + SIGCHLD completeness. | |
| ## Context | |
| The posix-next module implements POSIX Subprofiling Option Groups for Zephyr. The | |
| `POSIX_MULTI_PROCESS` group requires: `_Exit, _exit, assert, atexit, clock, execl, execle, | |
| execlp, execv, execve, execvp, exit, fork, getpgrp, getpgid, getpid, getppid, getsid, | |
| setsid, sleep, times, wait, waitid, waitpid`. Today only `getpid()` (constant 42), | |
| `times()` (no child times), and `sleep()` exist; Zephyr has **no process concept at all**. | |
| The goal is to introduce a real process abstraction into the Zephyr kernel (as upstreamable | |
| patches in the module's patch series), an exec loader, common-libc ISO C pieces, and the | |
| POSIX module glue + tests. The user (Chris) will hand-implement much of the kernel/arch | |
| work; this plan is the reviewable blueprint. | |
| ## Decisions (made by user) | |
| - **Process model — tiered**: a first-class `k_process` kernel object (thread group + | |
| memory domain + pid + parent/children + exit status) that works on MPU **and** MMU | |
| targets. Full address-space semantics only where MMU per-domain page tables exist. | |
| - **Static definition idiom (user requirement)**: processes are definable at build time | |
| via a `K_PROCESS_DEFINE()` macro placing objects in an iterable section, like other | |
| Zephyr kernel objects — the kernel manages statically allocated processes the same | |
| way. The system reserves three pids, all kernel-defined with that same macro: | |
| **pid 0 = swapper/idle process** (per-CPU idle threads; sleeps when nothing else is | |
| runnable), **pid 1 = init process** (parent of all userspace threads + processes), | |
| **pid 2 = kthreadd** (parent process of all kernel threads). | |
| - **init process start**: the init process (pid 1) is a statically defined process | |
| started at boot (`SYS_INIT`/`K_PROCESS_DEFINE`), independent of what the image's | |
| `main()` does; the image `main()` keeps today's semantics. | |
| - **fork() — COW on MMU arches only** (x86, arm64). On MMU-less targets (MPU/PMP), | |
| `fork()` returns **ENOSYS** and **`posix_spawn()` is the process-creation story**. | |
| - **exec() — LLEXT from filesystem** (`fs_loader`), PATH search for `execlp`/`execvp`. | |
| - **Bring-up platforms**: qemu_x86 (MMU/COW), qemu_cortex_a53 (MMU/COW), | |
| qemu_riscv64 (PMP → spawn tier), mps2/an385 (MPU → spawn tier). | |
| ## Key exploration findings (ground truth) | |
| ### Kernel (zephyr/ @ v4.4.1 + 116-patch series applied) | |
| - `k_mem_domain` already models an address space with a thread membership list; | |
| membership inherited at `k_thread_create()`. Per-domain page tables exist on x86 | |
| (`arch_mem_domain.ptables` + CR3 swap) and arm64 (per-domain TTBR0 + ASID). | |
| - **Blocker #1**: kernel/Kconfig.vm invariant — all page tables share one VA→PA map | |
| (permission views only); `virt_region_bitmap` in kernel/mmu.c is global; no page-frame | |
| refcounting (needed for COW). | |
| - `typedef k_tid_t k_pid_t` exists (thread.h:417); explicit `TODO(k_process)` markers at | |
| kernel.h:1995 (k_timer sig_fn_domain/perms_src) and sys/timer.h:51,104. | |
| - Signal subsystem (k-signal.patch) is process-shaped already: `k_sig_queue(k_pid_t,…)`, | |
| `K_SIG_CHLD` (default Ign), user-mode delivery on arm/arm64/riscv/x86. Gaps: `k_sig_info` | |
| lacks si_pid/si_status; signal action DB is global not per-process; STOP/CONT ignored. | |
| - Thread lifecycle: `thread_state` currently uses all 8 bits — if a per-thread state | |
| bit is ever needed, **widen `thread_state` to `uint16_t`** (user decision). The | |
| implemented design sidesteps this: the zombie lives in `k_process` after the threads | |
| are freed, so no new thread_state bit is required. `k_thread_exit/result/rejoin/ | |
| cancel/cleanup/detach` + `sys_thread_recycle()` reaper exist from the series. | |
| - fdtable: single global static array (lib/os/zvfs/zvfs_fdtable.c) with per-slot | |
| `K_OBJ_FILE` objects; FD_CLOEXEC stored (`fd_flags`) but never acted on; CWD global | |
| (zvfs_cwd.c). k_object perms are per-thread bitmaps (`CONFIG_MAX_THREAD_BYTES`=2 ⇒ 16 | |
| userspace threads) → **becomes 16 userspace *processes*** once permissions move to | |
| `k_process` (see Per-process fds/cwd and the kernel-risk note). | |
| - `typedef k_tid_t k_pid_t` (thread.h) — **kept as-is (user decision)**: `k_sig_queue` | |
| stays thread-directed, process APIs take `struct k_process *` explicitly, and | |
| `kill`/`sigqueue` resolve a pid to its process and deliver to the group leader. This | |
| is preferred over flipping `k_pid_t` to a process handle; it keeps the existing | |
| signal path intact and is what is implemented through M1e. | |
| - LLEXT: loads ET_REL/relocatable/shared (no ET_EXEC/PIE, no argv/envp/entry ABI); has | |
| `llext_add_domain()`, fs/buf loaders, per-image mem partitions; working | |
| load→domain→K_USER-thread recipe in tests/subsys/llext/src/test_llext.c:116-202. | |
| ### libc | |
| - `_exit` is a per-libc spin-forever stub (minimal's is NON-weak); `_Exit` exists nowhere; | |
| atexit registries are per-libc (minimal 8-slot; picolibc PICOEXIT; newlib reent). | |
| - `abort()` is already a common-libc unit — `libc-isoc-signal-handling.patch` and | |
| `libc-common-tmpfile.patch` are the established patterns for common-libc promotion. | |
| - picolibc `clock()` lights up via `times()` + `CLOCK_PROVIDED`; newlib needs `_times`. | |
| - After `main()` returns, `bg_thread_main` runs no atexit, propagates no status. | |
| - newlib hook pattern: `#ifndef CONFIG_POSIX_MULTI_PROCESS` around `_getpid` etc. | |
| ### Module | |
| - unistd.h already declares all 17 MULTI_PROCESS functions with Doxygen. | |
| - `sys/wait.h` exists nowhere (toolchain copies declare only wait/waitpid; no waitid/ | |
| idtype_t/id_t/P_*). `id_t`/`idtype_t` absent from sys/types.h. | |
| - `kill()` overloads pid>0 as a `k_thread *`; `POSIX_THIS_PID`=42 shared with signals. | |
| - `environ` is one global `char **` (options/shared/env_common.c); `pthread_atfork()` is a | |
| documented no-op that must become real. | |
| - siginfo_t already has si_pid/si_status, SIGCHLD, CLD_* — plumbing gap is kernel-side. | |
| - tests/posix/multi_process exists (assert/atexit/clock/getpid/sleep/times, six twister | |
| variants; scenario-name typo "muti_process" to fix). | |
| ## Constraints | |
| - New kernel work = **new patches** in zephyr/patches/zephyr/ registered in patches.yml | |
| (plain unified diffs, upstreamable; never touch frozen patches). | |
| - One ZTEST per unit (`test_<function>`), aspects as section helpers. | |
| - Copyright: novel .c → FPES form; headers/tests/Kconfig → ZPC. | |
| - runci.sh for full twister runs; no build-dir litter. | |
| - User hand-implements the deep kernel/arch pieces; module/libc/test work can be delegated. | |
| --- | |
| ## Design: exec loader + libc integration | |
| ### Process ABI (kernel-defined contract; user requirements) | |
| Exec support is a **later phase** (see roadmap) — statically defined | |
| `K_PROCESS_DEFINE()` processes come first, including **prelinked-blob processes** | |
| (prelinked ELF → binary blob linked into firmware, entry = numeric address), which | |
| give code-in-own-partition process coverage with **no LLEXT and no filesystem**. | |
| LLEXT-from-filesystem is strictly an *optional* dynamic image source layered on the | |
| same kernel machinery. When exec lands, layering is: | |
| - **The Zephyr kernel defines the process ABI framework**: the entry-point contract | |
| and the **process memory layout**, including where the program-arguments/environment | |
| block lives (Linux-style: a kernel-specified args/env block — argc, argv/envp | |
| vectors + packed strings, ARG_MAX-bounded — placed at a defined location, e.g. top | |
| of the initial stack). The kernel *places* this block at process start; it is | |
| **owned by the application** thereafter — the kernel never reaches back into it. | |
| - **crt0 is C-runtime territory, below POSIX**: the C library (minimal / picolibc / | |
| common), not the POSIX module, provides the crt0 that implements the kernel's entry | |
| contract — parses the args/env block per the ABI, initializes libc state, sets up | |
| `environ`/argc/argv, calls `exit(main(argc, argv, envp))`. libc and POSIX both act | |
| on the kernel-defined layout (shared, synchronized access to `environ` via libc's | |
| storage — the POSIX module's getenv/setenv become consumers of that layout rather | |
| than owners of a module-global). | |
| - The build helper that produces executables (cmake wrapper over `add_llext_target()` | |
| that links crt0 + static libc) is therefore Zephyr-side, not module-side. | |
| ### Executable format (when exec lands) | |
| - **ET_REL relocatable objects** (`CONFIG_LLEXT_TYPE_ELF_RELOCATABLE`) — the only LLEXT | |
| format common to all four platforms (`LLEXT_TYPE_ELF_OBJECT` is `!RISCV`). PIE/ET_DYN | |
| deferred (multi-month loader effort). | |
| - Apps write `int main(int argc, char **argv, char **envp)`; libc crt0 exports | |
| `_start` per the kernel ABI; missing `_start` ⇒ `ENOEXEC`. | |
| ### execve flow (failure-safe ordering; sys layer — POSIX is not the right layer) | |
| The load/commit split lives **below POSIX** as `sys_process_exec_load()` / | |
| `sys_process_exec_commit()` in lib/os; the module's `execve()` only marshals and calls | |
| them. Load (can fail cleanly: stage argv/envp → obtain image from the configured | |
| source — static blob registry or, optionally, LLEXT fs loader → find entry; on | |
| failure the old image is intact) then commit (point of no return): | |
| 1. Build new `k_mem_domain` + `llext_add_domain()` **before** killing siblings (so | |
| `-ENOSPC` is still a clean failure). No `z_libc_partition` needed — executables | |
| carry their own statically linked libc state in the image partitions. | |
| 2. `k_process_exec_begin(proc)` — terminate other threads in the group (POSIX). | |
| 3. **FD_CLOEXEC sweep**: new `zvfs_close_on_exec()` — first consumer of the stored flag. | |
| 4. Reset caught signal dispositions to SIG_DFL (keep IGN/mask/pending). | |
| 5. `llext_bringup()`; create **fresh main k_thread + stack** in the same k_process | |
| (avoids in-place stack-rewind arch magic), start at trampoline → `_start(args)`, | |
| K_USER drop when USERSPACE; old thread detaches old image (deferred | |
| `llext_teardown/unload` on workqueue) and self-exits. | |
| - Image outlives the entry thread: the **reaper** (wait/orphan auto-reap) tears down the | |
| llext image — k_process needs an image-destructor hook (kernel interface requirement). | |
| - `fexecve`: ~60-line module-side `llext_loader` adapter over zvfs read/lseek. No `#!`, | |
| no exec-bit, no execvp sh-fallback in phase 1 (documented deviations). | |
| - PATH search: `getenv("PATH")` falling back to `confstr(_CS_PATH)`. | |
| ### Symbol linkage — minimal surface (user requirement) | |
| The export surface is **predominantly Zephyr kernel APIs**: enable | |
| `LLEXT_EXPORT_SYMBOL_GROUP_SYSCALL` only — no libc symbol group, no POSIX-module | |
| export unit. Executables **statically link their own libc + POSIX wrapper library** | |
| (crt0 + libc.a + the module's thin `k_*()`-wrapping POSIX functions, partial-linked | |
| into the ET_REL image), so only `k_*()` syscalls cross the process boundary. Benefits: | |
| minimal ABI surface, per-process libc state for free (each image carries its own | |
| `.data/.bss` — no shared `z_libc_partition` wart), and clean fit with the | |
| POSIX-wraps-kernel separation. Cost: larger executables and partial-link-with-archive | |
| build mechanics (resolve `libc.a` members into the relocatable — verify the | |
| EDK/`add_llext_target` flow supports archive resolution; implementation detail to | |
| prove in the exec phase). Compiler intrinsics (`__aeabi_*`, `memcpy`) must resolve | |
| locally from the static link — audit during bring-up. | |
| ### Memory-protection fit | |
| - qemu_x86 / qemu_cortex_a53 (MMU): fresh domain per process, full COW tier; verify | |
| `llext_adjust_mmu_permissions()` in the per-process-domain flow (may need a small patch — | |
| it is domain-global today). | |
| - qemu_riscv64 (PMP): domain isolation feasible but tight; spawn tier. | |
| - mps2/an385 (3 MPU slots): 4 llext partitions + libc **cannot fit** — phase 1 runs exec'd | |
| processes with `POSIX_EXEC_ISOLATION_NONE` (correct semantics, no HW isolation; honest | |
| single-image trust model). Phase 2: `llext-merged-mem-partitions.patch` | |
| (TEXT+RODATA→RX, DATA+BSS→RW ⇒ 2 partitions) makes DOMAIN isolation fit. | |
| - MPU tier's realistic pattern: fork-then-immediately-exec / future posix_spawn (phase 2). | |
| ### Common-libc patches (precedents: libc-common-tmpfile, libc-isoc-signal-handling) | |
| - `libc-common-under-exit.patch`: common `_exit()` (+ `FUNC_ALIAS` `_Exit` — exists | |
| nowhere today) → `k_process_exit(status)` under `CONFIG_PROCESS`, today's spin | |
| otherwise; gates out minimal's **non-weak** `_exit` (picolibc/newlib are `__weak` — | |
| override free); `_Exit` decl in minimal stdlib.h. | |
| - `libc-common-exit.patch` — **DONE (pulled forward into M0, commit c105917b2)**: | |
| exit/atexit promoted to common libc (`COMMON_LIBC_EXIT`, default y for minimal + | |
| picolibc; `COMMON_LIBC_ATEXIT_MAX` default 32 per ISO C). Motivated by the M0 | |
| finding that the SDK's prebuilt picolibc has a STUB atexit (registration records | |
| nothing; exit() runs only `__libc_fini_array`). newlib keeps `__call_exitprocs`. | |
| Documented deviations: no stream flush at exit, fini-array functions don't run. | |
| M1's remaining libc work: `libc-common-under-exit.patch` (`_exit`/`_Exit` → | |
| `k_exit`) and clock(). | |
| - `libc-common-clock.patch`: promote minimal clock.c to common; per-process CPU time | |
| under `CONFIG_PROCESS`, uptime-based otherwise (documented deviation); newlib `_times` | |
| hook in libc-hooks.c (existing `#ifdef CONFIG_POSIX_MULTI_PROCESS` stub pattern). | |
| - `assert()`: nothing needed (→ abort → COMMON_LIBC_ABORT/SIGABRT). Requirement on | |
| kernel design: fatal-signal default action terminates the **process** with | |
| WIFSIGNALED-compatible status. | |
| - Boot: the static init process (pid 1) starts from a `SYS_INIT`, independent of the | |
| image `main()`; no change to `bg_thread_main`'s main()-return behaviour is required. | |
| (Process `exit()` belongs to process mains via crt0; coverage-dump ordering is | |
| handled where the dump already lives.) | |
| ### Headers | |
| - New module `include/zephyr/posix/sys/wait.h`: wait/waitpid/waitid, Linux-compatible W* | |
| encoding, `WNOHANG/WUNTRACED/WCONTINUED/WEXITED/WNOWAIT/WSTOPPED`, `idtype_t` | |
| (P_ALL/P_PID/P_PGID); `id_t` added to module sys/types.h. W* live in sys/wait.h only. | |
| - `libc-sys-wait-h.patch`: **full-replacement** forwarder shims for picolibc/newlib | |
| (toolchain sys/wait.h would collide on W* macros; matches libc-statvfs-utime pattern). | |
| Minimal libc resolves the module header directly. | |
| - `_POSIX_JOB_CONTROL`/`_POSIX_SPAWN` stay commented out (phase 2). | |
| ### Kconfig wiring (module) | |
| `POSIX_MULTI_PROCESS` umbrella unchanged (getpid/times/sleep keep working with today's | |
| deps). New sub-symbols: `POSIX_FORK` (depends PROCESS), `POSIX_WAIT` (depends PROCESS), | |
| `POSIX_EXEC` (depends PROCESS && LLEXT && FILE_SYSTEM && LLEXT_TYPE_ELF_RELOCATABLE; | |
| selects the two export-symbol groups), `choice POSIX_EXEC_ISOLATION` (DOMAIN|NONE). | |
| Unconfigured functions exist and fail `ENOSYS` (documented subprofiling posture). | |
| ### Test topology | |
| - **M1–M2 (pre-exec): statically defined processes are the coverage vehicle** — test | |
| apps `K_PROCESS_DEFINE` one or more child processes whose entries exercise | |
| exit-status, wait/waitpid/waitid, SIGCHLD, kill-by-pid, getpgid/getsid across real | |
| process boundaries with zero loader machinery. | |
| - **Prelinked-blob payloads (no LLEXT, no filesystem)**: the Zephyr-side executable | |
| build helper prelinks tiny executables (crt0 + static libc) at the per-arch entry | |
| convention, objcopys them to binary blobs linked into the test firmware, and | |
| `K_PROCESS_DEFINE()` points at them — code-in-own-partition processes on all 4 | |
| targets with zero loader dependencies. (hello: checks argv/envp, exits with known | |
| status; spinner: for kill/wait tests.) | |
| - Optional LLEXT-from-filesystem scenarios (exec phase only): same payloads embedded | |
| via `generate_inc_file_for_target()`, written to RAM littlefs `/bin/` by test setup. | |
| - Scenarios: base (no FS/LLEXT, proves ENOSYS config), `.exec` (all 4 platforms; an385 | |
| with ISOLATION_NONE), `.exec_userspace` (MMU platforms + riscv64 best-effort), | |
| `.fork` (COW on MMU only). | |
| --- | |
| ## Design: POSIX module layer | |
| ### Headers first (user requirement: declarations ironed out before linkage) | |
| Header inventory for POSIX_MULTI_PROCESS (audited against POSIX.1-2017): | |
| | Header | State | Action | | |
| |---|---|---| | |
| | `sys/types.h` | Present, near-complete | **Add guarded `id_t`** (missing entirely; required by spec and by waitid). **Add guarded `timer_t`** (spec requires it in sys/types.h; today only in posix_time.h:47, unguarded — add the standard `_TIMER_T_DECLARED`/`__timer_t_defined` guards in BOTH per the shared-symbol rule; sys/types.h is not the owner, posix_time.h keeps Doxygen). `trace_*` TODO stays (Trace group, not MULTI_PROCESS). | | |
| | `sys/wait.h` | **Missing everywhere** | Create per posix-header-layout skill: IEEE declaration order, `idtype_t {P_ALL,P_PID,P_PGID}`, W* constants/macros, wait/waitid/waitpid declarations with Doxygen `@ingroup posix_option_group_multi_process` (owning header). **Every W* constant/macro derives from a Zephyr-native symbol** — `WNOHANG`/`WUNTRACED`/`WCONTINUED`/`WEXITED`/`WNOWAIT`/`WSTOPPED` map to `K_PROCESS_W*` flags and the status macros wrap the kernel `K_WSTATUS_*` bit layout (Zephyr-native encoding is authoritative; Linux-shaped layout is a convenience, not a contract — toolchain headers are replaced by shims, not accommodated). Linear includes, basic types only. Plus the picolibc/newlib full-replacement shim patch (`libc-sys-wait-h.patch`). | | |
| | `unistd.h` | Complete | All 17 declarations already present. | | |
| | `sys/times.h` | Complete | No change. | | |
| | `stdlib.h`/`assert.h`/`time.h` | libc-owned | `_Exit` declaration added to minimal stdlib.h via the common-libc patch; nothing else missing. | | |
| M0 lands the header work **first**, before any implementation linkage — declarations | |
| compile-tested across all six libc variants before functions exist behind them. | |
| ### POSIX / kernel separation (user requirement) | |
| POSIX functions are **thin wrappers** over kernel `k_*()` syscalls or lib-layer | |
| `sys_*()` APIs — never bearers of kernel logic themselves (precedent: the series' | |
| `sys_thread` layer in lib/os, which pthreads and C11 threads both wrap): | |
| - `getpid/getppid/getpgid/getsid/setsid/_exit/fork/waitpid/waitid` → direct wrappers | |
| over `k_process_*()` syscalls (+ W*-encoding / signo mapping, which is POSIX-ism and | |
| stays module-side). | |
| - **The exec core moves Zephyr-side as `sys_process_exec()` / `sys_process_spawn()` in | |
| lib/os** (new patch, following the sys_thread precedent): composes LLEXT load, | |
| domain build, argv/envp staging + crt0 ABI, `z_process_exec_begin/commit`, deferred | |
| image teardown. Usable by non-POSIX apps; upstreamable on its own. The module's | |
| `execve()` et al. keep only the POSIX-isms: path/PATH resolution (environ, | |
| `confstr(_CS_PATH)`), varargs marshalling, errno mapping. (Supersedes the earlier | |
| placement of the full exec flow in module `exec.c`; the fexecve zvfs-fd loader | |
| adapter also moves into the sys layer.) | |
| - `pthread_atfork` registry stays module-side (pure POSIX bookkeeping); its hooks run | |
| around the `k_process_fork()` call in the module wrappers. | |
| ### Layout (one file per function, strict) | |
| `multi_process.c` dissolves into `getpid.c` / `times.c` (preserving existing notices per | |
| preserve-copyright). New files: `_exit.c`, `fork.c`, `execv.c execve.c execvp.c`, | |
| `execl.c execle.c execlp.c`, `exec_common.c` (va_list→vector marshalling), | |
| `getppid.c getpgrp.c getpgid.c getsid.c setsid.c`, `wait.c waitpid.c waitid.c`, | |
| `multi_process_internal.h` (W* encode helpers). Shared: | |
| `options/shared/atfork_common.c` (registry + `z_posix_atfork_run()`; shared/ compiles | |
| unconditionally so fork.c links without POSIX_THREADS), `env_common.c` gains | |
| `z_environ_init_from_vector()`, `posix_internal.h` gains `posix_this_pid()`. | |
| New module `.c` = FPES notice; headers/tests/Kconfig = ZPC. | |
| ### Key decisions | |
| - **W* encoding: the native Zephyr encoding (`K_WSTATUS_*`) is authoritative** — the | |
| module's W* macros must agree with it by derivation. Where a libc's own sys/wait.h | |
| encodings differ from Zephyr's for the same symbols, a **runtime mapping** is done — | |
| with **compile-time equality checks creating a zero-cost fast path** when the | |
| encodings coincide (the established pattern: `net/conversion.h`-style converters and | |
| the module's `z_sig_set_equal()` fast path). | |
| - **pid migration**: `posix_this_pid()` inline — `k_process_pid(k_process_current())` | |
| under `CONFIG_PROCESS`, else the existing 42. Three users switch (getpid, signals | |
| kill(), rtsig). Root process pid 1, ppid 0. `kill()` pid>0 branch becomes | |
| `k_process_kill()` under CONFIG_PROCESS; legacy thread-pointer overload preserved | |
| bit-exact when off. | |
| - **pthread_atfork becomes real**: prepare (reverse order) → clone/fork syscall → | |
| parent/child (registration order). Handlers become live with M4 COW fork. | |
| **Registry is elastipool-backed, no malloc** (user decision — malloc in the atfork | |
| path risks fork-time deadlocks): statically sized via | |
| `CONFIG_POSIX_THREAD_ATFORK_MAX` (default 0) raised to the largest distributed | |
| `POSIX_THREAD_ATFORK_MIN_*` contribution (SIGNAL_SET_SIZE_MIN aggregation pattern); | |
| empty table → ENOMEM. No `MIN_POSIX` floor — POSIX specifies no minimum. | |
| - **Docs posture (user decision)**: keep the `†` undefined-behaviour markers in the | |
| option-group table until the group is actually conformant — no interim | |
| "defined-fallback" table updates. | |
| - **Coexistence with libc infrastructure (user requirement)**: registration arrays are | |
| libc-owned and must be respected by all three layers. The contract: the **libc's | |
| atexit registry is the single source of truth** for exit handlers — libc `exit()` | |
| runs it (plus ISO C stream flush/tmpfile cleanup), then `_exit()` → `k_exit()`; | |
| the kernel never runs handlers (`k_exit` is below libc); POSIX `_exit()`/`_Exit()` | |
| bypass them by definition and fork/exec never duplicate or re-run them. The same | |
| discipline applies to atfork: the registry lives in one place (module shared code), | |
| and libc-internal atfork stubs (newlib/picolibc) must be checked for collision and | |
| routed to it. | |
| - **environ**: no fork hook needed — COW duplicates it for free. Exec side uses | |
| `z_environ_init_from_vector` in the new image. | |
| - **Kconfig (user requirement)**: the kernel exposes granular Zephyr-native symbols | |
| (`PROCESS`, `PROCESS_STATIC`, `PROCESS_FORK_COW`, | |
| `PROCESS_EXEC`, `ZVFS_PER_PROCESS_FDS`, …) and **`POSIX_MULTI_PROCESS` selects the | |
| kernel symbols it requires wherever the platform supports them** (`select PROCESS`, | |
| `select PROCESS_FORK if ARCH_HAS_THREAD_FORK && USERSPACE`, etc.). The POSIX layer | |
| stays a **very thin wrapper** around the native syscalls. ENOSYS fallbacks remain | |
| only where the substrate genuinely cannot exist on a platform (e.g. COW fork | |
| without MMU); wait* → ECHILD when no children; identity fns fall back to | |
| single-process identity only on `!PROCESS` builds of other option groups. | |
| ### Test design | |
| - **Native suites first (user requirement)**: every native Zephyr component gets its | |
| own dedicated testsuite in the Zephyr tree — `tests/kernel/process/static` | |
| (K_PROCESS_DEFINE, reserved pids, membership), `tests/kernel/process/spawn`, | |
| `tests/kernel/process/clone`, `tests/kernel/process/wait_exit`, signal-CHLD additions | |
| to `tests/kernel/signal` — landing in the same patch as their implementation | |
| (k-signal.patch precedent). **The native components must be working and tested | |
| satisfactorily before the corresponding POSIX components are wired**; POSIX suites | |
| are conformance re-checks over already-proven kernel behavior. | |
| - POSIX side: one ZTEST per function for all 23 units; fallback sections assert the | |
| defined ENOSYS/ECHILD behavior so the six existing variants stay green. | |
| - **Exit-status observation**: pre-fork, a separate non-ztest console-harness app | |
| (`harness: console`, ordered regex — exit() in a ztest binary kills the suite); | |
| post-fork, in-suite via child exit status (`test_exit`: atexit ordering folded into | |
| `_Exit(counter)`; `test__Exit`/`test__exit`: handlers skipped, WEXITSTATUS asserted). | |
| - `test_fork` sections: ENOSYS assertion wherever COW is off (all non-MMU | |
| targets); COW sections on MMU (memory isolation via exit status, fd offset | |
| sharing, atfork child ran). posix_spawn coverage lives in the POSIX_SPAWN | |
| group's own suite. | |
| - `test_execve`: build child LLEXT at test build time, embed, before-hook writes to | |
| ramfs; child checksums argv/envp and exits with it. ENOENT/ENOEXEC aspects. | |
| - Scenarios: existing six unchanged; `.process` (M1+, all four bring-up platforms); | |
| `.process.exec` (M3+). Fix the `muti_process` typo (grep CI yamls for stale | |
| references). | |
| ### ISO C consistency (user requirement) | |
| Process support is designed against **both** specifications, interoperating at the | |
| level POSIX and ISO C jointly specify — not POSIX alone: | |
| - ISO C exit sequence honored exactly: returning from `main()` ≡ `exit(main's | |
| value)`; `exit()` runs atexit handlers LIFO (≥32 registrations), flushes/closes | |
| open streams, removes `tmpfile()` files, then terminates; `_Exit()`/`abort()` skip | |
| handlers; `EXIT_SUCCESS`/`EXIT_FAILURE` round-trip through wait status. | |
| - POSIX↔ISO C interop points made explicit: `_exit()` ≡ `_Exit()`; `abort()` ⇒ | |
| SIGABRT ⇒ `WIFSIGNALED` in the parent; `CLOCKS_PER_SEC`/`clock()` (ISO C) vs | |
| `times()`/`_SC_CLK_TCK` (POSIX) unit consistency; `assert()` ⇒ abort path. | |
| - The ISO C pieces live in the **libc layer** (common libc patches), never in the | |
| POSIX module — the module only adds POSIX-only semantics on top. | |
| ### Conformance/docs | |
| - doc table cells → `yes` + a **Tiers and deviations** section (COW vs spawn per | |
| platform class; ENOSYS/ECHILD without CONFIG_PROCESS; | |
| WUNTRACED/WCONTINUED accepted-never-reported; no job control). | |
| - `_SC_CHILD_MAX` from new `CONFIG_POSIX_CHILD_MAX` (floor 25); `_SC_JOB_CONTROL` | |
| stays -1; `_POSIX_SPAWN`/`_POSIX_JOB_CONTROL` stay out through M4. | |
| - Audit `times()` µs clock_t vs `_SC_CLK_TCK` in M1. | |
| ## Phased roadmap | |
| **Commit-order rule (user requirement)**: the Zephyr-kernel-native implementations | |
| come **first** in the patch series, each landing together with **near-100% kernel test | |
| coverage** (`tests/kernel/process/...`, following the k-signal.patch | |
| tests-in-same-patch precedent) — the thin POSIX wrappers and their suites follow. | |
| Zephyr needs native syscalls for essentially all of the multi-process functionality, | |
| including waiting on other processes, and those syscalls + tests are the upstream | |
| deliverable in their own right. | |
| **Toolchain track (bottom-up, in parallel — user requirement)**: alongside the | |
| kernel-up work, explore the bridge from the compiler/toolchain end: what CRT pieces | |
| (crt0/crti/crtn, specs/linker integration, arch entry conventions) would live in | |
| gcc/llvm and in each libc (minimal, picolibc, newlib, common) to reach | |
| kernel/toolchain independence — how picolibc's existing crt0 machinery and newlib's | |
| crt0.o conventions map onto the Zephyr process ABI, what `-nostartfiles`/specs | |
| adjustments the executable build helper needs, and where the ABI contract must bend | |
| to meet stock toolchain behavior. Deliverable: a process-ABI document + a prototype | |
| crt0 in at least one libc, converging with the kernel ABI "in the middle" before the | |
| exec phase freezes the contract. | |
| - **M0 — Headers + API design at all three layers** (declarations before any | |
| implementation; user requirement): **headers first across kernel, ISO C, and | |
| POSIX** — the full API-interoperability design is done up front, revisable later by | |
| rewriting history: | |
| - *Kernel*: draft `include/zephyr/kernel/process.h` complete — `struct k_process`, | |
| `K_PROCESS_DEFINE()` (both entry forms), pointer `k_pid_t`, `k_spawn`/`k_clone` | |
| arg structs and flags, `k_waitpid`/`k_waitid`/`k_exit`/`k_getpid`/…, `K_WSTATUS_*` | |
| and `K_PROCESS_W*` symbols, hook section — compile-clean with stub/ENOSYS bodies. | |
| - *libc*: `_Exit` declaration, atexit/exit contract notes, crt0 entry contract | |
| sketch in the process-ABI doc. | |
| - *POSIX module*: sys/types.h completion (id_t, guarded timer_t), new sys/wait.h | |
| (deriving from the kernel symbols) + libc shims, unistd.h check — | |
| compile-tested across all six libc variants. | |
| Then: file split, all 13 new function files with fallback semantics, real | |
| pthread_atfork registry, all 23 ZTESTs (fallback sections), exit console-harness | |
| app, typo fix, docs UB→defined. Proves the whole declared surface on every | |
| platform/libc before implementation starts. | |
| - **M1 STATUS (in progress)**: M1a (membership, static start, exit/wait) + M1b | |
| (k_spawn, positive-reap tests) DONE, folded into commit `6739f0706`; kernel patch | |
| `k-process-exit-wait.patch`, `tests/kernel/process` 10/10 on qemu_cortex_m3 | |
| (kernel+userspace), qemu_x86, qemu_riscv64. Two kernel bugs found+fixed: | |
| halt-path wake used sched-lock-unsafe `z_unpend_all` then `z_unpend_first_thread` | |
| (recursive sched lock) → `z_sched_wake_thread_locked` + explicit 0 return value | |
| (a bare unpend leaves pended -EAGAIN, read as timeout). Also a CI overflow in the | |
| M0b waitpid test (`~(WNOHANG|WUNTRACED)` narrows to int on riscv64) — folded into | |
| M0b. **M1c DONE (4ffcc97a2)**: POSIX_MULTI_PROCESS selects PROCESS; getpid/ppid/ | |
| pgid/sid/setsid/wait/waitpid/waitid wrap k_process; kill/sigqueue process-aware; | |
| process syscalls gated CONFIG_PROCESS||__DOXYGEN__ moved to process.h (syscall header | |
| ifdef-registered); CLD_* added to picolibc/newlib signal.h shims. **M1d DONE | |
| (7413f7aef)**: _exit→k_exit via COMMON_LIBC_UNDERSCORE_EXIT + spawn-child-libc-exit | |
| test (kernel/process 11/11). M1c/M1d bundle their patch regens (fold at pre-upstream | |
| cleanup). **Remaining M1**: M1e SIGCHLD emission on zombify — needs k_sig_info | |
| si_pid/si_status extension (new k-process-signal patch) + halt-path-safe deferral | |
| (set per-process sigchld_pending under sched lock, submit a shared scan-work whose | |
| handler re-validates ZOMBIE state under process_lock to avoid slot-reuse races); | |
| M1f init-process start (SYS_INIT static pid 1). | |
| - **M1 — Process identity + static processes + exit/wait** (kernel core; **user | |
| hand-implements**): `k_process`, **`K_PROCESS_DEFINE()` iterable-section statics** | |
| (the reserved swapper/init/kthreadd trio + app-defined processes), pointer | |
| `k_pid_t` + slot-index numeric pids, k_exit/k_waitpid, k_sig_info si_pid/si_status, | |
| init process start (SYS_INIT static pid 1). Module: posix_this_pid switchover, | |
| _exit → k_exit, strict kill(), siginfo fill. Common-libc exit/atexit/_Exit/clock | |
| patches land here. **Statically defined processes are the API-coverage vehicle**: | |
| test apps define child processes at build time and exercise wait/waitpid/waitid, | |
| SIGCHLD, kill, getpgid/getsid, exit-status across real process boundaries — no | |
| loader, no fork. **M1 must verify all three protection configurations**: MMU | |
| (qemu_x86, qemu_cortex_a53), MPU (mps2/an385), and **no-MMU-no-MPU** (a | |
| `CONFIG_USERSPACE=n` build — kernel-mode processes; e.g. qemu_riscv64 or | |
| qemu_cortex_m3 without userspace). `.process` scenario green on all of them. | |
| - **M2 — `k_spawn()` + posix_spawn** (no arch context-copy work needed — spawn creates | |
| a fresh process from a spec): dynamic process creation on **all** targets including | |
| no-protection ones; posix_spawn() thin wrapper (POSIX_SPAWN group + suite, | |
| `_POSIX_SPAWN` enabled); wait/exit round-trips against spawned children; W* encoding | |
| finalized. All four platforms. | |
| - **M2.5 — `sys_spawn()` + POSIX_SPAWN option group**: the pool-allocating spawn | |
| tier and the full `<spawn.h>` surface, on all four platforms, no MMU required, | |
| no arch asm. | |
| - *sys layer (`lib/os/process.c`)*: `sys_spawn()` draws the child's thread + | |
| stack from the system thread pools (`SYS_THREAD`), builds `k_spawn_args`, | |
| calls `k_spawn()`; resources return to the pools at reap via a leader-release | |
| hook (pool-allocated leader threads/stacks must not be recycled before | |
| `k_waitpid()` reaps the process, or a reused thread object would collide with | |
| the unreaped zombie's leader identity; shape: a `__weak | |
| z_process_leader_release()` called at the reap point outside `process_lock`, | |
| overridden in lib/os, gated by a pool-owned-leader process flag). Includes | |
| fixing a pre-existing `sys_thread_create_common()` bug its ordering depends | |
| on: the `k_is_in_user_syscall()` branch passes `K_NO_WAIT` instead of the | |
| caller's `delay`, so paused creation silently starts the thread immediately. | |
| `SYS_SPAWN_*` constants as needed for attribute support. | |
| - *POSIX layer (`POSIX_SPAWN` group, own suite per suite-per-group)*: | |
| `posix_spawn()`, `posix_spawnp()`, `posix_spawnattr_t` init/destroy + | |
| flags/pgroup/sigmask/sigdefault/schedparam/schedpolicy get/set, | |
| `posix_spawn_file_actions_t` init/destroy + addopen/addclose/adddup2, | |
| `POSIX_SPAWN_{RESETIDS,SETPGROUP,SETSCHEDPARAM,SETSCHEDULER,SETSIGDEF, | |
| SETSIGMASK}` derived from `K_SPAWN_*`/`SYS_SPAWN_*` counterparts; | |
| `_POSIX_SPAWN` enabled. pgroup → `k_setpgid()` at spawn; sigmask/sigdefault → | |
| child signal state before start; file_actions honored once per-process fds | |
| exist (documented deviation before then). | |
| - *Open question (Chris)*: what an executable **path** names before M3 exec. | |
| Proposal: a build-time registry of prelinked images (iterable section, | |
| name → entry point) that `sys_spawn()` consults — the same prelinked-blob | |
| story M3 already prioritizes for the MPU tier; unknown paths → `ENOENT`. | |
| - `k_clone()` stays a declared `-ENOSYS` surface reserved for M4 COW fork. | |
| - **M3 — exec via LLEXT** (later phase, per user): kernel process-ABI framework | |
| (args/env block layout), libc crt0, Zephyr-side executable build helper, | |
| `sys_process_exec/spawn` in lib/os, module exec* wrappers + PATH resolution, | |
| FD_CLOEXEC sweep, syscalls-only export surface. an385 with ISOLATION_NONE; MMU | |
| qemus with domain isolation. Freeze the args/env ABI here. **On the MPU tier, | |
| prelinked-blob processes remain the prioritized/supported path; dynamic LLEXT exec | |
| on constrained MPU targets is documented best-effort** (RAM fragmentation from | |
| relocatable loading). From M3 onward, add at least one **physical board per tier** | |
| alongside qemu (board choice: user's call) — qemu masks timing/cache/TLB costs of | |
| ASID swapping and MPU reprogramming that silicon will expose. | |
| - **M4 — COW fork on MMU** (kernel; **user hand-implements**): module delta ≈ zero (one | |
| IS_ENABLED gate flips — atfork child handlers now run). COW sections activate on | |
| qemu_x86/a53. | |
| - **M5 — Groups/sessions + SIGCHLD completeness**: setsid/getpgid multi-process | |
| semantics, kill(0/-pgid), waitpid(0/-pgid), waitid(P_PGID), WNOWAIT corners, times() | |
| cutime/cstime, deviations doc final. (M6 future: job control.) | |
| Gates per milestone: suite-scoped `runci.sh -T tests/posix/multi_process` (+signals, | |
| threads_base for M0/M1) then full `runci.sh`; doc build at M4. | |
| ### Module-layer risks | |
| 1. execvp PATH policy (env PATH else Kconfig default) — needs a call. | |
| 2. Header-conflict posture (user direction): what matters is **posix-next ↔ | |
| Zephyr-native definition compatibility**, not agreement with toolchain headers. | |
| **Independent C library headers must be independent** — where a toolchain header | |
| (newlib/picolibc sys/wait.h etc.) would conflict, inject the correct redefinitions | |
| at the `zephyr/lib/libc/<libc>/include/` shim layer, as done previously | |
| (libc-sys-select-h, libc-statvfs-utime patterns). posix-next then defines its | |
| surface without worrying about excessive conflicts. | |
| 3. `.userspace` × `.process` matrix is the hardest kernel corner — claimed only from | |
| M4 (COW) onward. | |
| 4. linux_compat variant: process tests filtered `not CONFIG_NATIVE_LIBC` (host fork out | |
| of scope). | |
| --- | |
| ## Design: kernel `k_process` + fork tiers | |
| **Governing frame (user emphasis): a thread group is mostly synonymous with a process | |
| from the kernel perspective.** `struct k_process` IS the thread-group object — the | |
| Linux model, where the pid is the thread-group id. Everything follows from this: | |
| `k_getpid()` returns the group's id from any member thread; process-directed signals | |
| (`k_sig_queue`) target the group (any eligible member delivers) while | |
| `k_sig_queue_tid` targets one thread (the tgkill/kill distinction); `k_exit()` | |
| terminates the whole group; `k_waitpid()` observes group termination; a thread created | |
| by a member joins the group; per-process state (dispositions, fd table, resources) is | |
| per-group state. No separate "process" concept exists apart from the group. | |
| **Every thread group has a thread group leader** — the primary thread of the process | |
| (`proc->group_leader`, a `struct k_thread *`): the initial thread at creation | |
| (K_PROCESS_DEFINE initial thread, spawn's first thread, the clone child's thread), | |
| re-anchored to the surviving thread on exec. Adopted leader-exit semantics (Linux | |
| model, settled): | |
| - **Leader exits early (pthread_exit-style)**: siblings are NOT killed and **no | |
| promotion occurs** — the group persists, `getpid()` in siblings keeps returning the | |
| group's pid, and group termination completes only when the last member exits (then | |
| SIGCHLD → parent, reapable). Where Linux must keep a zombie leader `task_struct` | |
| alive to anchor the TGID, Zephyr's design is structurally simpler: **the pid lives | |
| in `struct k_process`, so the k_process object itself is the anchor** and the | |
| leader's `k_thread` can be fully recycled; `group_leader` becomes NULL/stale-marked | |
| until group exit. | |
| - **Unhandled fatal fault in ANY member (leader or sibling)**: the entire group dies | |
| immediately — the Zephyr analog of `do_group_exit()`: the fatal-error path | |
| (`z_fatal_error`/`k_sys_fatal_error_handler`) invokes group exit with | |
| `K_WSTATUS_SIGNALED(signo)` status (SEGV/BUS/ILL-class), aborting every member. | |
| This is also the fault story for `test_*_fault` ZTESTs against processes. | |
| ### pid model (revised per user question: pointer handle is feasible and preferred) | |
| - **`k_pid_t` = `struct k_process *`** — the idiomatic Zephyr handle, exactly as | |
| `k_tid_t` is `struct k_thread *`. Kernel-native APIs (`k_waitpid`, `k_sig_queue`, | |
| `k_getpgid`, …) take/return the pointer handle. | |
| - **The numeric pid is derived from the pointer, not allocated** — a pointer is | |
| itself numeric, and POSIX only requires `pid_t` to be a *signed integer type*. The | |
| raw pointer value can't be the user-visible pid directly (kernel addresses with the | |
| high bit set collide with the `waitpid`/`kill` sign encodings `0`/`-1`/`-pgid`, and | |
| it would leak kernel addresses to user mode), so the numeric pid is the **slot | |
| index**: `pid = (proc - base)` across the reserved trio + statics section + pool — | |
| a small positive int, pure arithmetic in both directions | |
| (`k_pid_of(proc)`/`k_process_by_pid(int)`), **no pid allocator or table at all**. | |
| Slot reuse = ordinary pid reuse (an optional generation counter can detect staleness | |
| if wanted). `K_PID_SWAPPER`=0, `K_PID_INIT`=1, `K_PID_KTHREADD`=2 name the reserved | |
| numeric ids; the reserved handles are `&k_process_swapper` etc. The thin POSIX | |
| wrappers translate handle↔index at the boundary. | |
| - **Lifetime**: the zombie IS the `k_process`, so a parent's handle stays valid until | |
| reap (`k_waitpid` returns); handles held past reap dangle when the slot recycles. | |
| Mitigation: `K_OBJ_PROCESS` becomes a tracked kernel-object type — statics from | |
| `K_PROCESS_DEFINE` are discovered by `gen_kobject_list` like every other static | |
| kernel object, pool slots use `k_object_recycle`, and syscall-boundary validation | |
| (`K_SYSCALL_OBJ`) rejects stale/foreign pointers. (Cost: perms-bitmap bytes per | |
| process object — acceptable at CONFIG_PROCESS_MAX scale.) | |
| - All existing `k_pid_t` users are in the local series/module — a coordinated flag-day | |
| inside `k-process-core.patch` is safe and **compiler-enforced** (pointer→int flips the | |
| type). `k_sig_queue()` becomes process-directed; new `k_sig_queue_tid()` carries | |
| today's thread-directed semantics (pthread_kill); in-series callers | |
| (alarm/msgq/aio/timer/cancel) are switched per the existing `TODO(k_process)` markers. | |
| ### struct k_process (kernel/process.c, include/zephyr/kernel/process.h — new) | |
| pid/pgid/sid, parent + children dlist, thread-group dlist (`k_thread.process` + | |
| `process_node` beside `mem_domain_info`), state (`K_PROCESS_ACTIVE` / | |
| `K_PROCESS_EXITING` / `K_PROCESS_ZOMBIE` — all public symbols namespaced | |
| `k_`/`K_`/`sys_`/`SYS_` per Zephyr convention), latched | |
| `wstatus`, wait queue, `k_mem_domain *domain` (owned/borrowed/default), per-process | |
| sparse sigaction table, utime/stime + cutime/cstime cycle accounting, `res[]` opaque | |
| slots for upper layers (zvfs fd map/cwd). | |
| **Static definition via macro + iterable section (the Zephyr idiom — user requirement):** | |
| ```c | |
| /* Defines: struct k_process in STRUCT_SECTION_ITERABLE(k_process, name), | |
| * K_THREAD_STACK_DEFINE(name##_stack, stack_size), the initial k_thread, | |
| * and a partition-pointer array for the process's domain. */ | |
| K_PROCESS_DEFINE(name, stack_size, entry, p1, p2, p3, prio, options, delay, | |
| ...partitions); | |
| /* Companions (reuse existing machinery): */ | |
| K_APPMEM_PARTITION_DEFINE(name_part); /* existing — process-private data */ | |
| K_APP_DMEM(name_part) int my_global; /* existing — tag globals into it */ | |
| K_PROCESS_ACCESS_GRANT(name, ...); /* forwards to K_THREAD_ACCESS_GRANT | |
| on the initial thread */ | |
| ``` | |
| - Parameter set mirrors `K_THREAD_DEFINE` (entry/args/stack/prio/options/delay — the | |
| initial-thread spec; `K_USER` in `options` selects unprivileged execution) plus a | |
| trailing partition list for the process's `k_mem_domain`. **`K_PROCESS_DEFINE()` | |
| composes the existing macros internally** (`K_THREAD_STACK_DEFINE`, the static | |
| thread-definition machinery, partition array construction) rather than reinventing | |
| them — one user-facing macro, existing building blocks underneath. | |
| - **Entry forms (user requirement — no LLEXT dependency)**: the entry may be | |
| (a) a **resolved/named symbol** in a specific memory domain of the process (code in | |
| the shared kernel image or in a process partition), or (b) a **bare numeric | |
| address** inside the process's domain, where no symbol name is known. Form (b) | |
| enables the key pattern: **prelink an ELF, convert it to a byte array via Zephyr's | |
| cmake utilities** (`generate_inc_file_for_target()`-style include-file generation, | |
| not bare objcopy artifacts) **included into the firmware image** (placed in a | |
| process partition), and Zephyr loads it as a process using the `K_PROCESS_DEFINE()` | |
| parameters — full process support with no LLEXT, no filesystem, no runtime loader. | |
| - **Same look and feel on MMU and non-MMU**, with under-the-hood differences: on MMU | |
| platforms the common pattern is a **fixed per-arch virtual entry address** — adopt | |
| the entry-point/text-base conventions Linux uses per architecture (e.g. the | |
| arch-default ET_EXEC link addresses) so prelinked blobs are directly buildable with | |
| stock toolchain defaults. **MMU-less systems do NOT use a common virtual entry**: | |
| each process gets a **different physically addressed region** for its initial | |
| address space + protection area (per-process prelink addresses; the build helper | |
| assigns non-overlapping regions). | |
| - **MMU-less memory-safety realities (user notes)**: `malloc` is optional, but | |
| processes on MMU-less systems still share the global `k_malloc()` pool, so | |
| cross-process corruption is possible — mitigate with per-process heaps (the | |
| resource-ledger `k_heap`) where configured, and with **hardware MPU protection of | |
| the running process** where slot budget allows. Where MPU regions are scarce, | |
| explore **sharing and dynamically updating MPU regions based on the currently | |
| running process** (Zephyr already reprograms the MPU from the thread's domain at | |
| context switch — this extends that to a per-process app region). Swapping a process | |
| out is a separate topic — explicitly out of scope for this plan. | |
| - Private data comes from `K_APPMEM_PARTITION_DEFINE` + `K_APP_DMEM`/`K_APP_BMEM` | |
| tagging (existing linker-section machinery); MMIO/peripheral access is just another | |
| partition; kernel object rights use the existing grant/inherit model. | |
| - **Image permission machinery is process-layer, not LLEXT-layer**: mapping a process | |
| image's partitions with correct W^X (RX text / RW data) uses | |
| `k_mem_update_flags()`/domain partitions directly and must work with LLEXT disabled; | |
| LLEXT's `llext_adjust_mmu_permissions()` becomes just one client of it. | |
| - Boot `SYS_INIT` walks `STRUCT_SECTION_FOREACH(k_process, ...)`: numeric pids follow | |
| section order after the reserved trio, `k_mem_domain_init` with the partition list, | |
| create the initial thread paused (`sys_thread_create_paused` precedent), add it to | |
| domain + thread group, start after `delay` (K_THREAD_DEFINE autostart semantics). | |
| - **Dynamic processes are managed by an elastipool (user decision)**: spawn/clone | |
| children come from a `sys_elastipool` (the series' elastipool.patch API, already | |
| used for the module's pooled objects) rather than a bare static array — | |
| `CONFIG_PROCESS_MAX`-sized static region with the elastipool's growth semantics. | |
| The elastipool is also how the number of active processes is managed, and process | |
| allocation/destruction will need **fairly elaborate custom handlers** (domain | |
| setup/teardown, kobject recycle, resource-ledger reclaim, zombie retention until | |
| reap — the elastipool's alloc/free hooks are the seam for them). | |
| Note for patch 1: numeric-pid-as-slot-index derivation must use the elastipool's | |
| static region indexing; entries from any dynamic-growth region need an index from | |
| the elastipool API (or a small spillover map) — verify what sys_elastipool exposes | |
| and constrain accordingly. | |
| - **`K_OBJ_PROCESS` is a new kernel object type** with the 4-character code **"PROC"** | |
| (matching the 4-char code convention of other Zephyr kernel object types), wired | |
| through `gen_kobject_list.py` exactly as the series' K_OBJ_FILE work did — statics | |
| discovered at build time, pool entries via `k_object_recycle`, `K_SYSCALL_OBJ` | |
| validation at the syscall boundary. | |
| - **Three kernel-reserved processes defined by the kernel itself** (via an internal | |
| no-initial-thread variant of the macro — their threads already exist and are adopted | |
| at boot): | |
| - `k_process_swapper` — **pid 0**, the swapper/idle process: exactly the per-CPU | |
| idle threads (Zephyr already has one idle thread per CPU in SMP — the idle process | |
| is the logical extension of that). Does nothing but sleep when no other task is | |
| runnable. Dummy/boot threads also start here. (No POSIX conflict: | |
| `kill(0)`/`waitpid(0)` mean "caller's pgrp", so pid 0 is never a valid explicit | |
| target.) | |
| - `k_process_init` — **pid 1**, the init process: parent of **all userspace threads | |
| and processes**; its **own** initial thread and stack (it does NOT adopt | |
| `z_main_thread` — see boot flow below); orphans reparent to it; it auto-reaps. | |
| **Existence is NOT gated on USERSPACE**: pid 1 exists whenever `CONFIG_PROCESS` | |
| is enabled, because it also parents "root"-owned processes running with | |
| supervisor/superuser privileges (on `!USERSPACE` builds, all processes are | |
| effectively root); USERSPACE affects isolation, never existence. | |
| - `k_process_kthreadd` — **pid 2**, kthreadd: parent process of **all kernel | |
| threads** (sysworkq, logging, driver/net threads, …). | |
| - No pid allocator: numeric pids are slot indices derived from the handle (see pid | |
| model) — 0–2 are the reserved trio, statics and pool slots follow in section order. | |
| `K_PID_NONE` (if kept) is `NULL` at the handle level. | |
| - `getppid()` of init returns 0 (swapper) — consistent with the earlier "root ppid 0". | |
| **Boot flow under CONFIG_PROCESS (main() repurposed; resolves the system-thread | |
| ownership question):** | |
| 1. Idle threads (one per CPU) belong to **swapper (pid 0)**. `z_main_thread` starts in | |
| swapper for bring-up; kernel/system threads created during init levels (sysworkq, | |
| logging, driver threads, …) are assigned to **kthreadd (pid 2)** — membership rule: | |
| idle → swapper; kernel-mode threads created during boot or by kthreadd members → | |
| kthreadd; everything else inherits the creator's process (userspace threads and | |
| processes thus descend from init, pid 1). The exact K_USER-vs-inheritance edge for | |
| supervisor-mode app threads in ISOLATION_NONE configs is a patch-1 review detail. | |
| 2. Image `main()` keeps today's semantics; no `bg_thread_main` change is required for | |
| process support. | |
| 3. The statically defined **init process (pid 1)** starts from a `SYS_INIT` at boot, | |
| independent of `main()`. Its initial thread runs the application entry — the | |
| `K_PROCESS_DEFINE`-style entry of a kernel-supplied default init process | |
| (weak/Kconfig-selectable symbol — ztest supplies it for `.process` test scenarios), | |
| or exec of an init executable (`/sbin/init` via LLEXT) in exec-capable configs. | |
| Exact default-entry naming is an implementation decision for patch 1/2 review. | |
| 4. When `!CONFIG_PROCESS`, today's semantics are unchanged (main() is the app). | |
| ### API naming (user decision) | |
| Process syscalls use the **bare `k_` prefix with the typical POSIX names** — the | |
| convention the series already set with `k_sig_queue`/`k_sig_timedwait`: | |
| `k_getpid()`, `k_getppid()`, `k_getpgid()`, `k_getsid()`, `k_setsid()`, | |
| `k_execve()`, `k_waitpid()` / `k_waitid()`, `k_exit()` (no collision with | |
| the existing `k_thread_exit()` — distinct scope), plus the creation primitives | |
| `k_spawn()` and `k_clone()` (Linux-inspired names rather than POSIX ones — see | |
| Process-creation primitives). The **object** keeps its full name: | |
| `struct k_process`, `K_PROCESS_DEFINE()`, `k_process_current()` (object accessor, no | |
| POSIX analogue). Internal helpers stay `z_process_*`. Wherever this plan says | |
| `k_process_<verb>()` for a POSIX-named operation, read the `k_<posix-name>()` form; | |
| final spellings reviewed in patch 1. (`kill()` needs no new syscall — it wraps the | |
| process-directed `k_sig_queue()`.) | |
| ### Lifecycle | |
| - `k_exit(code)`: latch status, abort all other group threads, self-abort. | |
| Per-thread hook `z_process_exit_thread()` in the existing halt epilogue | |
| (kernel/sched.c:1395, beside `z_mem_domain_exit_thread`/perms-clear). Last thread out | |
| zombifies: run `K_PROCESS_HOOK` exit callbacks (iterable section — fd close, cwd, | |
| timer/msgq retarget), tear down owned address space, reparent children to init | |
| (init auto-reaps zombie orphans), emit SIGCHLD (unless parent IGN → auto-reap), wake | |
| waiters. **The `k_process` object IS the zombie** — all k_threads recycle normally; | |
| pid/wstatus/times survive in the pool slot until reaped. No thread_state bit needed. | |
| - `k_waitpid(selector, &reaped, &st, opts, timeout)`: the selector carries the POSIX | |
| waitpid() encodings **natively in the handle domain** (user decision): a valid | |
| handle = that child; `K_PID_ANY` ((k_pid_t)-1) = any child; `K_PID_MY_PGRP` (NULL) = | |
| caller's process group; `K_PID_PGRP(leader)` (negated handle) = that group. No | |
| WPGID flag. Constraint: sign-discrimination of negated handles requires process | |
| objects at non-negative intptr_t addresses; otherwise validate-then-negate. | |
| WNOHANG/WNOWAIT, pend on parent waitq, ECHILD when selection empty. Distinct from | |
| `k_thread_rejoin` (thread results) — module must not conflate. | |
| - `k_clone(args, args_size, &child)` models **clone3** (user decision, superseding the | |
| earlier clone2 framing): extensible size-versioned `struct k_clone_args` (u64 flags | |
| with Linux flag values, exit_signal, stack/stack_size; zero = defaults). | |
| - kernel/process.c uses the shared `os` kernel log module (LOG_MODULE_DECLARE), not a | |
| private one (user decision). | |
| - Subsystem hook section `K_PROCESS_HOOK_DEFINE(fork, exec, exit)` keeps zvfs/LLEXT out | |
| of the kernel core; `z_process_exec_begin/commit` are the exec contract. | |
| ### Signals | |
| - `k_sig_info` gains `pid`/`status` (gated on CONFIG_PROCESS) + `K_SI_CLD_*` codes — | |
| flows through the existing fifo/timedwait/SA_SIGINFO paths unchanged. | |
| - Process-directed fifo entries (`proc` pointer alongside `tid`); first eligible | |
| unmasked thread in the group receives (POSIX one-thread rule); zombie-process purge | |
| mirrors the existing dead-thread purge. | |
| - Global `signal_actions[]` moves behind `z_sig_actdb(proc)` — per-process sparse DB | |
| (~28 B × DB size × processes), global retained for !PROCESS. | |
| - STOP/CONT stay ignored; WUNTRACED → -ENOTSUP (job control out of scope). | |
| ### Process-creation primitives (user decision) | |
| - **`k_spawn()` — first-class citizen for non-MMU targets** (MPU *and* targets with no | |
| memory protection at all): creates a new process + initial thread directly from a | |
| spec (image entry per the K_PROCESS_DEFINE entry forms, domain/partitions, args/env | |
| block, initial-thread parameters) — no address-space duplication, **no return-twice | |
| arch machinery needed** (fresh entry, no context copy), so it lands before any | |
| `arch_thread_fork` work and does not require USERSPACE. POSIX `posix_spawn()` is a | |
| **thin wrapper around `k_spawn()`** (it lives in the POSIX_SPAWN option group with | |
| its own suite, per suite-per-group; `_POSIX_SPAWN` can be enabled when it lands — | |
| earlier than previously planned). `k_spawn` is also the runtime backend for starting | |
| statically defined processes. | |
| - **`k_clone()` — the MMU-tier COW vehicle**, modeled on Linux's clone/clone3: | |
| one general primitive taking a `struct k_clone_args`, whose only planned flavor | |
| is the COW address-space copy (M4) — POSIX `fork()` calls `k_clone()` with | |
| those flags directly. Declared `-ENOSYS` until then. Linux-shaped room to grow | |
| retained (threads-as-clone-flavor is explicitly NOT a goal). | |
| ### fork tiers | |
| - **Return-twice primitive** (needed only by k_clone's M4 COW flavor; not by | |
| k_spawn/sys_spawn): `arch_thread_fork(child, parent)` per arch — builds child | |
| context so its first schedule returns from the clone syscall with retval 0 at | |
| the same user PC/SP. No setjmp tricks. No `returns_twice` annotation is needed: | |
| each returner runs on its own address-space copy, so neither can touch the | |
| other's frame or spill slots. | |
| - **COW tier** (`CONFIG_PROCESS_FORK_COW`, x86 then arm64): **keep the global shared-VA | |
| invariant; carve out a private window**. Reserve `CONFIG_PROCESS_VA_SIZE` (default | |
| 8 MB) in the global `virt_region_bitmap` at boot; the invariant becomes "shared | |
| outside the process window". Page tables are already per-domain (x86 ptables/CR3, | |
| arm64 TTBR0+ASID) — new arch APIs `arch_mem_domain_{map,unmap,protect,query}_private` | |
| are thin non-propagating wrappers over existing internals (`ARCH_HAS_MEM_DOMAIN_ | |
| PRIVATE_MAP`). `struct k_process_mm` (per-process VA bitmap + mapping records) lives | |
| in k_process, keeping `struct k_mem_domain` unchanged (upstream-friendly). | |
| Page-frame refcount = parallel `uint16_t` array in kernel/mmu.c (the pattern the | |
| page-frame comment prescribes); **refcount>1 ⇒ pinned** (simple, correct; reverse-map | |
| later). COW fault handler `k_mem_cow_fault()` runs before demand paging on user write | |
| faults. Fork walk: RO/shared → share+ref; RW private → write-protect both + COW-mark. | |
| Initial constraint: **x86 !KPTI only**. COW-process thread stacks must be | |
| window-mapped (Kconfig-enforced via the existing mem-mapped-stack machinery). | |
| **Honest scope**: full fork isolation holds for private-window state (exec'd images, | |
| window heaps/stacks); fork-without-exec from the init process shares image globals — | |
| documented; fork+exec/posix_spawn is the supported story. | |
| ### Scheduling & per-core address-space switching (user concern — mostly already exists) | |
| Requirement: threads from many different processes run concurrently; any thread is | |
| loadable onto any of the N cores (unless pinned), with that thread's virtual-memory / | |
| protection state applied before it resumes. Assessment against the tree: | |
| - **The machinery exists in the context-switch path today.** The scheduler is | |
| thread-granular and process-agnostic (consistent with thread-group ≈ process), and | |
| each arch already applies the incoming thread's domain on switch: x86 loads the | |
| domain's page tables into CR3 (`z_x86_swap_update_page_tables`, | |
| arch/x86/core/userspace.c:40, called from both ia32 and intel64 switch paths); | |
| arm64 swaps TTBR0 per-domain with ASIDs (`z_arm64_swap_ptables`) so cross-process | |
| switches avoid full TLB flushes; ARM MPU reprograms regions from | |
| `thread->mem_domain_info.mem_domain` at every context switch | |
| (arch/arm/core/mpu/arm_core_mpu.c:238); RISC-V PMP likewise. Because `k_process` | |
| wraps a `k_mem_domain` and threads carry their domain, per-core process switching | |
| requires **no new scheduler work** — SMP included (each CPU switches independently; | |
| threads of one process may run on several cores at once sharing the domain's | |
| tables). | |
| - **Real gaps to watch**: (1) SMP TLB shootdown — when COW/private-window mappings of | |
| a domain change while sibling threads run on other cores, an IPI-based invalidate | |
| is needed (bring-up is UP qemu; flagged in the COW risk list); (2) arm64 ASID | |
| wraparound full-flush fallback; (3) x86 KPTI exclusion for the private window; | |
| (4) MPU slot budget when adding a per-process app region to the per-switch | |
| reprogramming set (the dynamic-MPU-region-sharing exploration above). | |
| - **Swapping processes in/out of memory (paging a whole process) is a separate, | |
| massive topic — explicitly out of scope**; the demand-paging substrate would be its | |
| starting point if ever pursued. | |
| ### Per-process resources (user requirement: every non-kernel process owns its own) | |
| Each non-kernel process (init and its descendants — not swapper/kthreadd, which use | |
| the kernel's global state) owns: | |
| - its **file descriptor table** (below), | |
| - its **list of process-specific memory allocations and mappings** — generalized from | |
| the COW-tier `k_process_mm`: a per-process resource ledger (mappings, heap/resource- | |
| pool ownership, dynamically allocated kernel objects charged to the process) exists | |
| for **all** processes under `CONFIG_PROCESS`, not just `PROCESS_FORK_COW`, so | |
| process exit can reclaim everything the process created. **Mapped pages (MMU) are a | |
| per-process resource, exactly like fds (user direction)**: `k_mem_map`-style mappings | |
| are owned by the **thread-group leader** rather than any individual thread, tracked in | |
| the process ledger, and inherently reclaimed when the leader exits (after the group's | |
| child processes are killed) — the same lifecycle as the per-process fdtable. The COW | |
| tier extends the same ledger with the private-VA window records; per-process heap = | |
| a `k_heap` assigned at `K_PROCESS_DEFINE`/creation (the resource-pool model, promoted | |
| from per-thread to per-process — retargets the `TODO(k_process)` pool-ownership marker), | |
| - its cwd, environment block (app-owned per the ABI), and signal dispositions. | |
| **Termination cleanup (user requirement)** — when a process terminates, its exit hook | |
| must fully unwind the resources it owned: | |
| - **Heap**: clear every allocation and return the backing memory to the kernel — | |
| whether it was `k_mem_map`-mapped pages or a physical region granted from some | |
| memory pool. Nothing the process allocated survives it. | |
| - **Shared-memory objects** (POSIX `shm_*` etc.) are **separate from the heap**: they | |
| are refcounted, with **exactly one reference per process** (not per mapping or per | |
| thread). On termination each such object the process holds gets its refcount | |
| decremented once, and the object is closed/destroyed only when no other process | |
| still references it. | |
| - **Design (user direction): make a shared-memory object a first-class kernel object | |
| of a new "memory" type** (e.g. `K_OBJ_MEM` — Zephyr today has only `k_mem_slab`, a | |
| slab allocator, and no generic memory-region kobject; `k_heap` isn't a kobject at | |
| all). As a kobject it plugs straight into the existing grant/validate/refcount | |
| machinery: a process is *granted* access to a memory object, access is checked at | |
| the syscall boundary, and lifetime/reclaim ride the same per-process kobject | |
| cleanup as everything else — the one-refcount-per-process rule is just the kobject | |
| grant count. This also unifies "explicitly granted physical region" and | |
| "shared-memory object" under one type. | |
| ### Per-process fds/cwd (user direction, revised) | |
| Under `CONFIG_PROCESS` the **global fdtable is not compiled in at all** — the | |
| `fd → fd_entry` handle is abstracted behind an `ALWAYS_INLINE static inline` | |
| accessor in a zvfs internal header, so a `!PROCESS` build resolves to today's global | |
| array (zero cost) and a `PROCESS` build resolves to `k_process_current()`'s | |
| per-process fd table. Per-libc file-descriptor→`FILE *` glue may need updating to go | |
| through the same accessor. CWD likewise becomes per-process. FD_CLOEXEC (already | |
| stored in `fd_flags`) is finally honoured by the exec hook. The open-file-description | |
| objects and their refcounts stay shared so `dup()` across a fork works. | |
| ### Sessions/groups | |
| `k_process_{getpgid,getsid,setsid}` syscalls; setsid: -EPERM for group leader, else | |
| sid=pgid=pid. fork inherits pgid/sid. `k_sig_queue(0/-pgid)` fans out. No setpgid/ | |
| terminals/job control. | |
| ### Kconfig | |
| `CONFIG_PROCESS` does **NOT** require USERSPACE — identity/exit/wait/SIGCHLD/pgid work | |
| for kernel-mode threads on every platform incl. native_sim (huge CI surface for the | |
| early patches). Isolation (both fork tiers) requires USERSPACE. Zero overhead when off. | |
| ### Kernel patch series (appended after existing 116; **USER** = hand-implement) | |
| | # | Patch | Risk | Who | | |
| |---|-------|------|-----| | |
| | 1 | `k-process-core.patch` (struct, pool, pid alloc, k_pid_t flag-day, hooks, getpid/getppid) | Med | **USER** core; fixups mechanical | | |
| | 2 | `k-process-exit-wait.patch` (exit, zombify, reparent, wait) | Med-high (SMP halt ordering) | **USER** | | |
| | 3 | `k-signal-process.patch` (si_pid/status, process-directed delivery, per-proc DB, SIGCHLD) | Med | **USER** | | |
| | 4 | `k-process-pgrp-session.patch` | Low | mechanical | | |
| | 5 | `k-spawn.patch` (first-class spawn: process+thread from spec, all targets, no USERSPACE requirement; backend for static-process start) | Med | **USER** review; largely mechanical over patch 1 | | |
| | 5a-d | `arch-{x86,arm64}-thread-fork.patch` (for M4 k_clone COW; MMU arches only) | High | **USER** | | |
| | 7 | `mmu-page-frame-refcount.patch` | Low-med | mechanical + USER review | | |
| | 8 | `mmu-domain-private-map.patch` (VA window, k_process_mm, 4 arch APIs x86+arm64) | High | **USER** | | |
| | 9 | `k-clone-cow.patch` (COW clone flavor, backs POSIX fork(): fork walk, COW fault, syscall) | Highest | **USER** | | |
| | 10 | `zvfs-per-process-fd.patch` | Med | mechanical | | |
| | 11 | `zvfs-per-process-cwd.patch` | Low | mechanical | | |
| | 12 | `k-process-exec-hooks.patch` (exec_begin/commit contract) | Med | **USER** + exec coordination | | |
| | 13 | `k-timer-msgq-process-retarget.patch` (the TODO(k_process) sites) | Low | mechanical | | |
| After 1–4: processes/wait/SIGCHLD on every platform (kernel-mode too). After 5 + | |
| M2.5 sys_spawn: spawn-ready on all four boards (exec-ready with M3). After 7–9: | |
| COW fork on qemu_x86(!KPTI)/a53. | |
| ### Kernel risks | |
| SMP zombify-vs-halt ordering (patch 2, review hardest); refcount>1-pinned memory | |
| inflation; arm64 ASID wraparound full-flush fallback; KPTI exclusion; shared signal | |
| fifo cross-process starvation; **kobject permissions move from the per-thread bitmap | |
| to the process (user direction)** — grants/queries key on `k_process` instead of the | |
| thread, so `CONFIG_MAX_THREAD_BYTES`=2 becomes a cap of **16 userspace *processes*** | |
| (not threads), with effectively unlimited threads per process all sharing the | |
| process's permission bits; this is a significant change to how the kernel grants and | |
| validates object access. Upstream buy-in for | |
| the Kconfig.vm wording change + 4 arch APIs (private-window formulation minimizes | |
| friction; k_mem_domain untouched). | |
| Additional discipline items (external review, adopted): | |
| - **`__syscall` annotation rigor**: every user-callable process API must carry the | |
| `__syscall` signature + generated `z_vrfy_` boilerplate — a missing annotation | |
| silently breaks the user-mode boundary and the minimal export surface; audit as | |
| part of each kernel patch's review. | |
| - **pthread↔k_thread mapping stability**: new `k_thread` fields (`process`, | |
| `process_node`) must not disturb the module's existing thread-handle mapping | |
| (`to_k_thread()` etc.) — accessor-based, so appending fields is safe, but verify no | |
| layout/offset assumptions exist. | |
| - **Coverage flush ordering**: gcov data must be flushed before process partitions | |
| are torn down — with dynamically loaded images, the reaper must sync coverage | |
| counters before `llext_teardown`/partition reclaim, or exec'd-code coverage is | |
| lost (interacts with the coverage-dump-at-init-exit move). | |
| --- | |
| ## Cross-design reconciliations | |
| 1. **`_exit` ownership**: common libc owns `_exit`/`_Exit` (the | |
| `libc-common-under-exit.patch`, `#ifdef CONFIG_PROCESS → k_process_exit`); the | |
| module does NOT add `_exit.c` (unistd.h declaration only). Supersedes the module | |
| agent's `_exit.c`. | |
| 2. **wstatus encoding — REVISED (kernel-native symbols requirement)**: the kernel | |
| owns the encoding via **public** `K_WSTATUS_*` macros (Linux-compatible bit layout | |
| so toolchain sys/wait.h macros agree) and `K_PROCESS_W*` wait flags; the module's | |
| W*/WNOHANG/… are thin derivations of those kernel symbols. The one POSIX-ism left | |
| in the wrapper: `waitpid()` re-encodes the signo field from kernel numbering to | |
| POSIX numbering at return (bit layout identical, only the signal number mapped), | |
| so user code's `WTERMSIG` yields POSIX signos. | |
| 3. **Kconfig**: the POSIX group stays the single user-facing unit | |
| (`POSIX_MULTI_PROCESS`); kernel symbols (`PROCESS`, `PROCESS_FORK`, | |
| `PROCESS_FORK_COW`) + promptless module helpers do the gating; the exec agent's | |
| `POSIX_EXEC`-style symbols become promptless internals (select LLEXT export groups, | |
| isolation choice). | |
| 4. **fork() on the MMU-less tier**: fork()=ENOSYS on MMU-less; `posix_spawn()` | |
| (M2.5 `sys_spawn()` + POSIX_SPAWN group) is the MMU-less process-creation | |
| story. Module test impact: `test_fork` asserts ENOSYS + COW sections on MMU; | |
| POSIX_SPAWN gets its own suite. | |
| 5. **exec image lifetime**: k_process gets an image-destructor hook (folded into patch | |
| 12) so the reaper tears down the LLEXT image; exec agent's deferred-unload flow | |
| depends on it. | |
| 6. **an385**: a spawned child sharing the parent's domain costs 0 extra MPU | |
| regions; post-exec domain needs merged LLEXT partitions (phase-2 patch) — | |
| phase 1 runs exec'd processes with ISOLATION_NONE. | |
| 7. **Exec core placement**: per the POSIX/kernel-separation requirement, the exec flow | |
| lands as `sys_process_exec/spawn` in lib/os (new `sys-process-exec.patch`); the | |
| module keeps only PATH resolution, varargs marshalling, and errno mapping. The | |
| args/env memory layout is **kernel ABI**; **crt0 is provided by the C library** | |
| (below POSIX); the executable build helper (cmake, crt0 + static libc link over | |
| `add_llext_target`) is Zephyr-side. The POSIX module ships no crt0, no export | |
| unit, no build helper. | |
| ## Ownership / working agreement (revised) | |
| - **Claude drafts, Chris reviews**: Claude produces first drafts/proposals for ALL | |
| parts, including the deep kernel/arch pieces previously marked hand-implement; | |
| Chris actively reviews and will likely rewrite many parts over multiple drafts and | |
| iterations. The patch table's **USER** markers now mean "highest review scrutiny", | |
| not "Claude doesn't draft". | |
| - **Pause at the end of each milestone/phase** to review what was done — likely via | |
| GitHub code review on the module branch (`posix-multi-process`). | |
| - **The plan is synced to `~/Desktop/posix-multi-process.md`** after each round of | |
| plan edits. | |
| ## Verification | |
| - Per milestone: `runci.sh -T tests/posix/multi_process` (+ `tests/posix/signals`, | |
| `tests/posix/threads_base` when touched), then full `runci.sh`; doc build at M4. | |
| - **CI matrix management**: the platform × libc × protection-tier × scenario matrix | |
| is large — shard twister runs in CI **grouped by protection tier** (MMU / MPU / | |
| no-protection) rather than by platform, to keep cache reuse high and local | |
| `runci.sh` runs scoped; wire into the existing CI-container/shard setup from day | |
| one of M1. | |
| - New kernel patches get their own Zephyr-tree tests (`tests/kernel/process`) following | |
| the k-signal.patch precedent (patch adds tests alongside implementation). | |
| - Patch hygiene: `west patch clean && west update && west patch apply` after edits; | |
| sha256sums via the patch-series-rebase workflow; smoke build on all four platforms. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment