Design Document: A Portable `openssh`-Style Client Library

Context. The Rust openssh crate gives users a std::process::Command-like API for running commands on a remote host, while internally reusing a single authenticated SSH connection (OpenSSH's ControlMaster multiplexing). It ships two interchangeable backends behind one API:

process_impl — shells out to the system ssh binary for every operation.

native_mux_impl — talks the OpenSSH mux protocol directly over the master's Unix-domain control socket (no per-command ssh fork).

This document is a language-agnostic specification for reimplementing that library — the ergonomic surface API and both backends — in another language. It pays special attention to the binary mux protocol, because that is the part with no public, friendly reference outside the OpenSSH C source and PROTOCOL.mux.

Authoritative sources cross-checked while writing this: src/ of the openssh crate, the openssh-mux-client crate, the ssh_format serde crate, OpenSSH's PROTOCOL.mux and mux.c. Key facts below were verified against the actual code, not just the spec (the spec has at least one inaccuracy, noted inline).

Deliverable: on approval, this document is written to docs/PORTING_DESIGN.md in the openssh repo (new docs/ directory), including the Rust-mapping appendix in §11.

0. Mental model (read this first)

                    ┌─────────────────────────────────────────────┐
   your program     │  Public API: Session / Command / Child /    │
   ───────────────► │  Stdio / Error / SessionBuilder             │
                    └───────────────┬─────────────────────────────┘
                                    │  (compile-time or runtime dispatch)
                 ┌──────────────────┴────────────────────┐
                 ▼                                       ▼
        ┌──────────────────┐                  ┌────────────────────────┐
        │  process backend │                  │  native-mux backend    │
        │  spawn `ssh`     │                  │  speak mux protocol    │
        └────────┬─────────┘                  └───────────┬────────────┘
                 │ argv + OS pipes                        │ AF_UNIX socket
                 │ `ssh -S <ctl> …`                       │ + SCM_RIGHTS fd passing
                 ▼                                        ▼
        ┌───────────────────────────────────────────────────────────┐
        │   ONE long-lived `ssh` MASTER process (ControlMaster)     │
        │   listening on a Unix-domain control socket  <ctl>        │
        └────────────────────────────────┬──────────────────────────┘
                                         │ single encrypted TCP connection
                                         ▼
                                   ┌───────────┐
                                   │  remote   │
                                   │  sshd     │
                                   └───────────┘

The two backends are symmetric in how the master is created and differ only in how subsequent operations reach it. Both backends launch the master the exact same way: by spawning ssh in master mode (-M -N -f -S <ctl> …). The master forks to the background, opens the real SSH connection, and listens on the control socket <ctl>. From then on:

process backend: every command = a new short-lived ssh -S <ctl> … child process. The OS gives you the stdio pipes; the local ssh relays bytes to the master, which multiplexes them as a new channel over the one TCP connection.
native-mux backend: every command = a new AF_UNIX connection to <ctl> over which you speak the binary mux protocol yourself, including passing the three stdio file descriptors to the master with SCM_RIGHTS.

Both achieve the same thing: N concurrent remote commands sharing one authenticated connection and one auth handshake.

1. End-user API surface (what to replicate)

Keep the public types backend-agnostic; no backend type should leak. Model the API on the language's standard subprocess library so it feels native (Rust models it on std::process::Command).

Core types

Type	Role
`SessionBuilder`	Configures and launches a master connection. Fluent setters, then `connect()` / `connect_mux()`.
`Session`	A live (shared) master connection. Factory for `Command`s; can port-forward, health-check, and close.
`Command`	A builder for one remote command (program, args, stdio). Terminal ops: `spawn()`, `output()`, `status()`.
`Child` (a.k.a. `RemoteChild`)	A spawned remote process: `stdin/stdout/stderr` handles, `wait()`, `wait_with_output()`, `disconnect()`.
`Stdio`	How to wire a child's std stream: `inherit()`, `null()`, `piped()`, or from an existing fd/file.
`ChildStdin` / `ChildStdout` / `ChildStderr`	Async byte streams to/from the child; also exposable as raw fds (enables piping one remote command into another).
`Error`	One normalized error type across both backends (see §7).
`KnownHosts`, `ControlPersist`	Connection-policy enums.
`ForwardType`, `Socket`	Port-forwarding descriptors.
`OverSsh` (trait)	Lets a locally constructed `std`/`tokio` command be converted to run over a session. Rejects features SSH can't honor (env vars, cwd) with `CommandHasEnv` / `CommandHasCwd`.

`SessionBuilder` options (each maps to an `ssh`/`-o` setting; see §2)

user, port, keyfile, connect_timeout, server_alive_interval, known_hosts_check (Strict / Add / Accept), control_directory, control_persist (Forever / ClosedAfterInitialConnection / IdleFor(secs)), clean_history_control_directory, config_file, compression, jump_hosts, user_known_hosts_file, ssh_auth_sock.

Also support a resolve() step that parses ssh://[user@]host[:port] destinations and overlays them onto the builder.

`Command` semantics worth copying

arg() shell-escapes its argument; raw_arg() passes bytes through untouched. (args/raw_args are the plural forms.) This matters because the remote side runs the command through a shell — see §3.6.
command(program) shell-escapes the program name too; raw_command(program) does not.
shell(cmd) is sugar for raw_command("sh").arg("-c").arg(cmd).
subsystem(name) requests an SSH subsystem instead of a shell command (e.g. sftp).
Default stdio differs per terminal op: spawn() → all inherit; output() → stdin null, stdout/stderr piped; status() → all inherit.

Exit-status convention

wait() returns the remote process's exit status, except:

exit code 127 → normalize to a "command not found" error;
exit code 255 / no exit value → "remote process terminated" (see §7 for the 255 ambiguity and how the two backends differ here).

2. Connection management & lifetime

2.1 Launching the master (identical for both backends)

This logic is backend-independent: it just spawns the ssh CLI in master mode. Pseudocode:

launch_master(builder, destination) -> handle:
    socketdir = builder.control_directory
                or $XDG_RUNTIME_DIR-style state dir
                or ./                      # last-resort fallback
    if builder.clean_history_control_directory:
        remove every "<socketdir>/.ssh-connection-*" directory   # GC leaked masters

    tmp = make_temp_dir(prefix=".ssh-connection", in=socketdir)  # auto-deleted on drop
    ctl = tmp/"master"     # the control socket path (ssh creates the socket here)
    log = tmp/"log"        # master's stderr, captured for diagnostics

    argv = ["ssh",
            "-S", ctl,            # ControlPath
            "-M",                 # become ControlMaster
            "-f",                 # fork to background after auth
            "-N",                 # run no remote command (master only)
            "-E", log,            # write master stderr to log file
            "-o", control_persist.as_o_value(),    # ControlPersist=yes|no|<secs>s
            "-o", "BatchMode=yes",                 # never prompt
            "-o", known_hosts_check.as_o_value(),  # StrictHostKeyChecking=...
            ...connect_timeout, server_alive_interval, port, user,
               keyfile, config_file, compression, jump_hosts,
               user_known_hosts_file, ssh_auth_sock...,
            destination]
    set stdin/stdout/stderr = null
    status = run(argv).wait()              # returns quickly thanks to -f
    if status != 0:
        return interpret_ssh_error(read(log))   # see §7.2
    return handle{ tmp, ctl, log }

Notes:

-f makes ssh fork into the background after authentication completes, so the parent's exit status tells you whether auth/connection succeeded. The control socket exists by the time the parent returns 0.
The temp directory owns the lifetime: when it is deleted, the socket file and log vanish with it. Keep the temp-dir handle inside Session.

2.2 `ControlPersist` (how long the master lives)

Variant	`-o` value	Behavior
`Forever` (default)	`ControlPersist=yes`	Master persists until explicitly killed.
`ClosedAfterInitialConnection`	`ControlPersist=no`	Master exits when the last client disconnects.
`IdleFor(n)`	`ControlPersist=<n>s`	Master exits after `n` seconds idle.

server_alive_interval (→ ServerAliveInterval) keeps NAT/idle timeouts from killing long-lived masters.

2.3 Sharing & handoff

Session is cheap to share concurrently (it's just a path + temp-dir handle). All operations open their own transport to the master, so many Commands can run at once.
detach() → returns (ctl_path, log_path) and leaks the temp dir (prevents the destructor from cleaning up). resume(ctl, log) → rebuilds a Session from those paths with no owned temp dir (so it won't clean up on drop). This lets a master outlive the process that created it and be re-adopted later (e.g. across CLI invocations).

3. The mux protocol (native-mux backend) — the core of this document

3.1 Transport and connection model

The master listens on a AF_UNIX, SOCK_STREAM socket at <ctl>.
One mux operation = one fresh AF_UNIX connection. The client connects, exchanges HELLO, performs exactly one logical operation (alive-check, open-session, open-forward, …), and the connection is then either closed or — for a session — kept open only to read that session's exit message. There is no request pipelining of unrelated operations over a single socket in this design.
Therefore concurrency = many AF_UNIX connections to the same master. The master multiplexes them all into channels over the single TCP connection to the remote. (This is the key difference from the process backend, which gets the same multiplexing by spawning many ssh -S clients.)
A per-connection request_id counter exists (starts at 0, increments per request, wraps as u32) and is echoed by the server so replies can be matched to requests. Because each connection usually issues a single request, it's mostly a correctness check; still, always validate response_id == request_id and reject mismatches.

3.2 Message framing

Every message — both directions — is:

uint32  packet_length      // number of bytes that FOLLOW this field
uint32  packet_type        // one of the MUX_* constants
...     packet_body         // type-specific, packet_length-4 bytes

All integers are unsigned big-endian (network byte order).
packet_length counts everything after itself, i.e. 4 (type) + len(body).
packet_type is the first word of the framed body. (Implementation note: in the Rust serde encoding, packet_type is literally the enum variant index, which is why the numeric constants double as both message IDs and serde discriminants — a reimplementation can ignore that detail and just write the constant.)

3.3 Protocol constants (verified, current protocol version = 4)

SSHMUX_VER               = 4

# client → server
MUX_MSG_HELLO            = 0x00000001   # also server → client (handshake)
MUX_C_NEW_SESSION        = 0x10000002
MUX_C_ALIVE_CHECK        = 0x10000004
MUX_C_TERMINATE          = 0x10000005   # (defined; not used by this design — see note)
MUX_C_OPEN_FWD           = 0x10000006
MUX_C_CLOSE_FWD          = 0x10000007
MUX_C_NEW_STDIO_FWD      = 0x10000008   # (not implemented here)
MUX_C_STOP_LISTENING     = 0x10000009
MUX_C_PROXY              = 0x1000000f   # (not implemented here)

# server → client
MUX_S_OK                 = 0x80000001
MUX_S_PERMISSION_DENIED  = 0x80000002   # body has reason string
MUX_S_FAILURE            = 0x80000003   # body has reason string
MUX_S_EXIT_MESSAGE       = 0x80000004
MUX_S_ALIVE              = 0x80000005
MUX_S_SESSION_OPENED     = 0x80000006
MUX_S_REMOTE_PORT        = 0x80000007
MUX_S_TTY_ALLOC_FAIL     = 0x80000008

# forwarding type discriminants
MUX_FWD_LOCAL            = 1
MUX_FWD_REMOTE           = 2
MUX_FWD_DYNAMIC          = 3

Note on master shutdown: OpenSSH defines MUX_C_TERMINATE to kill a master, but this design closes a session by sending MUX_C_STOP_LISTENING instead (master stops accepting new clients and removes its listener socket; existing sessions finish). That matches what the reference Rust client does.

3.4 Handshake (HELLO)

Immediately after connecting, both peers send a HELLO; the client then reads the server's HELLO and checks the version.

client → server:                     server → client:
  uint32 packet_length (= 8)           uint32 packet_length (= 8, no extensions)
  uint32 MUX_MSG_HELLO                 uint32 MUX_MSG_HELLO
  uint32 4   (protocol version)        uint32 4   (protocol version)
                                       [string extension_name, string extension_value]*  # 0+, currently none

If the server's version ≠ 4, abort with an "unsupported protocol" error. Ignore any trailing extension name/value pairs (forward-compat).

3.5 Opening a session (the central exchange) + fd passing

This is "passenger mode": the client hands its three stdio fds to the master and then just waits. Sequence on a freshly-HELLO'd connection:

(1) Client sends MUX_C_NEW_SESSION:

uint32  packet_length
uint32  MUX_C_NEW_SESSION
uint32  request_id
string  reserved            # always empty (uint32 len = 0); server ignores contents
uint32  want_tty            # bool as u32 (0/1)
uint32  want_x11_forwarding # bool as u32
uint32  want_agent          # bool as u32
uint32  subsystem           # bool as u32  (1 ⇒ "command" is a subsystem name)
uint32  escape_char         # 0xFFFFFFFF disables escapes (see caveat below)
string  terminal_type       # e.g. value of $TERM; empty string is fine for no-tty
string  command             # the remote command line (or subsystem name)
[string environment_string]*  # OPTIONAL, zero or more "KEY=VALUE"; this design sends NONE

escape_char caveat. The spec says use 0xFFFFFFFF to disable the escape character. The reference Rust client is constrained by its char type and actually sends 0x0010FFFF (Unicode max). This is harmless for the non-interactive, no-TTY sessions this library creates (escape processing only applies with a TTY). In a reimplementation not bound by a 21-bit char type, prefer the spec value 0xFFFFFFFF.

(2) Immediately after, client passes 3 fds — one sendmsg per fd, in order:

send stdin_fd, then stdout_fd, then stderr_fd

Each fd is transmitted via a Unix-domain ancillary control message: a sendmsg(2) carrying a one-byte ordinary payload (0x00) plus a cmsghdr with cmsg_level = SOL_SOCKET, cmsg_type = SCM_RIGHTS, and exactly one int fd in the data. (Send them one fd per message, not three in one cmsg — that's what the master expects.) The passed fds must be in blocking mode (clear O_NONBLOCK); see §5.

(3) Server replies once:

uint32  packet_length
uint32  MUX_S_SESSION_OPENED
uint32  client_request_id     # must equal request_id
uint32  session_id            # remember this; exit message is keyed on it

…or an error: MUX_S_PERMISSION_DENIED / MUX_S_FAILURE (each followed by client_request_id then a string reason).

Spec vs. reality. PROTOCOL.mux mentions an additional MUX_S_OK "once the server has received the fds." The real master and the working Rust client do not exchange a separate MUX_S_OK for NEW_SESSION; the client treats MUX_S_SESSION_OPENED as the success signal and proceeds straight to waiting. Implement it that way.

(4) Client waits. The same connection now carries, eventually:

# optional, at most once, before exit:
uint32 packet_length
uint32 MUX_S_TTY_ALLOC_FAIL
uint32 session_id

# the terminal event:
uint32 packet_length
uint32 MUX_S_EXIT_MESSAGE
uint32 session_id           # must equal the opened session_id
uint32 exit_value

Handle three terminal outcomes:

MUX_S_EXIT_MESSAGE → remote exited with exit_value.
MUX_S_TTY_ALLOC_FAIL → record it (you could restore local TTY to cooked mode), then keep waiting for the exit message.
Connection EOF before any exit message → the remote process was killed by a signal (or the master died). Surface this as "terminated" with no exit value (see §7).

Session-open state machine

            connect AF_UNIX
                 │
                 ▼
        ┌──────────────────┐  send HELLO / recv HELLO (version==4?)──no──► ERROR(unsupported)
        │   HANDSHAKE      │
        └────────┬─────────┘
                 │ yes
                 ▼
        send NEW_SESSION; send stdin,stdout,stderr fds (SCM_RIGHTS)
                 │
                 ▼
        ┌───────────────────┐   recv …
        │  AWAIT OPEN       │── PERMISSION_DENIED / FAILURE ─► ERROR(reason)
        └────────┬──────────┘── SESSION_OPENED(session_id) ─┐
                 │                                          │
                 ▼                                          │
        ┌──────────────────┐ ◄──────────────────────────────┘
        │   RUNNING (wait) │── TTY_ALLOC_FAIL ─► (note it, keep waiting)
        └────────┬─────────┘── EXIT_MESSAGE(exit_value) ─► DONE(exit_value)
                 └──────────── EOF before exit ──────────► DONE(terminated/None)

3.6 Where escaping lives (important architectural point)

The mux command field is a single command string that the remote shell parses, just like ssh host "the command". So argument quoting must happen client-side before it goes on the wire:

Command::arg(a) → shell-escape a, then append " " + escaped to the command bytes.
Command::raw_arg(a) → append " " + a verbatim.

The native-mux backend builds the command as a running byte buffer (space-separated). The process backend instead lets the local ssh client assemble argv, but ssh also just concatenates and ships a string to the remote shell — so the same escaping rules apply to both backends. Reject commands containing NUL bytes (the wire string type can't carry NUL; see §4).

3.7 Health check

client → server:                    server → client:
  uint32 packet_length (=8)           uint32 packet_length (=12)
  uint32 MUX_C_ALIVE_CHECK            uint32 MUX_S_ALIVE
  uint32 request_id                   uint32 client_request_id
                                      uint32 server_pid       # master's PID; must be > 0

Use this for Session::check() in the mux backend.

3.8 Port forwarding

MUX_C_OPEN_FWD / MUX_C_CLOSE_FWD share one body shape:

uint32  packet_length
uint32  MUX_C_OPEN_FWD | MUX_C_CLOSE_FWD
uint32  request_id
uint32  forwarding_type        # MUX_FWD_LOCAL | MUX_FWD_REMOTE | MUX_FWD_DYNAMIC
string  listen_host
uint32  listen_port
string  connect_host
uint32  connect_port

Unix-domain endpoints: set the port to (uint32)-2 (0xFFFFFFFE) and put the socket path in the corresponding host string.
TCP endpoints: host string = address, port = the TCP port.
For MUX_FWD_DYNAMIC (SOCKS), there is no connect endpoint; send an empty connect host and the path-style port.

Replies:

MUX_S_OK (+ client_request_id) — success, fixed port.
MUX_S_REMOTE_PORT (+ client_request_id + allocated_port) — success with a dynamically allocated listen port (used by dynamic forwarding).
MUX_S_PERMISSION_DENIED / MUX_S_FAILURE (+ client_request_id + reason).

3.9 Stop listening (graceful master shutdown)

client → server:                    server → client:
  uint32 packet_length (=8)           uint32 MUX_S_OK + client_request_id
  uint32 MUX_C_STOP_LISTENING         (or PERMISSION_DENIED / FAILURE + reason)
  uint32 request_id

The master removes its listener socket and stops accepting new clients; existing sessions run to completion, then the master exits.

4. Wire-format serialization rules (`ssh_format`)

The mux protocol body is encoded with the SSH wire conventions. These are the exact rules (verified against ssh_format); implement a small encoder/decoder with them:

Datum	Encoding
`u8` / `i8`	1 byte.
`u16/i16`, `u32/i32`, `u64/i64`	2 / 4 / 8 bytes, big-endian.
`f32` / `f64`	IEEE-754, big-endian (not used by mux).
`bool`	encoded as a `u32`: `0x00000000` or `0x00000001` (4 bytes).
`char`	encoded as a `u32` of the Unicode scalar value.
string (`str`)	`uint32 length` prefix (big-endian) + raw UTF-8 bytes. NUL bytes are stripped and excluded from the length (the encoder removes `\0` from text strings).
byte string (`[u8]`)	`uint32 length` + raw bytes (NUL not stripped).
`Option::None`	zero bytes emitted.
`Option::Some(v)`	just the encoding of `v`.
unit / unit-struct	zero bytes.
tuple / struct / tuple-struct	fields concatenated, no length prefix, no field count.
sequence (`Vec`)	`uint32 count` + each element encoded in turn.
enum: unit variant	`uint32 variant_index`.
enum: newtype/tuple/struct variant	`uint32 variant_index` + the variant's field(s).
map	unsupported (never needed by mux).

Top-level framing. A complete packet is produced by reserving 4 bytes, encoding the body, then writing the body length into those first 4 bytes as a big-endian u32. The length value = number of body bytes (everything after the length field). This is the uint32 packet_length from §3.2.

Worked examples (from the encoder's own tests):

u32 0x12345678 → 00 00 00 04 | 12 34 56 78
bool true → 00 00 00 04 | 00 00 00 01
"Hello, world!" → 00 00 00 11 | 00 00 00 0D | 48 65 6C 6C 6F ...
A NEW_SESSION request is just the concatenation of: MUX_C_NEW_SESSION (u32), request_id (u32), reserved empty string (00 00 00 00), four bool-u32 flags, escape_char (u32), terminal_type string, command string — all wrapped in the outer packet_length prefix.

Implementation shortcut used by the reference client (optional): for variable-length messages it encodes the fixed prefix, then appends the big command/term/address byte-strings via scatter/gather (writev) to avoid copying, computing the outer length as fixed_len + 4 + var_len. The bytes on the wire are identical to encoding everything through the serializer; you can do the simple thing.

Reading responses. Read the 4-byte packet_length, then read exactly that many bytes, then decode: first u32 is the type, dispatch on it, decode the rest per the table. Ignore trailing bytes after a successfully decoded message (forward-compat).

5. File descriptors, I/O, and concurrency

5.1 The `Stdio` abstraction → fds

Stdio has four shapes; each backend turns them into an fd plus (optionally) a local handle the caller keeps:

`Stdio`	fd given to the child/master	Local handle returned
`inherit()`	the parent process's own `stdin`/`stdout`/`stderr` fd	none
`null()`	an fd for `/dev/null` (open RDWR once, cache it)	none
`piped()`	one end of a freshly created OS pipe	the other end, as async `ChildStdin`/`Stdout`/`Stderr`
from fd/file	the supplied fd	none

For a child's stdin the child gets the read end and you keep the write end; for stdout/stderr the child gets the write end and you keep the read end.

5.2 Blocking vs non-blocking (critical)

The fds handed to the master via SCM_RIGHTS must be blocking (O_NONBLOCK cleared). When you create a pipe with an async runtime, convert the child-facing end to a blocking fd before passing it; for caller-supplied fds, force-clear O_NONBLOCK.
The local ends you keep (ChildStdin/out/err) are the runtime's non-blocking async pipe handles, so the caller gets async read/write. Also expose them as raw fds, which is what makes "pipe one remote command's stdout into another remote command's stdin" possible (you hand command B an fd taken from command A's handle).

5.3 Concurrency model

process backend: concurrency is just OS processes. Each Command::spawn() launches an ssh -S <ctl> child; the async runtime drives its pipes. Set kill-on-drop on the child so a dropped Child tears down its local ssh (which closes the channel). Use a discard-port trick (-p 9) plus BatchMode=yes so the child only ever multiplexes through the master and never tries to open its own TCP connection if the master is gone.
native-mux backend: concurrency is many AF_UNIX connections (§3.1). Each Command::spawn():
1. converts the three Stdios to fds (+ keeps local handles),
2. opens a new connection to <ctl>, does HELLO,
3. sends NEW_SESSION + the 3 fds,
4. returns a Child wrapping the still-open connection (now in "await exit" state) plus the local stdio handles. Child::wait() reads on that connection until the exit message (or EOF). Dropping the Child closes the connection, which signals the master to end that channel.

5.4 `wait_with_output()`

Concurrently drain stdout and stderr to EOF while waiting for exit (don't serialize them, or a full pipe buffer can deadlock). Then assemble { status, stdout, stderr }.

6. Abstraction boundary between the two backends

Define one internal interface per public type; each backend supplies an implementation with identical method shapes. The reference uses a tagged-union + macro rather than dynamic dispatch:

Session  wraps  SessionImpl  = ProcessSession | MuxSession
Command  wraps  CommandImpl  = ProcessCommand | MuxCommand
Child    wraps  ChildImpl    = ProcessChild   | MuxChild

A delegate!(self, inner => expr) macro expands to a match over whichever variants are compiled in, so every public method is a thin forward to inner.method(...). Backend selection:

Compile-time: feature flags process-mux (default) and native-mux. Either or both may be enabled; with both, the enum carries whichever was constructed.
Runtime: the variant is chosen by which constructor you call — builder.connect() → process backend, builder.connect_mux() → mux backend.

In a language without conditional compilation, use an interface/abstract base class with two implementing classes and pick at runtime; the surface API stays identical.

Each backend must implement, behind the boundary: check, raw_command/subsystem, request_port_forward/close_port_forward, close, plus Command::spawn and Child::wait/wait_with_output/disconnect/stdio accessors. The master-launch code (§2.1) sits above the boundary and is shared.

7. Error handling & normalization

7.1 One error enum for both backends

Normalize everything into a single error type. Representative variants:

Variant	Meaning
`Master(io)`	Failed to set up / talk to the master process.
`Connect(io)`	Initial connection to the remote failed (categorized from stderr; see §7.2).
`Ssh(io)` (process backend)	The local `ssh` command itself failed to execute.
`SshMux(e)` (mux backend)	mux-protocol-level failure (carries the mux client error).
`InvalidCommand` (mux backend)	Command contained a NUL byte (unrepresentable on the wire).
`Remote(io)`	Remote process failed; includes the normalized "command not found" (exit 127).
`RemoteProcessTerminated`	Remote process ended with no clean exit value (likely a signal). Best-effort in the process backend — may actually be a remote exit code of 255.
`Disconnected`	Connection severed (best-effort).
`Cleanup(io)`	Failed to remove the temp dir.
`ChildIo(io)`	Setting up/operating on a child's stdio failed.
`CommandHasEnv` / `CommandHasCwd`	`OverSsh` rejected a command using features SSH can't honor.

7.2 Categorizing connection failures (both backends)

When the master launch or a connect fails, parse the ssh stderr / master log and map to an io-style error kind:

"Could not resolve hostname …" / "Network is unreachable" → generic/other
"Connection refused" → connection-refused
"Connection timed out" / "Operation timed out" → timed-out
"Permission denied (…)" → permission-denied
"Connection to … closed by remote host" → connection-aborted

Strip noise first (a leading ssh: prefix; "Warning: Permanently added …" host-key lines). This interpret_ssh_error step is what turns opaque exit code 255 from the master into a meaningful error.

7.3 The exit-code-255 ambiguity (the key cross-backend difference)

ssh uses 255 for its own failures (auth, connection dropped, …). But a remote program is also free to exit 255. So:

process backend cannot tell them apart from the child's exit code alone. It treats child-exit-255 as RemoteProcessTerminated, and exposes Session::check() so callers can disambiguate: check() runs ssh -O check; if that fails, read the master log (discover_master_error) to learn the real connection error, else report Disconnected.
mux backend has no ambiguity for the connection: a dead master surfaces as a socket error (mapped to Disconnected for connection-reset/refused/aborted/not-found kinds). The exit value still comes straight from MUX_S_EXIT_MESSAGE; an EOF before the exit message is reported as RemoteProcessTerminated (no value). The mux backend builds a wait-style status from the value (e.g. exit_value << 8 to mimic a Unix wait status), normalizing 127 → command-not-found.

7.4 `check()` per backend

process: ssh -S <ctl> -O check; on failure, read master log.
mux: open a connection and send MUX_C_ALIVE_CHECK; a valid MUX_S_ALIVE with a nonzero pid means healthy.

8. Teardown & cleanup

8.1 Explicit `close()`

process: run ssh -S <ctl> -O exit (tells the master to quit), check the master log, then delete the temp dir; surface deletion failure as Cleanup.
mux: open a connection and send MUX_C_STOP_LISTENING, then delete the temp dir.

In both, close() takes ownership, performs graceful shutdown, then removes the temp dir explicitly (so the caller learns about cleanup errors, unlike the destructor path).

8.2 Destructor / `Drop` (best-effort, synchronous)

If a Session is dropped without close():

process: synchronously spawn ssh -S <ctl> -O exit with stdio nulled, ignore errors (optionally log), then let the temp-dir handle delete the directory.
mux: call a synchronous "shutdown mux master" routine (open the socket with blocking std I/O, send STOP_LISTENING), ignore errors, then the temp dir is deleted.

The synchronous shutdown matters because destructors usually can't run async code. The temp dir's own destructor deletes the socket + log files regardless of whether the graceful shutdown succeeded.

8.3 Orphan & leak handling

Per-command children (process backend): kill-on-drop ensures a dropped Child's local ssh is reaped; the master then closes that channel.
Per-command connections (mux backend): dropping a Child/session closes the AF_UNIX socket; the master tears down the channel.
ControlPersist interplay: with =no, the master self-exits once the last client leaves even if your -O exit never ran; with =yes, an orphaned master can linger — which is why clean_history_control_directory exists: on the next connect, it sweeps <socketdir>/.ssh-connection-* and removes stale directories (and thus stale sockets) left by crashed processes.
detach() deliberately opts out of all of the above by leaking the temp dir; the master is then owned by whoever later calls resume() (or by a manual cleanup).

9. Suggested build order

Wire codec (§4): big-endian primitives, bool-as-u32, length-prefixed strings, the outer packet_length framing. Unit-test against the worked examples.
Master launch + SessionBuilder (§2): get a working master, control socket, and temp-dir lifetime. This alone makes the process backend mostly functional.
Process backend (§3.6, §5.3, §7.3): raw_command → ssh -S <ctl> -T -p 9 … -- cmd, stdio via OS pipes, kill-on-drop, 255/127 normalization, check, -O exit.
mux client (§3): HELLO, ALIVE_CHECK, NEW_SESSION + SCM_RIGHTS fd passing, exit-message waiting, OPEN_FWD/CLOSE_FWD, STOP_LISTENING. Test against a real master created in step 2.
mux backend (§5, §6): blocking-fd conversion, Stdio plumbing, Child/wait, wire it into the dispatch boundary.
Error normalization + teardown (§7, §8): unify both backends' errors; destructors; clean_history_control_directory; detach/resume.

10. Verification

Codec: golden-byte tests for each message (compare against the byte layouts in §3/§4).
End-to-end against a real sshd (e.g. localhost or a container): run a command and assert stdout/exit; run several concurrently and confirm a single master/auth is reused (check the master pid via ALIVE_CHECK and that no extra TCP connections open).
Interop: point the mux backend at a master launched by the stock ssh binary, and point stock ssh -S <ctl> at a master your library launched — both must work, proving protocol conformance.
fd passing: cat-style round-trip through piped stdin→stdout (the reference client's own test does exactly this).
Teardown: assert the temp dir/socket are gone after close() and after drop; assert clean_history_control_directory removes a deliberately-leaked dir.
255 disambiguation: kill the master out from under a running command and confirm the process backend reports a connection error via check(), while the mux backend reports Disconnected/RemoteProcessTerminated.

11. Appendix — mapping to the reference Rust implementation

For readers cross-referencing the original crates. Paths are relative to each crate root.

11.1 Public API & dispatch boundary (§1, §6)

Concept	Rust location
Crate root, re-exports, feature gates	`openssh/src/lib.rs`; features `process-mux` (default) / `native-mux` in `openssh/Cargo.toml`
`Session(SessionImp)` + `delegate!` macro	`openssh/src/session.rs` (`enum SessionImp { ProcessImpl(..), NativeMuxImpl(..) }`)
`Command = OwningCommand<&Session>`, `CommandImp`	`openssh/src/command.rs`
`Child = RemoteChild<'_>`, `RemoteChildImp`	`openssh/src/child.rs`
`Stdio` / `StdioImpl` (Null/Pipe/Fd/Inherit)	`openssh/src/stdio.rs`
`OverSsh` trait (+ `CommandHasEnv`/`CommandHasCwd`)	`openssh/src/command.rs`
`ForwardType`, `Socket`	`openssh/src/port_forwarding.rs`

Dispatch is a tagged union + delegate! macro, not dyn trait objects. Backend chosen by constructor: connect() (process) vs connect_mux() (mux). Both backends expose the same private method set; the macro forwards to whichever variants are compiled in.

11.2 Connection management & lifetime (§2)

Concept	Rust location
`SessionBuilder`, all options, `resolve()`, `launch_master()`	`openssh/src/builder.rs`
`ControlPersist`, `KnownHosts` enums	`openssh/src/builder.rs`
Temp dir (`.ssh-connection-*`) ownership	`tempfile::TempDir` held in each backend `Session`
`new_process_mux`/`new_native_mux`, `resume`/`resume_mux`, `detach`	`openssh/src/session.rs`
`clean_history_control_directory` sweep	`openssh/src/builder.rs`

11.3 mux protocol (§3, §4)

Concept	Rust location
`Connection` (AF_UNIX, `exchange_hello`, `read_response`, `write`, `get_request_id`)	`openssh-mux-client/crates/mux-client/src/connection.rs`
Protocol constants (`SSHMUX_VER=4`, `MUX_*`)	`.../src/constants.rs`
`Request` enum + `Serialize` (variant index = packet type)	`.../src/request.rs`
`Response` enum + hand-written `Deserialize`	`.../src/response.rs`
`Session`/`SessionZeroCopy` (flags, `escape_ch=char::MAX`, `term`, `cmd`)	`.../src/request.rs`
`EstablishedSession`, `wait`/`wait_impl`, `SessionStatus`, EOF→`Exited(None)`	`.../src/session.rs`
`Socket` (Unix port = `-2`), `Fwd` (LOCAL/REMOTE/DYNAMIC)	`.../src/request.rs`
`open_new_session_impl` (scatter/gather write + per-fd send loop)	`.../src/connection.rs`
`request_stop_listening` / `request_stop_listening_sync`	`.../src/connection.rs`
Wire codec: `Serializer`/`create_header`, `to_bytes`, bool/char/str rules, NUL stripping	`ssh_format/src/ser.rs` (decoder in `ssh_format/src/de.rs`)
Codec error type	`ssh_format/ssh_format_error/src/lib.rs`

Notable quirks already called out in the body: escape_ch is char::MAX (0x0010FFFF), not the spec's 0xFFFFFFFF, because of Rust's char type (§3.5); the reserved field is serialized as the empty string in NewSession's tuple &(request_id, "", session) (request.rs); no environment strings are sent.

11.4 native-mux backend (§5)

Concept	Rust location
`Session` (`check`→`send_alive_check`, `request_port_forward`, `close`, `Drop`→`shutdown_mux_master`)	`openssh/src/native_mux_impl/session.rs`
`Command` (`cmd: Vec<u8>` byte buffer, `raw_arg` appends `b' '` + bytes, `spawn`)	`openssh/src/native_mux_impl/command.rs`
`RemoteChild` (wraps `EstablishedSession`, `wait` maps `SessionStatus`, `exit_value << 8`)	`openssh/src/native_mux_impl/child.rs`
`Stdio`→`Fd` (`Owned`/`Borrowed`/`Null`), `set_blocking` via `fcntl(F_SETFL, !O_NONBLOCK)`, `/dev/null` cache, `into_blocking_fd()`	`openssh/src/native_mux_impl/stdio.rs`
fd passing	`sendfd::SendWithFd` over `tokio::net::UnixStream` (`SCM_RIGHTS`); local pipes via `tokio::net::unix::pipe`

11.5 process backend (§3.6, §5.3, §7.3)

Concept	Rust location
`Session` (`new_cmd`/`new_std_cmd`, `-S/-T/-p 9/BatchMode`, `check`→`-O check`, `discover_master_error`, `close`→`-O exit`, `Drop`)	`openssh/src/process_impl/session.rs`
`Command` (wraps `tokio::process::Command`, `kill_on_drop(true)`, `spawn`)	`openssh/src/process_impl/command.rs`
`RemoteChild` (255→`RemoteProcessTerminated`, 127→`NotFound`)	`openssh/src/process_impl/child.rs`
stdio	`tokio::process::{ChildStdin,ChildStdout,ChildStderr}` directly

11.6 Errors & teardown (§7, §8)

Concept	Rust location
`Error` enum, `From<openssh_mux_client::Error>` (→`Disconnected`), `interpret_ssh_error`	`openssh/src/error.rs`
`Session::close` (calls backend `close`, then `TempDir::close`→`Error::Cleanup`)	`openssh/src/session.rs`
Synchronous master shutdown for `Drop`	`openssh-mux-client/.../shutdown_mux_master.rs` (`request_stop_listening_sync`)

11.7 Third-party crates the reference leans on (find equivalents)

tokio (async runtime, process, unix pipes, UnixStream) · tempfile (auto-cleaned temp dir) · serde + ssh_format (wire codec) · sendfd (SCM_RIGHTS fd passing) · shell-escape (arg quoting) · libc (fcntl) · once_cell (cached /dev/null) · typed-builder (Session builder). A port should map each to its ecosystem's counterpart; none are load-bearing on Rust specifically except where noted (the char/escape_ch quirk).

cstrahan/openssh_crate_analysis.md

Design Document: A Portable openssh-Style Client Library