dcload-ip fork

REPORT.md

dcload-ip Fork Report: Nonblocking Console and Reliable Reset

This report summarizes the local HyperSolar fork of dcload-ip under lib/dreamcast/dcload-ip. It is written for Dreamcast homebrew maintainers who want to understand what changed, why it changed, and which parts are good candidates for upstream discussion.

The short version: compared with upstream dcload-ip, this fork makes Dreamcast printf traffic safe for frame-time-sensitive programs. Upstream console writes are synchronous: the Dreamcast sends a write request and waits in bb->loop() until the host dc-tool replies. That is reliable for file-like I/O, but it means debug logging can stall the SH4 render thread. HyperSolar hit visible hitches when printing per-frame profiling data over dcload. This fork changes stdout/stderr writes to a best-effort, fire-and-forget UDP push while preserving the upstream blocking path for file I/O and oversized writes.

Comparison Target

This is a final-state comparison, not a commit-by-commit changelog.

Upstream non-fork comparison point in this checkout:

a78576cde532 - upstream dcload-ip 2.0.3 base

Fork state inspected:

8c97ac928272 - Add fire-and-forget console write support and revive console latch on host attach

The fork is intentionally small. Ignoring end-of-line normalization, the behavioral diff is:

11 files changed, 146 insertions(+), 3 deletions(-)

The raw diff is larger because several upstream CRLF files are LF-normalized in the fork. That has no behavioral intent.

Problem

The upstream target-side write() path does this:

1. Target sends CMD_WRITE to the host.
2. Host pulls bytes from target RAM through the usual binary-transfer path.
3. Host writes the bytes to its local stdout/stderr/file descriptor.
4. Host sends CMD_RETVAL.
5. Target exits bb->loop() and resumes the running program.

That is a fine model for file operations, where reliability and return values matter. It is a poor model for debug console spam from a real-time game loop. Every printf becomes a network round trip on the render thread. If the host terminal stalls, the host process is interrupted, or a UDP packet is dropped and retransmit behavior kicks in, the Dreamcast waits.

Observed in HyperSolar:

• Per-frame profiling prints caused occasional large frame hitches.
• The render workload itself stayed flat, pointing away from GPU/scene load.
• Killing the host console made the hitches vanish, but also removed the logs.

The conclusion was that console output should not be a reliable RPC. For stdout/stderr, dropping a log line is better than stalling a frame.

Main Behavioral Changes

1. Fire-and-forget console writes

Files:

• target-src/dcload/syscalls.c
• target-src/dcload/syscalls.h
• host-src/tool/syscalls.c
• host-src/tool/syscalls.h
• host-src/tool/dc-tool.c

The fork adds a new syscall packet:

CMD_WRITE_PUSH / DC22

Packet layout uses the existing command_3int_t header:

Field	Meaning
id	DC22
value0	host file descriptor, normally 1 or 2
value1	unused; data is inline, not a target RAM address
value2	byte count
inline payload	console bytes immediately after the command header

Target behavior:

• If fd is stdout or stderr, DCTOOL_MAJOR >= 2, and count <= 1440, the target sends one DC22 packet and returns immediately.
• It does not wait for CMD_RETVAL.
• It does not ask the host to pull data from target RAM.
• It runs one nonblocking receive pass after the send so an inbound reboot can still be serviced.
• It returns count, matching a successful console write from the running program's point of view.

Host behavior:

• dc-tool dispatches DC22 to dc_write_push().
• dc_write_push() writes the inline payload to the requested host fd.
• It deliberately does not send a return packet.

The push size cap is 1440 bytes. With 42 bytes of Ethernet/IP/UDP headers and 16 bytes of command header, that fits inside one standard Ethernet frame and the existing dcload transmit buffer. Larger writes fall back to the old path.

2. Blocking path kept for file I/O and oversize writes

Files:

• target-src/dcload/syscalls.c

The existing blocking CMD_WRITE path remains in place for:

• non-console file descriptors
• console writes larger than one push packet
• legacy host/tool combinations

That keeps file-like PC I/O reliable. The important semantic split is:

• stdout/stderr: best effort, nonblocking
• file I/O: reliable, return-value-driven

The fallback blocking path is still bounded by a timeout. If the host disappears, write() waits up to CONSOLE_WRITE_TIMEOUT_SECS and then latches console_dead. Later writes short-circuit rather than freezing the target.

3. console_dead self-healing

Files:

• target-src/dcload/syscalls.c
• target-src/dcload/syscalls.h
• target-src/dcload/commands.c

The fallback timeout/latch path has a Dreamcast-specific state issue: console_dead is a static in the loader-resident code path, and a soft reboot via dc-tool -r jumps back to the RAM loader without necessarily giving that state a clean power-on reset. Once latched, console logging can stay dead across later runs.

The fork adds:

console_revive()

It clears console_dead when:

• the host sends CMD_VERSION, which happens during every dc-tool attach
• the target handles CMD_REBOOT, before jumping back to the loader

This makes console availability follow the host connection again.

4. Standalone dc-tool -r handshake fix

Files:

• host-src/tool/dc-tool.c

The reset-only command path used to skip the upload/download flow. That flow is where prepare_comms() normally runs, assigning global_socket and discovering the negotiated dcload v2 port. Without it, standalone dc-tool -r could send on socket 0 on Windows and never reach the Dreamcast.

The fork runs a small prepare_comms() handshake before sending CMD_REBOOT in the -r command path.

5. Reboot reliability while a game is running

Files:

• host-src/tool/dc-tool.c
• target-src/dcload/adapter.c
• target-src/dcload/adapter.h
• target-src/dcload/lan_adapter.c
• target-src/dcload/rtl8139.c
• target-src/dcload/syscalls.c

A running game only services inbound dcload commands when it enters dcload's network loop through a syscall, usually a console or file operation. A single fire-and-forget reboot packet can miss that small polling window.

The host now sends a short reboot burst:

• REBOOT_RETRIES = 8
• REBOOT_RETRY_USEC = 10000
• total span is about 80 ms

That covers several likely poll windows and tolerates occasional UDP loss while staying below the longer post-reboot NIC reinitialization/link gap, so the target should not catch a late packet and reboot-loop.

On the target side, the fork adds loop_nonblock. When set, adapter bb->loop() does a single receive pass and returns instead of waiting for a reply. This is used after fire-and-forget console sends and after console-dead short-circuits so dc-tool -r remains responsive without putting frame timing at the mercy of a blocking receive wait.

The blocking write() path also calls bb->start() after receiving its ACK. Without that, cmd_retval leaves the NIC receive filter off until the next syscall rebuilds and sends a packet. A reboot arriving between printfs could be dropped at the NIC. Re-enabling RX lets the packet queue for the next poll.

6. On-screen fork marker

Files:

• target-src/dcload/dcload.c

The loader name now displays:

dcload-ip 2.0.3 - P5

The P5 marker is a practical hardware-debug canary. If the Dreamcast screen shows P5, the patched boot CD/loader is running. If it shows P5 but the host prints no console output, the common explanation is that the host-side dc-tool-ip was not rebuilt/installed and therefore does not understand DC22.

7. LF normalization

Files:

• example-src/startup_support.c
• target-src/dcload/dhcp.c
• target-src/dcload/dhcp.h
• target-src/dcload/memfuncs.c
• target-src/dcload/memfuncs.h
• target-src/dcload/perfctr.c
• target-src/dcload/perfctr.h
• target-src/dcload/startup_support.c

These files were converted from CRLF to LF. This is source hygiene only and should be reviewed separately from the behavioral patches.

Compatibility Notes

The host and target should be updated together.

Patched target plus old host:

• The target may send DC22.
• An old host that does not dispatch DC22 will not print those console bytes.
• The target will not hang, because it is not waiting for an ACK.
• File I/O and oversize writes can still use the old path.

Old target plus patched host:

• The old target never sends DC22.
• The patched host still handles existing CMD_WRITE traffic.
• The patched dc-tool -r path still benefits from the handshake and burst behavior where compatible.

Current compatibility caveat:

The target gates DC22 on DCTOOL_MAJOR >= 2, not on an explicit feature bit. That was enough for the local fork, but it is not ideal for upstream. A public version should negotiate console-push support explicitly during CMD_VERSION, or reserve a minor/patch version threshold that cannot be confused with older v2 tools.

Validation

This change was driven by real-hardware profiling, not emulator-only behavior. The relevant HyperSolar test case was a Dreamcast city-stage run with periodic profiling output over dcload.

Observed after the fire-and-forget console path:

• no random in-city logging hitches over a five-minute stage-3 hardware run
• render/vertex load unchanged between blocking and push console builds
• killing the host console no longer stalls the running game
• dc-tool -r remains usable as long as the running program still enters a dcload syscall path often enough to poll inbound packets

Known limitation:

If a running program never calls any dcload syscall after launch, it may not see an inbound reboot until it next enters dcload's network loop. The fork improves the common "game is printing or doing PC I/O" development loop; it does not turn dcload into an interrupt-driven background network service.

File Map

File	Role in this fork
host-src/tool/dc-tool.c	Runs prepare_comms() for standalone -r, sends reboot burst, dispatches DC22.
host-src/tool/syscalls.c	Adds dc_write_push() host-side inline console write handler.
host-src/tool/syscalls.h	Declares dc_write_push() and CMD_WRITE_PUSH.
target-src/dcload/syscalls.c	Adds console push, timeout/latch fallback, nonblocking poll, latch revival, RX re-enable.
target-src/dcload/syscalls.h	Declares CMD_WRITE_PUSH and console_revive().
target-src/dcload/commands.c	Revives console on host version attach and reboot.
target-src/dcload/dcload.c	Adds the on-screen P5 fork marker.
target-src/dcload/adapter.c	Defines shared loop_nonblock.
target-src/dcload/adapter.h	Exposes loop_nonblock to syscalls and drivers.
target-src/dcload/lan_adapter.c	Makes LAN adapter loop honor one-pass nonblocking mode.
target-src/dcload/rtl8139.c	Makes BBA adapter loop honor one-pass nonblocking mode.

Upstreamability Notes

Strong candidates:

• The dc-tool -r prepare_comms() fix.
• Reboot burst sending, with tunable retry count/delay.
• One-pass nonblocking adapter polling.
• Re-enabling RX after blocking write ACKs.
• A best-effort console channel that never waits for host ACK.

Needs cleanup before upstream:

• Add explicit feature negotiation for CMD_WRITE_PUSH.
• Document console push as lossy/best-effort and keep file I/O on the reliable path.
• Decide whether DC22 is the right permanent command ID.
• Consider whether P5 should become a proper fork/version suffix or be removed before upstreaming.
• Keep the LF-only cleanup as a separate patch so review does not mix formatting churn with protocol behavior.

Maintainer Takeaways

This fork treats dcload's console as telemetry, not storage. That distinction is the whole design. A game frame should never wait for a terminal emulator, a host stdout flush, or a dropped UDP ACK just to print debug text.

The reliable dcload syscall path is still valuable and still present. The fork only moves stdout/stderr-sized console output to a lossy push path, because logs are allowed to drop and real-time code is not allowed to hitch.

For development workflows, the second important theme is reset reliability. Once the console no longer blocks, the loader still needs a way to notice dc-tool -r. The nonblocking receive pass, reboot burst, standalone handshake, and RX re-enable all support that loop.

The result is a dcload-ip fork that behaves more like a real-time dev telemetry channel while preserving the old reliable protocol where reliability actually matters.