RFC: Well-defined boundaries between bare-metal rust and "the runtime"

down-the-rabbit-hole.md

Executive summary

• Radical re-org of the std lib layout. Tease out stuff that lives in librustc that should be in the std lib (all small-r-runtime functionality to include TLS, malloc, fail!(), etc etc etc .. i don't have a great grasp on what all this includes)
• Global re-evaluation of patterns/practices used in libstd and better understanding of rules/implications of coupling any given language/library feature to any given aspect/level of the holistic concept of "the runtime"
• getting libuv-coupling out of libstd is the ultimate goal.. but a lot of other stuff shakes out as well, when taken to its logical conclusion
• Add a nort {} block annotation, akin to unsafe {}, where the compiler can guarantee that some arbitrary set of "small-to-big-R runtime" functionality is not used. What this arbitrary set is: Not sure.

How things are, today

• libuv is a submodule of mozilla/rust, built and statically linked into librt
• EventLoop and IoFactory's default impls are atop libuv
• trait object cruft in std::rt::rtio is strongly coupled to libuv impls of rtio traits
• code running in the main() function runs within a Task/Scheduler (implicitly always above libuv)
• getting to a "zero.rs" state requires a noticable burden on the part of the programmer
• a huge amount of coupling to "the small-r runtime" is in libstd, as well as in librustc (other places too?)

What I would like, in the abstract (plus asides about definitions & terminology)

• surface all of the "small-r runtime" coupling and make it replacable
• libuv out of the main tree and as a hard dep
• The ability to (easily) substitute in new EventLoop/IoFactorys, as needed, in a sane and clean manner
• Not put programmers in situations where "big-R Runtime Baggage" leaks into code running in critical/bare-metal contexts (kernel/driver dev)
• "Runtime Baggage" is a fuzzy term that means different things at different levels with impact on compiler, small-r-runtime and language semantics.

A schizophenic exposition on the proposal

• A bright line is drawn around every language/compiler/library feature that is "part of the small-r runtime". A non-exhaustive list (spanning language/compiler/library features at different levels):
  • TLS
  • comm
  • Task/Scheduler stuff
  • fail!()
  • conditions
  • malloc, free, lock/mutex/barrier/synchronization, etc primitives as leveraged by rustc to place into compiled programs and make them work on their target platform and runtime context
  • all IO primitives within and/or without the context of scheduler (do we keep a separate path for maybe-truly-blocking, fast-path ie "platform-backed" IO?)
  • what else? This has to include anything that has to be stubbed out in zero.rs, not just std::rt or things event-loop-related.
• The main() function entry point, out-of-the-box, doesn't run inside of a scheduler/task context. It runs directly on the main thread.
• Implicitly, in line with above: A simple hello world on the main thread, making use of no scheduler-backed-IO related calls (eg the default println() impl), actually runs in a single thread for the entire process lifetime, full stop.
• MAJOR FALLOUT:
  • A set of traits are defined in a new (ugh) ur-crate (libkernel, libspec? help me out, here..) . it includes traits for all "small-to-big-R runtime" functionality.
  • All runtime implementation that is not libuv/eventloop/io-specific is in a newly created libstdrt (have a better name?). Could we get most of the c++ to be called from here, instead of from libstd ?
  • Everything that's libuv-specific (just std::rt::uv, after libuv-coupling is factored out of the rest of std::rt) goes into a new, blessed crate. Users explicity link this "the hard way" via extern mod.. i guess just make sure its in the ld-lookup path. not sure.
  • Everything else stays in the existing libstd. See discussion of nort {} block below
  • This implies that there would be large and non-trivial overlap between the module/namespacing layout of libstd and libstdrt.
    • My personal preference would be that they both worked under the std module hierarchy and "overlay" each other.. this would probably require compiler-hackery/rules-exemption to allow as a special case.
    • Alternatively, we have shadowed module layouts in two crates.
    • In any case, it implies an audit to discover and deal with all uses of things like TLS and fail!() in libstd.
  • Questions/apprehension has been brought up in the past about how to deal with the consequences of this: How to redesign libstd so that given assumptions, namely (but non-exhaustively) the existence of fail!() and TLS, are accounted for? Is fail!() always available? Perhaps it just means a process abort in zero.rs scenario (Is an attempt at task unwinding moved into the libuv crate, while listdrt's default strategy is to abort? Perhaps there's an implicit order-of-precedence for which "version" if something like fail!() is used based on what's available at build-time? mumble-mumble the compiler employs a heuristic, driven by attributes, to determine which versions of small-r-runtime functionality get linked into the final program)
• libuv and std::rt::uv leave the rust tree (or at least get put into their own crate... and linked via special build rules) .. this is in-line with the "blessed crate" scenario discussed at times in the past
• To not use libstdrt, a user should be able to opt-out at build-time and supply a replacement library (or something in-source) .. bring the "zero.rs scenario" into the main story. Using zero.rs is an opt-out action by the programmer/build
  • Bottom line: if you choose not to use libstdrt, you have to provide a replacement (whether a wrapper lib of it's platform-specific syscalls, absolutely bare minimum stubs, etc)
  • This also implies that things in libstd can use traits defined in libkernel, whose behavior might change, from build-to-build, based on compile/link-time environment configuration (a fail!()'d out of bounds check in std::vec might be an process-abort in one build scenario, while it'd be task unwinding in another).
  • Obvious opening for dependency hell in a large, binary-artifact-driven package ecosystem (does community-maintained libfoo ship runtime-friendly and zero-friendly bins?)
• To opt-in to using the "higher-level" runtime, you have to explcitly initialize the runtime. Each implementation (including the OOTB libuv blessed-crate) would have a wrapper fn/macro to do this setup. It would, presumably, be called very early in the program's lifetime.. like the first line of main()
  • Running within this context, rust code behaves "the same" as it does today (all of the background threads and attendant benefits)
• Add a new block annotation, akin to unsafe that specifies nort {} (or perhaps zero {} ?) blocks of code. What this consists of is up for discussion. At a minimum:
  • ABSOLUTELY nothing that would be in std::rt::uv or std::rt::io
• Things that would be defined in libstdrt or a replacement that could be forbidden in a nort block:
  • fail!()
  • managed box use
  • TLS access
• Things that would be defined in libstdrt or a replacement that could be allowed in a nort block:
  • unique box creation (malloc/free)
• I would want to know more about what is provided in zero.rs before making a definitive list. Everything in the above lists is up for discussion/modification

This strikes me a possible net-increase-in-complexity of the compile-time/run-time semantics of the rust.

Is it worth it?

Is there an easier way to get the same effect?