Dagr is a schema-driven binary serialization format: you describe your data once and it generates zero-dependency reader/writer libraries in Swift, Rust, TypeScript, and more. This is part of a series on what it can do — the series index lists every post. You don't need the others to follow along; this one is about a container called SharedBuffer, for fixed-size, zero-copy buffers.
Back in June I showed Dagr to my friend Ivan Najman — walked him through everything you can do with a data graph: use it as a replacement for SQLite for your application data, lazy-load large configurations, build small network messages, yada yada yada. He looked at all of it and asked me: can I use Dagr as an interop layer between a WASM worker and a TS renderer?
He is building an interactive simulation, which he documents on his devlog. The simulation is written in Rust and compiled to WASM; the rendering is done in TypeScript in the browser. He already has an efficient way to exchange data between the simulation worker and the renderer — he just passes a pointer to a struct-of-arrays (SOA) data structure. The tedious part is keeping the access logic in sync between Rust and TS.
I was a bit baffled by the question, because this was not a use case I had in mind when I started building Dagr. But my mission is to discover the inherent complexity of data representation and provide solutions for it — so let's have a think.
What my friend does is this: he allocates a fixed-size blob in the WASM runtime, the simulation writes into the blob, and the renderer reads from it. No serialization, no deserialization — you just read and write values at the right positions in the blob. This is about as fast as data interchange gets. You do need to figure out concurrency, but that is a known problem with several known solutions.
There is a catch, though, and it is exactly the one my friend put his finger on. The speed is the easy part; what it costs you is a hand-written access layer — every field's offset, width, and alignment, encoded by hand and kept in sync across every language that touches the buffer. Get one offset wrong in the TS reader and it silently disagrees with the Rust writer. And this is not unique to WASM and TS: the same tax shows up in FFI, in GPU interop, in IPC — anywhere a fixed-size buffer crosses a language boundary. The layout is the one thing all sides must agree on, and it is the one thing everyone re-implements by hand.
Quite a lot, as it turns out. The whole idea is to describe that fixed layout once, as a schema, and let Dagr generate a byte-exact read/write overlay for every target language — so nobody hand-maintains offsets again. I already have the infrastructure for defining schemas — a Python DSL and a visual editor — and the ability to define nodes with fields, enums, and unions. For this use case, though, we should not reuse the DataGraph container — Dagr's default, for serialized graphs you read back later. Instead I introduced a new one called SharedBuffer.
The rules follow directly from "fixed size":
- Only frozen nodes are allowed. You cannot evolve a fixed-size container, so every node in a SharedBuffer must be frozen.
- Fixed-width fields are no problem — they already have a known size.
- Strings, arrays, and data blobs need a bound. For those I introduced a new
capacitymodifier: how many entries for an array, how many bytes for a string or a data blob. We reserve exactly that much space. - Unions become max-sized. We look at every case of a union, take the largest, and size the field to fit it — much like a C
union, except ours is tagged with the type. Arrays of unions are supported too. - Optionals reserve space plus a presence bit. We keep room for the value regardless, and add a bitset for the optional fields so we can tell whether a field is set or should be returned as
None/nil/null(whatever your language calls it).
One thing that caught my eye: because the whole layout is known at compile time, we can show it directly in the schema editor. That helps developers see exactly how the data will be placed in the buffer — how big the buffer is, what the alignment is, and how many padding slots are wasted. Speaking of padding: right now we lay the data out to minimize it, which I think is a great default, but I can imagine offering other strategies in the future.
A SharedBuffer can be useful even without any concurrency strategy. Take FFI: a foreign function call is blocking by itself, so you can allocate a buffer on the caller's side and pass just the pointer to the foreign function. The callee wraps that pointer into the SharedBuffer root type — which is really just an accessibility overlay — reads and mutates as it pleases, and returns. The caller then accesses the result through the same overlay.
The same goes for handing a complex but fixed-size structure to a GPU for parallel computation. Nowadays the typical interface for a GPU kernel is either a pointer or a tensor. But not everything is a tensor — or rather, you might be able to express your SOA structure as a tensor, but it is much nicer to have an actual domain-specific overlay. Here is a working example that uses a SharedBuffer to call a Mojo GPU kernel from Rust: https://github.com/mzaks/dagr-shared-buffers-ffi-and-gpu-example
Then there are cases like the one my friend described — a producer/consumer situation, which is common in WASM + Worker setups or in IPC scenarios generally. For those we introduced three concurrency strategies: seqlock, double_buffer, and ring. I will give a detailed explanation of each in a separate post, so I do not blow this one out of proportion. From the user's perspective, you wrap the pointer into a Producer or a Consumer and get the root node as an overlay in a safe way. And all three strategies are lock-free, so you do not have to worry about deadlocks. Here is a SharedBuffer-powered IPC example where a Mojo process runs the simulation and an Odin process does the rendering: https://github.com/mzaks/dagr-shared-buffers-ipc-mojo-odin-example
After I implemented SharedBuffer in Dagr, I went looking for something similar out there — and to my surprise I could not find much. FlatBuffers' struct leans slightly in this direction, but it is not even close to what SharedBuffer does. I have seen hints that some complex systems in the automotive industry have something like it baked in, but nothing general-purpose.
All in all, I think SharedBuffer has huge potential, because it is a simple way to unlock capabilities that most of the industry does not know it might need. Here is one example. I was talking with a fellow Mojo community member Owen Hilyard, who introduced me to P4, a language for network devices, which specifies parsers and actions based on the parse data. A SharedBuffer can be used as the header of a Dagr message, and thanks to P4 that header can be read directly on the network device — enabling extremely cheap packet identification, routing, and so on. (By the way, Dagr graphs and trees can define custom headers too — but that is a topic for another post.)
PS: docs, examples, and interactive demos live at dagr.one.