Dagr is a schema-driven binary serialization format: you describe your data once and it generates zero-dependency reader/writer libraries in Swift, Rust, TypeScript, and more. This is part of a series on what it can do — the series index lists every post. You don't need the others to follow along; this one is about safely deprecating required fields.
Required fields have a bad reputation — and mostly it is deserved. But I think we can rehabilitate them.
If you have worked with Protocol Buffers or FlatBuffers, you probably know that required fields are frowned upon, because they get in the way of evolving your schema.
While developing Dagr, I keep searching for the inherent complexity of serialization: what actually motivates a given design decision, and what possibilities open up once you take it seriously. So let's start from first principles.
When a field is optional it can be either null (absent) or set to a value, and the setter/getter API in your target language has to reflect that. Crucially, because a reader already expects the field to possibly be absent and checks for its presence, an optional field is safe to remove or deprecate later — absence is a state the reader is prepared for.
So optional is a safe default. But it is also less ergonomic than a required field: nobody enjoys unwrapping optionals or checking for null everywhere, especially for a field they know is always set. The catch is that software offers no such certainty. Worse, a bad actor might hand you a buffer where a "required" field is missing, specifically to crash your application.
Taking all of this into account, Dagr also makes fields optional by default, but lets you mark them as required. And when a required field turns out to be missing at read time, we do not panic — we raise a manageable error instead.
A deprecated field still has a getter (you may need to read old data for migration) but no setter, since it is a field your new code should no longer actively write. This is exactly why deprecating a required field breaks forward compatibility: old code cannot read data produced by new code, because the required field it expects is simply gone.
If we could, we would get to keep both things we want: the ergonomics of a field that is always there, and the freedom to evolve it away later. It turns out the solution builds on a concept that Protocol Buffers and FlatBuffers users already know, though they use it for a slightly different purpose: default values defined in the schema.
In Protobuf and FlatBuffers, default values exist so a value can be elided from the binary. If a field is absent, the generated library returns the default instead. On serialization, the library checks whether a value equals the default and, if so, does not store it at all — saving space, safe in the knowledge that the reader will synthesize the default on absence.
Dagr takes this a couple of steps further. If a required field was defined with a default, it is safe to deprecate: on absence the reader synthesizes the value anyway, so old code keeps working — you just have to remember that a deprecated field always materializes to its default. And if a required field did not have a default yet, you can add one together with the deprecation flag. Seeing a required, deprecated field that carries a default, the serializer writes that default value into the binary, so old code still reads exactly what it expects.
In Protobuf and FlatBuffers, default values exist only for elision, so the schema supports defaults only for numeric and boolean types. In Dagr we decided to go beyond that and allow defaults for all field types — which means you can also define a default for a node.
To make node defaults ergonomic, we introduced a feature called prefabs: pre-fabricated node instances declared right in the schema.
Node(
"Color",
fields=[
"r" >> t.u8 >> required,
"g" >> t.u8 >> required,
"b" >> t.u8 >> required,
"a" >> t.u8 >> required,
],
frozen=True,
) >> {
"black": {"r": int_val(0), "g": int_val(0), "b": int_val(0), "a": int_val(255)},
"white": {"r": int_val(255), "g": int_val(255), "b": int_val(255), "a": int_val(255)},
"transparent": {"r": int_val(0), "g": int_val(0), "b": int_val(0), "a": int_val(0)},
},Above is a Color node with RGBA fields and three prefabs — black, white, and transparent.
This means that if I have a node with a required field of type Color and I want to deprecate that field, I can set one of the prefabs as its default. It will always be stored in the binary, so old code reads it without breaking.
Node(
"Paint",
fields=[
"kind" >> t.ref("PaintKind") >> required,
"color" >> t.ref("Color") >> required >> ref_val("black"),
"opacity" >> t.f32 >> required >> float_val(1.0),
],
packed=True,
),We designed prefabs explicitly for defaults and deprecation, but we stumbled onto another rather cool application.
In Dagr, an enum is stored as an unsigned integer representing the index of the case. This representation is very compact, but not very expressive.
What if we associate each enum case with a value?
Enum("HttpStatus", ["ok", "notFound", "teapot"], values=[200, 404, 418])In this small example we define an enum for HTTP status codes together with their integer values. On the wire we still store only 0, 1, 2, but the reader materializes 200, 404, 418. In other words, we persist the case in 2 bits yet hand back values that would otherwise take at least 2 bytes each.
And here is a more impressive example:
# PlanetInfo is a Node that defines prefabs "mercuryInfo", "venusInfo", "earthInfo".
Enum("Planet", ["mercury", "venus", "earth"],
value_type=t.ref("PlanetInfo"),
values=["mercuryInfo", "venusInfo", "earthInfo"]) # prefab names on PlanetInfoThe Planet enum has only 3 cases, so it fits in 2 bits — yet on read we can materialize a full, complex PlanetInfo node.
Dagr is full of small innovations like this: we fix a minor inconvenience — not being able to deprecate required fields — and, by combining features, land on surprisingly powerful results.
If you want to see any of this in action, the docs, examples, and interactive demos live at dagr.one.