Skip to content

Instantly share code, notes, and snippets.

@puremourning

puremourning/capnp-type-alias-design.md

Last active July 17, 2026 18:06
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save puremourning/ef74982663d97c4a263ce02860f2528e to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save puremourning/ef74982663d97c4a263ce02860f2528e to your computer and use it in GitHub Desktop.
Download ZIP
Raw
capnp-type-alias-design.md

Cap'n Proto 'alias' feature requirements

TL;DR

This proposal implements this item from the roadmap:

Type aliases: Ability to define a type which is just an alias of some other type, and have it show up as e.g. a typedef in languages that support that. (The current using keyword is intended only for local use and does not affect code generation.)

In summary, I propose supporting the "newtype" idiom in Cap'n Proto schema defintions - a general purpose "type macro" that inline the underlying type representation at usage site, and are thus fully wire-compatible with existing messages, code generators, parsers, etc. but allows newtype-aware code generators to produce semantic types in place of the inlined types.

The proposed syntax is a type declaration that can name any type which can appear in the type slot of a field declaration. This, I propose, subsumes "inline structs" with generalised "type aliases" (or newtypes if you will) and provides almost all features of inline structs along with strong semantic typing using a single syntax and compatible implementation.

type <name> = <type> [annotations]

Here:

  • <name> is a type identifier (UpperCamelCase per the usual rules)
  • <type> is the RHS of the : in a structure field declaration
  • [annotations] are optional annotations following the same rules as the above <type> slot

Thus we can decalre not only "inline structs" but "semantic types" as well, using a unified syntax:

// inline struct
type Vector2f = group {
   x @0 :Float32;
   y @1 :Float32;
};
// struct with inlined Vector2f fields
struct Rectangle {
    topLeft @[0-2] : Vector2f;
    botRight @[2-4] :Vector2f;
}
// semantic type with annotations
type UUID = Data $Json.hex $MyCompany.myCustomAnnotation;
// types referencing types
type UserId = UUID;

// nested inline structs
type Price = struct {
    value @0 :Int64;
    scale @1 :UInt32;
};
// inline union
type OrderPrice = union {
     priced @[0-1]: Price,
     unpriced @2: Void
};
struct Order {
    orderId @[0] :UUID;
    price @[1-2,4] :OrderPrice;
    ownerId @[3] :UserId;
}

Encoding wise, this is identical to copying and pasting the RHS of the type into the type slot of the enclosing struct, recursively.

Schema-wise, we introduce a new node kind and reference it from the Field and Type entries, but otherwise remains mostly compatible with any existing code generator. Code generators can be enhanced to generate wrapper types using compile-type offset mappings, and runtime offset tables.

In essence this is the newtype pattern when combined with a code generator that uses strongly-typed versions of field accessors/mutators, but does not require this of all generators.

Background

Cap'n Proto supports arbitrary nested structs, annotations on fields, structs etc. and the Cap'n proto compiler supports a form of alias declaration using.

The Capn'n Proto wire format requires each field in a struct to have a unique and predictable ordinal number such that the layout algorithm can determine an unambiguous location (offset) for each field in the serialised data. Currently this is done by having the user specify the ordinal number in the capnp schema file explicitly.

When a struct is embedded in another struct, this field type is represented as a pointer to the embedded struct, allowing for the embedded structs layout to be determined completely independently of the parent struct. This allows for the embedded struct to be reused in multiple parent structs without having to worry about the layout of the parent struct, and for it to change according to the schema evolution rules without breaking the parent struct.

The using alias feature is 'compile time only' (compilation of the schema to code, that is). What that means is it allows a certain amount of boilerplate to be avoided in the schema file, but this does not allow for any "semantics" to propagate to the generated code at 'usage time' (the point where the generated code is used by the programmer), or runtime.

Annotations cannot be applied to a using alias. Thus there is no mechanism in the schema syntax to apply a particular annotation in all places where a particular semantic type is used, where that type is represented as an alias.

Terminology

We use the term type or newtype to refer to a named type which is semantically distinct from any other type with the same underlying representation. This was chosen to be unambiguous.

However, originally, I used 'type alias' or similar 'alias' in this document, so any references to 'type alias' in this document should be read to mean 'newtype'. Hopefully it's clear from the context.

Motiviating examples

Here we present a few real, but perhaps somewhat contrived, examples of where the current Cap'n Proto features are limited in their ability to communicate semantics, and where a strong typedef, or newtype, style feature could be used to improve robustness and reduce boilerplate.

Principally, we would like to introduce strongly-typed named for semantic concepts in the schema language (and generated code) without compromising on runtime performance and to ensure that all uses of these types are consistent in their encoding and usage.

Note that the proposed syntax below is for illustration purposes only, and is not intended to be a final proposal, rather to convey the programmer's (theoretical) intent.

Inlining of vocabulary types

The primary motivation for inlining vocabulary types is to avoid the pointer indirection of the struct encoding. This is, in a sense, a pure optimisation and does not modify the expressability of semantics.

Consider a timestamp field, used by the programmer to represent a point in time. The wire representation of such a field might need to include:

  • The number of seconds since the some epoch
  • The number of nanoseconds since the last second
  • The timezone offset from UTC
  • The epoch itself (e.g. an enum of Unix epoch, GPS epoch, etc.)

A programmer will likely want to use a timestamp field in multiple structs, and will want to be able to use a strongly-typed Timestamp type in the generated code, but might wish to avoid the additional runtime cost of a pointer chase to a Timestamp struct, and instead have the timestamp field be represented "inline" within the parent struct, allowing for more compact wire representation and marginally faster runtime access to individual struct fields.

Currently we would represent this as follows:

struct Timestamp {
    seconds @0 :Int64;
    nanoseconds @1 :Int32;
    timezoneOffset @2 :Int16;
    epoch @3 :Epoch;

    enum Epoch {
        UNIX @0;
        GPS @1;
    }
}
struct Event {
    timestamp @0 :Timestamp;
    rxTime @1 :Timestamp;
    txTime @2 :Timestamp;
    //...
}

This works, but encodes the 3 fields as pointers, leading to a slight bloat of the message size and a slight increase of runtime access cost. By inlining the Timestamp type in its parent, we can save 3 pointers, potentially 3 allocations, and a few bytes by packing the offsets and epochs into the gaps in the Event struct (I presume; I have not actually checked the offsets, but let's imagine that there are a few bytes saved in the eventual encoding).

Annotations on vocabulary types

This example shows how it is currently not possible to define a primitive type alias (newtype) and also specify annotations on that type alias, requiring that all usages of the alias must also spell out the annotations.

Consider a UUID field, represented as a 128-bit integer. A programmer currently has 2 choices to represent this:

  1. A struct UUID with 2 UInt64 fields. As noted above, this carries some runtime overhead due to pointer indirection.
  2. A Data field with a strict length of 16 bytes. It's not possible to express this lengh restriction in the schema file, so we might invent an annotation @length(16). When encoding this as JSON, we might always want it encoded using hex: $Json.hex.

The later option is more compact and requires no runtime translation at all. That is we're representing the 16 bytes as a contiguous array and expressing the encoding semantics in the schema file.

However, if we want to use this type in multiple places, we must copy and paste the annotations:

using Json = import "/capnp/compat/json.capnp";
using UUID = Data;

struct IdentityProvider {
  providerId @0 :UUID $Json.hex $length(16);
  organisationId @1 :UUID $Json.hex $length(16);
}

We would like to be able to express this as something like:

using Json = import "/capnp/compat/json.capnp";
type UUID = Data $Json.hex $length(16);

struct IdentityProvider {
  providerId @0 :UUID;
  organisationId @1 :UUID;
}

It's important to note here that the annotations in question are actually field level annotations. And the effect of them is applied to the field on which the newtype is applied. This is a subtle but important distinction, and it means that the way we want to express annotations on types is different from the way we can currently attach them to structs, unions etc.

Semantic typing

This example follows from the previous one, and intends to avoid "type confusion" in the programmer's usage code. We would like to be able to expression that a particular field has a data type representation that is identical to some other field, but that they are semantically different types.

Again, consider a UUID type represented as above. Given our IdentityProvider type above, we have 2 fields which are both UUIDs, but contain different semantic data: providerId and organisationId. We would like to be able to express this in such a way that a user cannot put a providerId into a field that expects an organisationId, and vice versa.

using Json = import "/capnp/compat/json.capnp";
type UUID = Data $Json.hex $length(16);

type ProviderId = UUID;
type OrganisationId = UUID;

struct IdentityProvider {
  providerId @0 :ProviderId;
  organisationId @1 :OrganizationId;
}

Taking this a step further, we might want to express that an OrganisationId is actually PII (Personally Identifiable Information) and should be treated as such. We might define a $pii annotation, and apply it to the OrganisationId type, so that any field of that type is automatically tagged as PII.

using Json = import "/capnp/compat/json.capnp";
type UUID = Data $Json.hex $length(16);
annoation pii(field) :Void;

type ProviderId = UUID;
type OrganisationId = UUID $pii;

struct IdentityProvider {
  providerId @0 :ProviderId;
  organisationId @1 :OrganizationId;
}

Fixed size "array" types

One final example is a Vector3f (or similar). This type is represented as exactly 3 floats. In C we might represent this as something like:

union Vector3f {
  struct {
    float x;
    float y;
    float z;
  };
  float data[3];
};

That is, strictly 3 floats. We cannot represent fixed size arrays in Cap'n Proto, so we might represent this as a struct with 3 Float32 fields, but this is not ideal, as it requires a pointer indirection to access the Vector3f fields, and we might have many of these on a struct, such as a Rectangle:

struct Vector3f {
    x @0 :Float32;
    y @1 :Float32;
    z @2 :Float32;
}
struct Rectangle {
    topLeft @0 :Vector3f;
    bottomRight @1 :Vector3f;
}

Using a List(Float32) might be more compact (I forget the exact encoding of lists, but we can imagine it is), but still requires a pointer indirection, and does not express the semantics of the Vector type having exactly 3 dimensions.

However, it's undeniable that a Rectangle type having exactly 6 floats would be more compact than any alternative presented. Thus we'd like to be able to inline the Vector3f type into the Rectangle type:

type Vector3f = group { 
    x @0 :Float32;
    y @1 :Float32;
    z @2 :Float32;
}
struct Rectangle {
    topLeft @[0-2] :Vector3f;
    bottomRight @[3-5] :Vector3f;
}

Ad Extremum

Consider an Event with 3 timestamps, each of which is represented as a Timestamp type, and a few rectangles, keyed by some IDs.

in today's schema, this might look like this:

using Json = import "/capnp/compat/json.capnp";
using UUID = Data;
struct Timestamp {
    seconds @0 :Int64;
    nanoseconds @1 :Int32;
    timezoneOffset @2 :Int16;
    epoch @3 :Epoch;

    enum Epoch {
        UNIX @0;
        GPS @1;
    }
}
struct Vector3f {
    x @0 :Float32;
    y @1 :Float32;
    z @2 :Float32;
}
struct Rectangle {
    topLeft @0 :Vector3f;
    bottomRight @1 :Vector3f;
}

annoation pii(field) :Void;

struct Event {
    eventId @0 :UUID $Json.hex $length(16);
    objectId @1 :UUID $Json.hex $length(16);
    timestamp @2 :Timestamp;
    rxTime @3 :Timestamp;
    txTime @4 :Timestamp;
    boundingBox @5 :Rectangle;
    innerBox @6 :Rectangle;
    ownerId @7 :UUID $Json.hex $length(16) $pii;
}

This is clearly a completely contrived example, but it does illustrate how quickly we can approach something challenging to work with, especially if we are using Cap'n Proto structs as a primary data representation.

Syntax discussion

Here we present 3 possible approaches and discuss their strengths and weaknesses.

  1. struct keyword with inlinable modifier as proposed by Kenton
  2. struct with $inline(true|false) annotation as an extension of that idea
  3. type alias (newtype) with any valid :Type syntax on the RHS, including group {} and union {}

I believe that all of the 3 alternatives are aligned with Knton's comments in #907: they represent some form of named "group macro", reusing the fact that groups are already inline structs, and allow code generators to use offset tables to provide typesafe wrappers over them. The first two alternatives focus stricly on structs and the final one generalises to any type alias.

All cases require some way to map the ordinals in the 'inlinable type' into the parent type, and all cases use the same proposal: new syntax is introduced in struct definitions to specify a mapping from inlined ordinals to parent struct ordinals. This is done using the @[<range[,range...]>] where range is either:

  • A-B - a fully closed range of ordinals from A to B, inclusive
  • A - a single ordinal A

The choice of A-B is not arbitrary. Other options were considered and rejected:

  • [A:B] - this is too close to Python syntax where the meaning of B (exclusive) is different from the meaning here (inclusive).
  • [A..B] - this looks nice syntactically, but is much more difficult to implement in the capnp grammar/parser
  • [A;B] - this is a reasonable alternative, but the ; is already used in the grammar at a terminator and make parsing unnecessarily difficult.

A-B however can be parsed without any changes to the lexer.

NOTE: It was hoped that a syntax could be created for "B ordinals, starting at A", but there is not an obvious and intuitive syntax for this that would be both teachable, learnable and trivial to lex/parse, so this was rejected. In some of the below examples you might see @[A;B] - this was the illustrative syntax intended to mean "B ordinals, starting at A".

Ordinals are mapped one to one in declaration order with the first ordinal in <range...> mapping to the @0 field in the inlined type, the second ordinal mapping to the @1 field, and so on. Thus it is not possible to "miss" an ordinal out (leaving a gap), but it is possible to specify and incomplete mapping (leaving out a set of ordinals at the end of the inlined struct). In such cases, the compiler produces a warning, but does not fail to generate an encoding. In such cases, this is treated in the same way as if the inlined struct has been extended with new fields, and similarly for the parent struct (such inlined fields must by definition occupy some parent-ordinal higher than the current maximum parent-ordinal).

Can you inline types within inlined types? Yes! This is important for vocabulary types. The ordinals explicitly mapped in the inlined type are extended into the parent struct and must be explicitly mapped in the parentmost struct, as if using nested group{} or union{} values.

At runtime, in practice, the inlined structs must carry a pointer to the parentmost struct and an offset table. As the offset table is defined by the compiler (statically) it can be defined in any way that is convenient for the language runtime, and does not need to be agreed amongst different participants. One option is to define vtables for each inlined struct per-parentmost struct, but nothing prevents multiple indirections or declaration-order tables with simple offsets.

Alternative 1: struct keyword with inlinable modifier

Kenton proposed inlinable as a struct keyword:

struct Timestamp inlinable {
# ...
}

struct Event {
    timestamp @[0-5] :Timestamp;
    rxTime @[5-10] :Timestamp;
    txTime @[11-13,16,17] :Timestamp;
    foo @15 :Text;
}

Per Kenton, this would instruct code generators for the Timestamp struct to use an offset table to determine the location in the parent struct. Encoding-wise when placed in a struct, it would be indistinguishable from a group having the same fields, where the offsets are detemined by the explicitly specified range in the parent structs. This manual offset mapping is irksome, but necessary: the code generator must know the exact ordinals to use in the parent struct and schema evolution requires that the ordinals be stable across schema versions.

The main advantage of this struct syntax is that code generators will already know how to create a struct. And a 'named group' (as in type Foo = group {}) might not have an obvious representation in an existing code generator, other than having to recognise this pattern and generate a struct for it.

However there is no "don't inline in this context" equivalent, that would require noinline, which I suppose could work.

Taking the extreme example above, we get the following:

using Json = import "/capnp/compat/json.capnp";
struct UUID inlinable {
    data @0 :Data $Json.hex $length(16);
}
struct Timestamp inlinable {
    seconds @0 :Int64;
    nanoseconds @1 :Int32;
    timezoneOffset @2 :Int16;
    epoch @3 :Epoch;

    enum Epoch {
        UNIX @0;
        GPS @1;
    }
}
struct Vector3f inlinable {
    x @0 :Float32;
    y @1 :Float32;
    z @2 :Float32;
}
struct Rectangle inlinable {
    topLeft @[0-3] :Vector3f;
    bottomRight @[4-7] :Vector3f;
}

annoation pii(field) :Void;

struct Event {
    eventId @[0] :UUID;
    objectId @[1] :UUID;
    timestamp @[2-4] :Timestamp;
    rxTime @[5;4] :Timestamp;
    txTime @[9;4] :Timestamp;
    boundingBox @[13,14-16,17] :Rectangle;
    innerBox @[18;5] :Rectangle;
    ownerId @[24] :UUID $pii;
}

Alternative 2: $inline(true|false) annotation

The proposal here would be to create an annotation inline that can be applied to a struct or a field. When applied to a struct, it indicates 2 things:

  1. that the type can be inlined and thus might require a runtime offset table
  2. the default behaviour for all usages of this struct (to inline or not).

When applied to a field, it overrides the default defined on the strut. Thus, the inlining can be overridden per-field if for some reason it should not be inlined in a given context. Of course, the annotation can be applied, but by default the field is not inlined, requiring explicit opt-in at usage sites.

annoation inline(struct, field) :InlineDecl;

struct InlineDecl {
    is_inlined @0 :Bool = false;
    # TODO: What actually do we want to express here? Should it just be a bool?
}

The double-usage of this annotation might not be ideal. Why not inlinable(struct) and inline(field)? It's already a bit of an inversion of responsibility that you have to declare that a struct "may be used inline": it seems more natural to declare that a struct is inline, and then allow the user to override that default behaviour on a per-field basis. This could easly be changed without modification to the general proposal. Also inlinable is a bit of a mouthful, and inline is already a keyword in many languages.

It might seem that using an annotation avoids changes to the syntax and could lead to older compilers simply generating the wrong encoding, seeing the struct as a plain pointer-struct. But actually, in order to specify ordinals at usage sites of inlined structs, we have to introduce a new "range" syntax for mapping the inlined field ordinals to the parent struct.

Thus, the when a field is inlined in a parent struct, the ordinals must be specified using @[<range|list...>] syntax, even if there is only one field in the inlined type. It's tempting then to say that if ordinals are specified this way, then the struct is implicitly inlined. However, I think for the sake of a little additional punctuation, it is better to be explicit about the ordinals and any $inline(true) annotation, and indeed any $inline(false) override. The compiler can relatively trivially verify this and provide a clear and unambiguous warning to the user, and potentially fix-it like hints in a language server.

Taking the extreme example above, we get the following:

using Json = import "/capnp/compat/json.capnp";
struct UUID $inline(true) {
    data @0 :Data $Json.hex $length(16);
}
struct Timestamp $inline(false) {
    seconds @0 :Int64;
    nanoseconds @1 :Int32;
    timezoneOffset @2 :Int16;
    epoch @3 :Epoch;

    enum Epoch {
        UNIX @0;
        GPS @1;
    }
}
struct Vector3f $inline(true) {
    x @0 :Float32;
    y @1 :Float32;
    z @2 :Float32;
}
struct Rectangle $inline(false) {
    topLeft @[0-3] :Vector3f;
    bottomRight @4 :Vector3f $inline(false); # illustration: override the default
}

annoation pii(field) :Void;

struct Event {
    eventId @[0] :UUID;
    objectId @[1] :UUID;
    timestamp @[2-4] :Timestamp $inline(true);
    rxTime @[5;4] :Timestamp $inline(true);
    txTime @[9;4] :Timestamp $inline(true);
    boundingBox @[13,14-16,17] :Rectangle $inline(true);
    innerBox @[18;5] :Rectangle $inline(true);
    ownerId @[24] :UUID $pii;
}

Alternative 3: type alias with any valid :Type syntax on the RHS

The main disadvantage of both previous proposals is this is that neither provide a convenient alias or newtype syntax. In order to define a new type OrganisationId that is a UUID with a $pii annotation, we would have to define a new struct:

struct OrganisationId $inline(true) $pii {
    data @0 :Data $Json.hex $length(16) $Json.flatten;
}

Or, realistically, a 2-level struct:

struct OrganisationId $inline(true) $pii {
    data @0 :UUID $inline(true) $Json.flatten;
}

Encoding-wise, this works. But ergonomics-wise, it's a mess. You end up with a type OrganisationId in the generated code with a member data which is a UUID with a member data which is a Data. This is not ideal, leading to awkward usage code like this:

SomeMessage msg;
msg.setOrganisationId().setData().setData(uuidBytes); // or whatever

or in rust:

let mut msg = SomeMessage::new();
let mut o = msg.reborrow().init_organisation_id();
let mut d = o.reborrow().init_data();
d.set_data(uuid_bytes);

These cases are better expressed as a type newtype, but the neither proposal supports that. One option is to co-opt the Json.flatten annotation approach and allow single-field structs to be flattened in generated code. This is entirely doable, but still feels like a bit of a hack. A better approach would be to allow type newtypes as well.

So the third proposal is allowing something akin to "type macros" in the schema language which survive into the schema and thus generated code.

Thus we introduce type as a keyword to make it clear that we're defining a new name which appears in the type slot in a field declaration, with its annotations, etc.

type UUID = Data $Json.hex $length(16);
type OrganisationId = UUID $pii;

The quesion becomes what is valid on the RHS of such a type declaration. It seems reasonable to allow any declaration that could be on the RHS of a field in a struct. Afterall, the goal is to allow aliasing (or macros if you will) the values in the type slot of a struct field declaration (the bit after the :). Thus we might have:

type Foo = <privmitive type|struct name|enum name> [field-scoped annotations] ;
type Foo = <group {...}> [group-scoped annotations];
type Foo = <union {...}> [union-scoped annotations];

It follows then, that we allow any annotation in a type declaration that would be valid in the field slot of the appropriate type, and such annotations are merged with annotations specified in the field declaration inline. This merging can be trivial: if the same annotation is specified in both the type newtype declaration and the field, then the field-level override entirely replaces the type-level annotation. In the case of group and union, the annotations on the type newtype apply to the resulting group and inline annotations on the field. cases, where the annotations are merged in the same way (the only difference being, as today with a group/union field, the annotations must be group/union scoped when specified at the field declaration site).

Using the group {...} syntax to define inline structs this way is more honest about how they are "group macros" and embed in the encoding-space of the parent. Further, for semantic typing, we can avoid the awkward internal data fields by simply aliasing a primitive.

Importantly, this type Foo = <type decl> is a strict superset of the struct Foo <inlinable> syntax with the additional ability to alias primitive types and unions, with the concession that type Foo = group { ... } must always be inlined and there is no equivalent to "optionally" inline as in $inline(true/false), but there is no known usecase for the latter.

As mentioned above, there are still unique advantages to having a type alias feature that survives the compiler into generated code and can carry annotations. Most programming languages have some way to define types in terms of other types (using declarations, typedef, newtype patterns, etc.) and so it seems a simple addition to the schema language to add newtype:

type OrderId = UUID;
type UserId = UUID;

struct Order {
    orderId @0 :OrderId;
    ownerId @1 :UserId;
}

We can quickly imagine use cases where we might want to define a union and give it a name, and the group {} version is an entirely equivalent way of expressing an inlinable struct:

type OrderPriceInstruction = union {
    market :Void;
    limit :group {
        price @0 :Float32;
    };
    stop :group {
        stopPrice @1 :Float32;
    };
    stopLimit :group {
        price @2 :Float32;
        stopPrice @3 :Float32;
    };
}
type UUID = Data $Json.hex $length(16);
type Timestamp = group {
    seconds @0 :Int64;
    nanoseconds @1 :Int32;
    timezoneOffset @2 :Int16;
    epoch @3 :Epoch;

    enum Epoch {
        UNIX @0;
        GPS @1;
    }
}
struct Vector3f { 
    x @0 :Float32;
    y @1 :Float32;
    z @2 :Float32;
}
type Rectangle = group {
    topLeft @[0;3] :Vector3f;
    bottomRight @[4;3] :Vector3f;
}

struct OrderEvent {
    orderId @0 :UUID;
    ownerId @1 :UUID;
    timestamp @[2;4] :Timestamp;
    boundingBox @[6;6] :Rectangle; # ok one of these is not like the others...
    instruction @[12;4] :OrderPriceInstruction;
}

As can be seen above, the type X = T syntax is a superset of the struct syntax with inlining, where inlining is implicit in the use of a group. This is more honest than struct X $inline(true) { ... } as it is clear that the type is a group and thus incorporated into its parent struct.

Proposal

I propose to proceed with the type (newtype) approach and flesh it out below.

Encoding and layout

The encoding and layout of a struct field declared using a type alias is indistinguishable from the encoding and layout of the equivalent field definition having the RHS of the type alias pasted into the field's type slot, and applying the trivial 'annotation merge' operation: if an annotation $F is specified on the field, and also on the type alias, the alias-level annotation is dropped and replaced by the field-level annotation. Otherwise, annotation lists are concatenated.

Schema

So how would type aliases be represented in schema.capnp, and thus passed to code generation plugins in the CodeGeneratorRequest message?

It helps to review what we want to achieve:

  • We want structs with fields whose type is a type alias to be encoded in the exact same way as if the type alias was transparent (+/- annotation merging). Thus we would like the Node structure of the parent struct to look the same as it would in that case.
  • We want code generators to be aware of the type alias, and to be able to generate code that uses the type alias in the generated code, i.e. such that an inlined type setter uses a typesafe API
  • We need code generators to be able to build offset tables such that these typesafe APIs can be used to access the fields of the inlined types, whether they be compound or primitive. Offset data must include the discriminant in the case of unions.

The best representation of this is not obvious. Here are some options:

  1. A new kind of Node to represent a type alias, having a single Type member. Add member alias to the Type union which itself contains a typeId which points to the actual alias's node, similar to a struct but without branding.

  2. A Node to represent an alias. A new member on the Field and Type structs which optionally denotes the alias type aliasTypeId (or similar), with Field and Type otherwise unchanged, having the type alias expanded into the actual Field's slot or group entry and the actual Type's union member.

It turns out that we actually need both of these two, chosen so that (a) a naive code generator needs no changes to remain wire-compatible, and (b) newtype identity survives to code generators in every type position (fields, const types, List(UUID), annotation types), not just struct fields:

  • A new type Node kind (holding the target Type) is added for newtype-capable code generators. For an inline group/union newtype, the target is a struct Type whose typeId points at a separate template struct node.
  • We add typeId to Type containing the id of the type node it was written as (typeId 0 = none). Existing consumers switching on Type.which() and see the underlying type (e.g. Data); newtype-aware consumers additionally read typeId to recover the name.
  • Inline-group newtypes have no Type, so their newtype backreference is added to Field directly.

Thus, a 'compatible but naive' implementation still requires no changes: the compiler performs the expansion, and a code generator may simply ignore the type Node and the typeId back-references, and will produce complete, working, wire-compatible code. A newtype-aware code generator can use the type Node and back-references to produce semantic named types in the target language.

Let's explore how this impacts code generators in more detail.

Code generation

As briefly mentioned above, in the most trivial case, a code generator that does not wish to offer type safety, but remain syntax and encoding-compatible should have to do absolutely nothing other than ignore a Node of type type. The result would be as-if the newtype was not used and the RHS of the newtype was pasted into the field's type slot, with annotation merging having been applied by the compiler.

More advanced code generators can use Field.typeId (and Type.typeId) to generate type-safe APIs, and thus provide wrappers around the underlying real types.

The question remains how to generate offset tables per embedding. Each Field of group kind which has a Field.typeId requires generation of some new field mapping table. (Primitive types, or aliases which resolve to structs do not require mappings, only 'groups' : group or union aliases). The mapping table is constructed by enumerating the group's members and recording their actual offset in an array where array indexes correspond to the indexes in the group's fields list. These offset tables are created per-field per-struct and used at runtime by wrappers to determine the location of the fields.

Let's work through a simple example:

type Timestamp = group {
    seconds @0 :Int64;
    nanoseconds @1 :Int32;
}

struct Event {
    rx @[0-1] :Timestamp;
    tx @[2-3] :Timestamp;
}

The resulting CodeGeneratorRequest looks sort of like this:

Node(file) => {
    id = 0;
    nested = [1,2];
}

Node(type) => { // New node kind
    id = 1;
    displayName = "Timestamp";
    // Node.type is a `Type`; for an inline group newtype it is a struct type
    // pointing at the template struct node:
    type = (struct = (typeId = 3)); // 3 = the template struct Node, below
}

Node(struct) => {
    id = 3;
    scopeId = 1; // id of the parent `type` Node (above)
    displayName = "Timestamp.(template)";
    pointerCount = 0;
    dataWordCount = 2; // 1x Int64 + 1x Int32 = 2 words
    isGroup = false; // NB: the *template* is a plain struct, not a group -- a
                     // standalone isGroup=true node is rejected by the loader
                     // (isGroup requires a matching-size struct scope). The
                     // per-instance nodes below are the ones with isGroup=true.
    discriminantCount = 0;
    fields = [
        seconds(slot) { type = Int64; offset = 0; }
        nanoseconds(slot) { type = Int32; offset = 2; }
    ]
}

// Instance rx of "Timestamp" group inlined into Event struct
Node(struct) => {
    id = 4;
    scopeId = 2; // id of the parent type (Event)

    // From Event struct
    pointerCount = 0;
    dataWordCount = 3;

    // From template
    isGroup = true;
    discriminantCount = 0;
    fields = [
        seconds(slot) { type = Int64; offset = 0; }
        nanoseconds(slot) { type = Int32; offset = 4; }
    ]
}

// Instance tx of "Timestamp" group inlined into Event struct
Node(struct) => {
    id = 5;
    scopeId = 2; // id of the parent type (Event)

    // From Event struct
    pointerCount = 0;
    dataWordCount = 3;

    // From template
    isGroup = true;
    discriminantCount = 0;
    fields = [
        seconds(slot) { type = Int64; offset = 1; }
        nanoseconds(slot) { type = Int32; offset = 5; }
    ]
}

Node(struct) => {
    id = 2;
    displayName = "Event";
    pointerCount = 0;
    dataWordCount = 3; // Maybe .. 2x Int64 + 2x Int32 = 3 words
    isGroup = false;
    discriminantCount = 0;
    fields = [
        rx(group) {
            typeId = 1; // Field.typeId: the `type` node (the newtype's identity)
            group.typeId = 4; // the per-instance group node (rx) -- as for any group
        }
        tx(group) {
            typeId = 1; // Field.typeId: the `type` node
            group.typeId = 5; // the per-instance group node (tx)
        }
    ]
}

To summarise:

  • we define the newtype with a "template" struct node. Its layout is largely irrelevant; it represents what the group would look like if it were a struct. The type Node points at it, and the compiler uses it for the correct field metadata.
  • for each instance of the newtype we define a concrete group node (isGroup=true) with the correct offsets "as if" the group had been typed out in the parent struct. The stamped group field carries Field.typeId pointing at the type node (the newtype identity), while the field's group.typeId points at this per-instance node (as for any group).
  • as named unions are represented equivalent to a group with an unnamed union, the exact same approach is used (a group Node instantiated per usage with the discriminantOffset populated, and Field.typeId pointing at the type node).

Thus code generators can generate types when they see the type node, and offset tables when they see a group field with a Field.typeId by gathering the offsets into a simple array:

struct Timestamp {
    MessageBuffer* buffer;
    std::span<uint16_t> offsetTable;

    
    Timestamp(MessageBuffer* buffer, std::span<uint16_t> offsetTable)
        : buffer(buffer)
        , offsetTable(offsetTable)
    {}

    int64_t seconds() const {
        return buffer->readInt64(offsetTable[0]);
    }

    int32_t nanoseconds() const {
        return buffer->readInt32(offsetTable[1]);
    }
    // etc.
};

struct Event {
    static const std::array<uint16_t, 2> rxOffsetTable = {0, 4};
    static const std::array<uint16_t, 2> txOffsetTable = {1, 5};
    MessageBuffer* buffer;

    // ...

    Timestamp rx() {
        return Timestamp{buffer, rxOffsetTable};
    }

    Timestamp tx() {
        return Timestamp{buffer, txOffsetTable};
    }
}

Of course, this is illustration only - the actual C++ codegen might be wildly different from this. Indeed, as the tables are actually compile-time constants we can use template parameters to encode the table and monomorphise each instantiation at compile time, should we wish to (in both c++ and rust, I believe this should be possible).

Indeed, the code generators should try to produce optimal code. Each useage of a newtype requires a distinct set of offsets in its parent. One way to acheive this is a runtime offset table, such as in the previous example. However, given the offsets are known statically at compile time, we can generate a type which has those offsets baked in. In c++, we can use a template. In rust we can emit per-field calls with the offsets baked directly in the calls.

But that does leave an ergonomics problem: if each usage of a newtype is genuinely a distinct type, how do we write code that is generic over all usages of a specific newtype? For example, if we have a Timestamp newtype, and we have 3 fields rxTime, txTime, and timestamp in a struct, each of which is a Timestamp, we want both optimal code for each usage, and a single type to represent "any Timestamp" in generic code. In such cases, we must downshift to a runtime offset table. Indeed, we propse that the 'per usage' types are always translatable to a AnyReader (or AnyBuilder) equivalent, with a pracically identical API, but using a runtime offset table emitted by the compiler.

The following section describes the two-tier API in more detail.

Code Generation: C++ and Rust

We present the code generation for C++ and Rust as they are sufficiently similar that they can be described together, and happen to be the 2 cases that the author is familiar with and thus likely to implement.

Both code generators produce a two-tier API for an inline group/union newtype. The design is driven by one fact: because a newtype is stamped into each parent struct at that use site's ordinals, every use site is a distinct concrete type (its offsets are baked in). To recover a single semantic identity from those distinct types, each generator emits:

  1. A zero-cost, per-use-site form — the concrete type for that stamping, with its offsets baked in as compile-time constants. Field access is exactly as cheap as a hand-written group: no runtime offset table, no indirection. This is the common path.
  2. An erased form (AnyReader / AnyBuilder) — a single type that spans all use sites of the newtype, carrying the offset table at runtime. This is what lets you hold "any Timestamp" as one type — in a collection, as a return value, as a function parameter — regardless of which struct it was stamped into.

The two languages express the same shape with their native idioms:

  • C++. The per-use-site form is a class template parameterised on the offsets (Timestamp::Reader<0, 4>), monomorphised so access is resolved at compile time and stays zero-cost; the template name is the shared semantic type. The erased form is Timestamp::AnyReader / AnyBuilder, holding a runtime offset pointer, reached from the templated form via asAny().
  • Rust. In stable Rust, we cannot ergonomically parameterise a type over a variadic offset list, so we split the two roles the C++ template plays: each use site is a distinct concrete type, and a trait (timestamp::Reader / timestamp::Builder) is their shared name — used as impl timestamp::Reader bounds to write code generic over every use site (a union newtype's Reader adds which() returning a shared Which; a nested-newtype member is surfaced through an associated type). The erased form, timestamp::AnyReader / AnyBuilder, is a concrete struct over a runtime offset slice, obtained with as_any().

Both generated APIs are deliberately the same shape — a per-use-site zero-cost form plus an erased runtime-offset form — and the Rust AnyReader/AnyBuilder mirror the C++ ones. The mechanisms differ (C++ templates vs Rust traits) only because each language expresses "a family of types sharing one interface" with its native tool; the concepts match, keeping the two ecosystems' generated code recognisably the same.

Scalar newtypes (type UUID = Data) generate only a transparent name over the underlying type — a C++ using, a Rust type alias. These are wire-identical, carrying just the name and any merged annotations. C++ does not have a true "newtype" pattern without a lot of boilerplate, so we use a simple using declaration, while in Rust we are able to have a real newtype for "free".

One important caveat with the "erased" versions (AnyReader/AnyBuilder and the rust trait): Incomplete mappings. An @[...] may map fewer ordinals than the newtype has fields; the trailing fields are then unmapped at that use site. In both cases, reading an unmapped field yeilds its default value (from the template) and is not an error. However, attempting to set an unmapped field is a strict programming error. For concrete, per-use-site types a compile error is raised when trying to set an unmapped field, but for the erased form, we can only raise a runtime error:

  • In C++, the setter throws an exception
  • In Rust, the setter panics.

Generics

What would it mean to have generic versions of alias types?

There are 2 cases: Declaring concrete instantiations of generic types, and declaring partial instantiations of generic types. The former is trivial, the latter requires more work.

A non-generic newtype expressed as an instatiation (type Foo = List(Bar)) is fully suppported, and simply uses the existing branding machinery. The template node simply includes the requisite type parameters and isGeneric.

A fully generic, inlinable newtype may be feasible, but out of scope. The syntax would be type Entry(K, V) = group { key @0 :K; value @1 :V; }. This might allow to inline key and value into a parent struct, but they are always pointers _anyway, which sort of defeats the object of inlining.

However, there might be semantic typing benefits to allowing generic newtypes which alias generic types with the same, or fewer type parameters.

struct Entry(K, V) {
    key @0 :K;
    value @1 :V;
}

type MapEntry(K,V) = Entry(K, V);
type DictionaryEntry(V) = Entry(Text, V);

These are feasible but currently not proposed. If adding generic newtypes of the above form, there is nothing to stop us also adding type Entry(K,V) = group { ... } too, but again - feasible, but not proposed.

Note a type Optional(T) = group { present @0: Bool, value @1: T } alias for a bool and a T seems like a useful idea: the bools would be packed conveniently and the Ts would be slots as if otherwise inlined. But alas, it isn't any use: T would need to be a pointer type anyway. Thus the T naturally has a presence "bit" check due to the null pointer value. This is a limitation of generics: all type paramters must be pointer types, and so an Optional(T) would only be useful for primitive types, and we can already express that:


type OptionalInt32 = group { present @0: Bool, value @1: Int32 }
type OptionalInt64 = group { present @0: Bool, value @1: Int64 }
type OptionalFloat32 = group { present @0: Bool, value @1: Float32 }
// and so on

Constants

Values of alias types (e.g. in consts or annotation/default value): No changes, the values are simply instances of Value with the appropriate underlying type. There is no special syntax for this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment