DTBN Draft

DTBN_Doc.md

Descriptive Typed Binary Notation (DTBN)

Revision 3501

---

1. Overview

DTBN is a substrate binary encoding notaion in which data is described in terms of its structure, type, and location. It's meant as a foundation format for information packaging, distribution, on-disk storage, and interchange. DTBN is descriptor-oriented: it focuses on providing complete descriptions of the data it stores or references.

DTBN is designed for flexible, typed (or dynamically typed) data that's easy to extend, and intended to be heavily extended and customized by implementors. This document exclusively defines the expected binary layouts and rules.

DTBN is optimized to operate efficiently across a wide range of storage sizes, from systems with as little as 64 KiB of memory up to the 64-bit address limit. The format minimizes overhead for individual data objects while efficiently amortizing descriptor costs across large homogeneous collections, and keeping reasonable costs for heterogeneous collections.

SideNote: "[Canonical]" Tag and related words like Canonical and Canonically refer to recommended guidelines to increase interoperability between different implementations. Or viewed from another way, describing a specific "Canonical" implementation. Anything referred to as Canonical is not mandatory.

1.1 Design Intent

The Overview lists intended use cases, which is worth being precise about: DTBN gives you the byte-level vocabulary to describe typed data: "this is a Uint32," "this is a 12-byte UTF-8 string," "this is a 5-field record", but it doesn't define a top-level file contract, a magic number, a required structure, or a guarantee that two independent implementations agree on anything beyond the TypeIDs they choose to share (and built-in types with defined representations). Building a real packaging format, storage engine, or streaming protocol on top of DTBN is exactly the intended use. DTBN just isn't, by itself, one of those things, a phrase isn't a sentence.

When building something, you tend to use a base to build upon: instruction sets, APIs, spec sheets, etc.

• DTBN is NOT:
  • A file format
  • A file system
  • A data streaming or interchange format
• DTBN is:
  • a low-level, self-describing binary encoding format

A Metaphor

If we metaphorically imagine implementations as gardens, what DTBN aims to be is the dirt of the gardens. DTBN is the dirt you grow the gardens on. How the garden looks and operates is decided by the gardeners.

Now, sometimes you may need to share your dirt with other gardens, so you may need to make sure it'll work for the other garden, so you might decide to follow certain guidelines, change its composition a bit, etc (see §8). You might want to transplant some of your plants to another garden as well, so the dirt may need to change a bit on the other garden so the plants can be planted there without dying.

Coming out of the metaphor: DTBN is the dirt, gardens are implementations and DTBN Systems, and gardeners are developers. DTBN is a substrate: it doesn't care about anything other than data description and representation. No validation, no error checking (see §4.4). That's the garden's and gardener's job, making sure the dirt is used effectively.

So, based on all the former text, the goal is not to have a single authoritative implementation of the spec, or to mandate interoperability between implementations. Those decisions are up to the implementors.

In one line: DTBN describes data; it doesn't define what you build with it or how.

Why DTBN?

The reason this format was conceived is not to solve an existing problem, rather, this is an attempt to provide a wide range of data storage, representation and exchange capabilities.

Implementors can engineer towards a small or large complexity burden, here are some examples:

• Ultra simple, no nesting parsers for embedded devices that only work with a subset of this document.
• Large scale distributed file systems that include multiple Type Registries administered dynamically.
• Data archives with lots of schemas and fixed structures repeated millions of times that hardcode their implementation-defined types into their implementations to trade flexibility for speed.
• Compile Time schema generators that use DTBN as the serialization format, leveraging structural types for extra features.

DTBN's main and most basic features are a high degree of self description and sequential parsing with the right layouts, so any system that would appreciate or require those capability can benefit immediately, while systems that don't require them can tweak their implementations and optimize for their use case.

DTBN can be considered a universal binary language for typed binary data, but doesn't specify a dialect; implementations can build a streaming parser, a zero-copy memory-mapped parser, a schema-bound parser, or a fully self-describing parser; all from the same binary language.

---

2. Terminology

2.1 Core Concepts

Term	Definition
Field	A typed value within an Entry.
Entry	Either a single typed field described by a FDH, or a collection of fields described by an EDH.
DTBNContainer	A complete MH + MHE + mandatory/optional FDHs/**EDH*s + Remote Link Entry; a "complete" DTBNContainer that, once fully parsed, allows parsing the Payload
Payload	Meta, Data or Remote Entry(s) described by an DTBNContainer.

2.2 Acronyms and Named Terms

Acronym	Full Name	Notes
MH	Main Header	1-byte header that begins every DTBNContainer. See §3.1.
MHE	Main Header Extras	Optional fields following the MH, present per MH flags. See §3.2.
EDH	Entry Descriptor Header	Describes an Entry consisting of multiple fields. Crucial Component of the format See §3.3.
FDH	Field Descriptor Header	Special Form of EDH that makes field count implicit, and allows to describe only 1 field. See §3.4.
TypeID	Type Identifier	The 16-bit value identifying a data type; also a built-in type in its own right (a type for storing a TypeID). See §6.4.

2.3 Field Count Terminology

These three terms are easy to conflate. They're defined once, here, and referenced by name everywhere else in this document (EDH, SchemaType, EnumType).

Term	Definition
Effective Field Count	Number of fields in an entry.
Field Count (the encoded byte)	Effective Field Count − 1. This 8-bit encoding yields an effective range of 1–256 fields.

SideNote: EnumType's "Variant Count" byte (§7.7) uses the same count − 1 encoding pattern, but it is a distinct concept from Field Count, it counts variants, not fields.

---

3. Binary Layout Reference

In this section we cover the components of an DTBNContainer.

3.1 Main Header (MH)

• Size: 1 byte

Primary entry point for any kind of parsing, Encodes the DTBNContainer setup which in turn describes everything else.

Bits	Field	Values
1	SUID Present	0 no SUID · 1 SUID present
2	Data Entry Count Format	00 0 entries · 01 single Entry · 10 16-bit Entry Count · 11 64-bit Entry Count
1	Meta Entries Present	0 no Meta Entry · 1 Meta entries present
1	Meta Entry Format	0 FDH · 1 EDH
1	Data Entry Format	0 FDH · 1 EDH
1	Data Entry Locality	0 local · 1 remote
1	Remote Link Format	0 FDH · 1 EDH

3.2 Main Header Extras (MHE)

• Size: 0, 2, 8, 10, or 16 bytes

Encodes optional Fields as per the Main Header.

Field	Present when
64-bit SUID	SUID Present = 1
16-bit or 64-bit Entry Count	DECF = 10 or 11 respectively

3.3 Entry Descriptor Header (EDH)

• Size: 3, 5, 7, … 513 bytes

Encodes a variable-Field Entry. This is used on SchemaTypes and EntryTypes (See §7)

Field	Notes
8-bit Field Count	Encoded as Effective Field Count − 1 → effective range 1–256 fields
16-bit Field TypeID	Repeated Effective Field Count times

3.4 Field Descriptor Header (FDH)

• Size: 2 bytes

Encodes a Single Field Entry. As a Variant of EDH, entry count still exists logically but is implied and not stored (See §7.4 SchemaTypes)

Field
16-bit Field TypeID

3.5 Remote Link Entry

Described using a FDH or EDH exactly like Data Entries. Its shape is selected by the Remote Link Format bit in the Main Header. It is only present when Data Entry Locality = 1. Full behavior is covered in §4.1 Data Entry Locality.

3.6 Don't-care bits

• Meta Format must be zero when Meta Present = 0.
• Remote Format must be zero when Entry Locality = 0.

3.7 DTBNContainer Field Order

With MH, MHE, FDH, EDH, and Remote Link Entry all now defined, here is the order in which they appear in an DTBNContainer, and how the Payload is laid out after that.:

1. MH goes first.
2. MHE fields are present as per MH flags.
3. If Meta Present, the corresponding FDH/EDH is present as per Meta Format.
4. The corresponding FDH/EDH for data entries is present as per Data Entry Format.
5. If Meta Present, the Meta Entry is present.
6. If Data Entry Locality is 0, data entries are present.
7. If Data Entry Locality is 1, a FDH/EDH is present as per the Remote Link Format flag, containing the link data into the data entries.

3.8 DTBNContainer Size

Excluding Meta or Data Entries, and excluding Remote Link payload: 3 to 1556 bytes.

Derivation:

• Minimum (3 B): MH (1 B) + MHE (0 B, no SUID/count) + Data FDH (2 B) = 3 B.
• Maximum (1556 B): MH (1 B) + MHE (16 B max: 8 B SUID + 8 B 64-bit Entry Count) + Meta EDH (513 B max) + Data EDH (513 B max) + Remote Link EDH (513 B max) = 1 + 16 + 513 + 513 + 513 = 1556 B.

---

4. Format Rules

4.1 Data Entry Locality

• Local (0): entries are stored inline with the DTBNContainer
• Remote (1): The Remote Link Entry contains the information necessary to index and locate the Remote Data Entries.

4.2 Undetermined Entry Count DTBNContainers

A Data Entry Count Format (DECF) other than 00/01, combined with Entry Count = 0, signals an undetermined number of entries. A simple example of a termination mechanism: a FDH DTBNContainer of DynamicTypes, with the last element marked using TypeID End. Nonetheless, Termination Mechanisms are implementation defined

• The Entry Count Format used (16 or 64 bit entry counts) represents the maximum number of entries in the undetermined Entry Count collection.

SideNote: The Canonical rules governing termination are covered in §9.5.

4.3 EDHs/FDHs, Entries and DTBN Determinism

no matter where an EDH or FDH is declared, two identical EDHs must end up with the same Entry layout, in the case of multiple Entry collections (DTBNContainers generally), same number of entries and same setup/ordering must result in the exact layout for all the respective Entry collections.

4.4 Validation and Error Handling

Consistent with DTBN's role as a substrate (§1.1), error handling and validation is implementation-defined.

There's no Canonical error-handling procedure to define, because Canonical guidelines exist to foster interoperability between well-formed DTBNContainers, a malformed DTBNContainer has already left that domain. That said, the Meta section is well-suited to carrying things like checksums, integrity hashes, or error-correction data that implementations can use to detect or flag parsing errors on their own.

Scenarios like entry overcounts will spill into unrelated memory unless out-of-bounds reads are restricted or some other safeguard is in place.

Another example is malformed UTF-8 strings: DTBN doesn't care, it will fetch the data as declared in the size, and whatever system is consuming that data has to deal with the malformation.

---

5. Types

5.1 Overview

Types are identified by a TypeID. There is room for 65536 distinct data types. The first 4096 TypeIDs (0000–0FFF) are reserved for built-in types. TypeIDs 1000–FFFF are available for implementation-defined types.

TypeIDs exist to describe shape (how to parse the bytes) and light semantic hinting (a Key isn't a Counter), not to enumerate every domain concept; most application-level meaning is expected to be elsewhere in the implementation. if you really need millions of types you may want to go for more dynamic routes like EntryTypes.

5.2 Sizedness

The Main Characteristic of any type.

Sizedness	Description
Static	Fixed size, defined by this standard or by the implementation.
Variable	Each type has a Size Prefix (indicated by the type) giving its size in bytes.

Types only have one sizedness variant. The Built-in Data Types table (§8) labels each block's sizedness class as part of its Block Name (e.g. "Numeric – Static 0").

5.3 Type Categories and Blocks

As a convenience the built-in TypeID space (0000–0FFF) is organized into 1024 blocks of 4 consecutive TypeIDs. Every block belongs to a single category and sizedness class. categories are completely convenience driven and irrelevant to types themselves but are meant to be an useful classification label for many types at a time. See §11.2

5.4 Normal Types

The following categories are meant to have their type's values behave like their name suggests, acting as data containers without any special rules.

• Boolean
• Numeric
• Blob
• Text
• Time
• Hash
• Cryptography
• Network
• Graphic

Many types in these categories have the same serialized shape or sizes. The distinction is purely semantic, a Key is not the same as a Counter, and both are different from an Integer.

5.5 Normal Type Formats

• Bool behavior is implementation-defined, but any implementation must define True and False states.
• BoolField operates as a bundle of 8-bit flags. Indexing into those 8 bits is implementation defined.
• Numeric types follow two's complement.
• Floats have standard layouts; Float8 format is implementation-defined.
• Variable-length types are straightforward: the size prefix is immediately followed by the payload.
• IPv4 and IPv6 are always Big Endian, for compatibility with already-existing conventions.
• Port is just a 16 bit unsigned integer.
• Hash and Cryptography-category types are implementation-defined: the point of giving them their own TypeIDs isn't to mandate an algorithm, it's so a reader knows roughly what a value is, a hash or a key, and of what size without decoding its contents.
• The RGB Type Family (RGB, RGBA, RGB16, RGBA16) stores its channels in the order red, green, blue, alpha (when present), plain RGB/RGB16 have no alpha channel.
• UnixTimeStamp32 and UnixTimeStamp64 are signed integer counts of seconds elapsed since the Unix epoch (1970-01-01T00:00:00 UTC).
• UUIDv4 Follows Standard Format
• URIs follow Standard Format
• Size, Count and Key Type Families are unsigned integers.
• ISO8601 follows standard text format.
• MIMEType follows standard text format.

---

6. Special Types - Trivial

These have no meaningful binary sub-structure beyond their raw value; what matters is their semantics.

6.1 SUID

As the name implies (Semantically Unknown Identifier), this value is specifically non-significant or non-meaningful. How the number is used is entirely implementation-defined.

• SUIDs can be used, for example, as a reference, id, randomly assigned number, bit field, etc. as long as it identifies the DTBNContainer somehow.
• SUIDs don't have to be unique per DTBNContainer.

6.2 Null/None

For when you need something, but it has to be nothing.

6.3 Start & End

Convenient markers.

6.4 TypeID

Type for storing a TypeID.

6.5 Reference Types ByteRef, PageRef, ByteOffset, PageOffset

• ByteOffset and PageOffset are signed, using two's complement representation.
• ByteRef and PageRef are unsigned, and as the name implies the offset is measured from the start of the address space.
• For PageRef and PageOffset, what a "Page" means is not defined, a Page can be a block, an in-memory page, or any other logical or physical storage unit of any size.

6.6 Reference Types NestedOffset8, NestedOffset16, NestedOffset32, NestedOffset64

Meant for internal reference inside nested DTBNContainers. Byte-addressed and unsigned. The base Address is at the start of the root DTBNContainer within any arbitrary but local nesting; that is, these references only work for local data, and must never be used for remote referencing.

7. Special Types - Structural

These types have special behaviours and are a large part of the flexibility of DTBN.

7.1 DTBNContainer

The Format itself as a type, allows recursive nesting of DTBNContainers. Its implicitly sized. DTBNContainers also work as the official entry point for DTBN system parsing. any kind of arbitrary parsing of an DTBN system must start at a Main header by necessity.

7.2 SizeFixedType

• Size: 5–12 B

Takes a dynamically-sized type and produces a new TypeID, semantically equal to the original but "size-fixed," i.e. no longer requiring a size prefix.

• Statically-sized TypeIDs can't be size-fixed.
• TypeIDs can't be recursively size-fixed (doesn't make sense and wastes TypeIDs).

Field
16-bit Original TypeID
16-bit New TypeID
8–64-bit Fixed size in bytes (width depends on the original type's size prefix)

7.3 DynamicType

• Size: 2 – >2⁶⁴+9 B

Stores any value of any type; the stored value is simply prefixed with its TypeID.

7.4 SchemaType

• Size: 5–515 B

A reusable EDH, used to group multiple types into a new logical type. A complete Entry layout can be referenced by a single TypeID inside an FDH or EDH.

• Only previously-defined types may be used inside a SchemaType. No recursive declarations allowed.
• SchemaTypes have a hard limit of 256 fields.
• SchemaTypes declared inside EDHs (other schemas, EntryTypes, etc) count as 1 field regardless of the internal amount of fields in the schema
• A SchemaType has no default serialized size ranges. The serialized size of an instance is the sum of the sizes of its expanded fields.

Field
16-bit SchemaType TypeID
8-bit Field Count (encoded as Effective Field Count − 1; range 1–256)
16-bit Field type, repeated Effective Field Count times

When Instancing a Schema in an Entry we simply flatten them into a sequence of fields on the Entry. For Entries themselves, Schemas are invisible.

7.5 EntryType

• Size: 3 – >2⁶⁴ B

Consists of an EDH, followed by a single Entry instance.

• as schemas, when described inside EDHs it counts as a single field regardless of the amount of internal fields.

7.6 UnknownType

• Size: 3-4-5-7-11 B

Encodes structural information about implementation-defined or external unknown types, so they can be parsed even if their structure or usage is unknown to other, arbitrary implementations. Only structural information is related to parsing the type is described, semantic meaning is implementation-defined.

Field
16-bit TypeID
1-bit sizedness (0 static, 1 variable)
3-bit size prefix (see table below)
4-bit reserved (must be zero)
16–64-bit size in bytes (present only if static sizedness)

Size prefix options:

Flag	Prefix
000	Implicit/Zero Sized
001	Size8
010	Size16
011	Size32
100	Size64

• Implicit Size Prefix: no prefix at all is used when instancing the type, structure itself encodes size (e.g. DTBNContainers, EntryTypes). If declared sizedness is static it means the Type is zero sized.

• Remaining flags are reserved.

The five sizes in the header above correspond to sizedness/prefix combinations: 3 B with no size field (Implied), 4 B with a Size8 field, 5 B with a Size16 field, 7 B with a Size32 field, 11 B with a Size64 field.

7.7 EnumType

Defines a closed set of variants; an instance selects exactly one variant and stores a value of that variant's type. Only previously-defined EnumTypes may be used elsewhere. Same restriction as SchemaTypes (no recursive/forward declarations).

Definition structure - Size: 5–515 B

Field
16-bit EnumType TypeID
8-bit Variant Count (encoded as Variant Count − 1; range 1–256)
16-bit Variant Type, repeated Variant Count times. One TypeID per variant

Instance structure - Size: 1 – >2⁶⁴ B

Field
8-bit Variant Index
Variant Value

---

8 Built-in Data Types Table

Types are organized in blocks and Categories. See §11.2.

Name	Size	TypeID
Intrinsic – Dynamic 0
DTBNContainer	3 – >2⁶⁴ B	0000
Identifier – Static 0
SUID	8 B	0004
TypeID	2 B	0005
UUIDv4	16 B	0006
Structural – Dynamic 0
UnknownType	3–11 B	0008
SchemaType	5–515 B	0009
EnumType	5–515 B	000A
DynamicType	2 – 2⁶⁴+9 B	000B
EntryType	3 – >2⁶⁴ B	000C
SizeFixedType	5–12 B	000D
Control – Static 0
Null/None	0 B	0010
Start	0 B	0011
End	0 B	0012
Reference – Static 0
ByteRef8	1 B	0014
ByteRef16	2 B	0015
ByteRef32	4 B	0016
ByteRef64	8 B	0017
PageRef8	1 B	0018
PageRef16	2 B	0019
PageRef32	4 B	001A
PageRef64	8 B	001B
ByteOffset8	1 B	001C
ByteOffset16	2 B	001D
ByteOffset32	4 B	001E
ByteOffset64	8 B	001F
PageOffset8	1 B	0020
PageOffset16	2 B	0021
PageOffset32	4 B	0022
PageOffset64	8 B	0023
NestedOffset8	1 B	0024
NestedOffset16	2 B	0025
NestedOffset32	4 B	0026
NestedOffset64	8 B	0027
Boolean – Static 0
Bool	1 B	0028
BoolField	1 B	0029
Numeric – Static 0
Uint8	1 B	002C
Uint16	2 B	002D
Uint32	4 B	002E
Uint64	8 B	002F
Int8	1 B	0030
Int16	2 B	0031
Int32	4 B	0032
Int64	8 B	0033
Size8	1 B	0034
Size16	2 B	0035
Size32	4 B	0036
Size64	8 B	0037
Count8	1 B	0038
Count16	2 B	0039
Count32	4 B	003A
Count64	8 B	003B
Key8	1 B	003C
Key16	2 B	003D
Key32	4 B	003E
Key64	8 B	003F
Float8	1 B	0040
Float16	2 B	0041
Float32	4 B	0042
Float64	8 B	0043
Blob – Dynamic 0
ByteArray8	1–256 B	0044
ByteArray16	2–65537 B	0045
ByteArray32	4–2³²+3 B	0046
ByteArray64	8 – 2⁶⁴+7 B	0047
Text – Dynamic 0
UTF-8-8	1–256 B	0048
UTF-8-16	2–65537 B	0049
UTF-8-32	4–2³²+3 B	004A
UTF-8-64	8 – 2⁶⁴+7 B	004B
Time – Static 0
UnixTimeStamp32	4 B	004C
UnixTimeStamp64	8 B	004D
Hash – Static 0
Hash128	16 B	0050
Hash256	32 B	0051
Hash512	64 B	0052
Cryptography – Static 0
CryptoKey128	16 B	0054
CryptoKey192	24 B	0055
CryptoKey256	32 B	0056
CryptoKey2048	256 B	0058
CryptoKey3072	384 B	0059
CryptoKey4096	512 B	005A
Network – Static 0
IPv4	4 B	005C
IPv6	16 B	005D
Port	2 B	005E
Reference – Dynamic 0
URI8	1–256 B	0060
URI16	2–65537 B	0061
URI32	4–2³²+3 B	0062
URI64	8 – 2⁶⁴+7 B	0063
Graphic – Static 0
RGB	3 B	0064
RGBA	4 B	0065
RGB16	6 B	0066
RGBA16	8 B	0067
Time – Dynamic 0
ISO8601	1-256 B	0068
Identifier – Dynamic 0
MIMEType	1-256 B	006C

SideNote: Why so many types that serialize to the same underlying thing exist? implementations can represent a Key8 as a Uint8 and be done with it since they know the data type they are expecting. However this completely ignores fully runtime based DTBN implementations that will benefit of encoding the same data shapes into different types for semantic decoding. Also many types have many size variants for scalability of avaiable space for DTBN systems.

---

9. Canonical Recommendations [CANONICAL]

DTBN is meant to be implementation-driven first; this section exists to encourage interoperability for those who care about interacting with foreign systems. Nothing in this section is mandatory, required, or obligatory unless specified otherwise.

9.1 Canonical Entry Point

A Canonical DTBN system has an Entry-point DTBNContainer with the following properties:

• No SUID (or SUID 0).

• Data Entries use the EDH format with fields ByteArray8, DTBNContainer i.e. each entry is (tag, nested DTBNContainer) pair.

• If implementation-defined types of any kind are used, a data Entry must have its first field as bytes "GTR" (Global Type Registry); the second field must be a single EDH DTBNContainer, single entry, local, with 4 DTBNContainer FDH local fields. See Example 19 for a fully worked, hollow GTR structure.

• Each FDH will have UnknownType, SchemaType, EnumType, SizeFixedType as the entry type respectively.

• All non-built-in types must be described in the GTR.

• Optionally, extra entries on the top level DTBNContainer may be added, e.g. "NAME", "AUTHORS", with their contents implementation-defined.

• Meta entries on the Entry-point DTBNContainer and nested structures are fully implementation-defined.

The single-GTR design above is the Canonical answer to enable full interoperable type interpretation. (See also §11.3)

9.2 Non DTBN Formatted data on DTBN Systems

Implementations are not limited to exclusively use DTBN's or related structures at any point in their Systems. But a canonical System ensures every piece of data is described or at least reachable by parseable DTBN formatted structures.

Canonically, even if the remote data entries' access strategy is unknown, the information in the Remote Link Entry should lead to more DTBN Formatted structures that will eventually lead the parser to the entries described, even if by brute-force search.

9.3 SchemaTypes, EntryTypes, and EDHs: How they relate

• Descriptors are essential to DTBN, defining the format of entries, every DTBNContainer carries at least one data FDH or EDH.
• Although similar at first glance, SchemaTypes let you create and save a custom EDH, then set it as the TypeID on an DTBNContainer EDH or FDH, saving you from writing a possibly-large list of TypeIDs over and over across many DTBNContainers.
• EntryTypes are useful when your data doesn't share the same format across entries: an EntryType instance is an EDH followed by a single Entry in the prescribed format. FDH DTBNContainers with EntryType as the type can be completely heterogeneous entry-to-entry.
• many types allow for "brainless nesting". 256 fields in an Entry is plenty for many use cases; splitting large Entries across multiple DTBNContainers is another option instead of nesting values within values.

9.4 Canonical Nesting

Local Value nesting can't exceed 3 levels (excluding root level).

9.5 DTBNContainer Termination

• Canonically, termination of an undetermined Entry Count collection (see §4.2 for what makes a collection undetermined) must include a termination mechanism; the specific mechanism used is implementation-defined.
• If using Enums, Dynamic Types, or Entry Types on entries, End type can be considered termination, but must be on the last entry, on the last field, even if nested. to be canonically considered the end of the sequence.
• For usual cases where the amount of entries is determined, Canonical termination is defined by reading the amount of entries stated in Entry Count.

9.6 Extra Canonical Format Rules

• Alignment and padding: Canonically, parsing and decoding of DTBNContainers and their payloads is intended to be completely sequential, there is no cannonical padding or alignment. Implementations may use their own alignment, padding, and ordering of non-bit fields, e.g. using natural field-size alignment for zero-copy techniques.

• Bit/byte ordering: Canonically, bit fields for flags are ordered MSB to LSB; multi-byte fields are Little Endian. Implementations may choose different endianness, but must respect bit-field ordering.

---

10 DTBN implementation floor

• DTBN implementations dont have to agree on many things; padding, alignment, custom defined TypeIDs, implementations dont even need to implement all noncanonical features. However what EVERY implementation must converge on is the final binary language on the resulting DTBN objects:
• Every builtin type implemented is serialized as per the spec
• Every feature is implemented as per the spec.
• Partial feature implementations are completely valid as long as they produce valid DTBN structures.

In summary, the official binary format stored or transmitted must be compliant. everything else is fair game.

11. Examples

Big Endian is used throughout for readability. Every example is a byte-offset table (Offset | Field | Size | Value | Hex) followed by a linear hex block for copy-paste use. Where a field's value is itself a byte offset into the same structure (e.g. a NestedOffset), the row it points to is placed immediately below it, so the two can be checked against each other by eye.

Example 1: Simplest DTBNContainer

A single Null-typed field, no Meta, no SUID, no MHE at all, the smallest legal DTBNContainer.

Offset	Field	Size	Value	Hex
0	MH	1 B	SUID=0, DECF=00, Meta=0, DataFmt=FDH, Locality=local	00
1	Data FDH	2 B	Null/None	0010

Total: 3 B

00 0010

Example 2: Collection of user records with SUID

A dataset of five (username, login count) pairs, tagged with a SUID and a 16-bit Entry Count.

Header

Offset	Field	Size	Value	Hex
0	MH	1 B	SUID=1, DECF=10, Meta=0, DataFmt=EDH, Locality=local	C4
1	MHE: SUID	8 B	1235	00000000000004D3
9	MHE: Entry Count	2 B	5	0005
11	Data EDH: Field Count	1 B	Effective=2	01
12	Data EDH: Field Type[0]	2 B	UTF-8-16	0049
14	Data EDH: Field Type[1]	2 B	Count32	003A

Header total: 16 B

Data Entries (username, login count)

#	username	login count	Hex
1	"alice92"	142	0007 616C6963653932 0000008E
2	"bmiller"	7	0007 626D696C6C6572 00000007
3	"quinn_dev"	3021	0009 7175696E6E5F646576 00000BCD
4	"kenji"	58	0005 6B656E6A69 0000003A
5	"sora.tanaka"	415	000B 736F72612E74616E616B61 0000019F

C4 00000000000004D3 0005 01 0049 003A
0007 616C6963653932 0000008E
0007 626D696C6C6572 00000007
0009 7175696E6E5F646576 00000BCD
0005 6B656E6A69 0000003A
000B 736F72612E74616E616B61 0000019F

Example 3: DTBNContainer with FDH Meta, EDH Data, remote data, EDH link

A media catalog whose items live remotely. Entry Count (3,200,000) is how many items this catalog currently lists; the Meta value is a separate, independent counter of the total number of items ever ingested into the catalog over its lifetime (only ever grows, even if items are later removed from the live count). The Data EDH describes the shape each catalog item would have (name, revision, pointer), but because Data Entry Locality = 1, no actual item data is stored locally; only the Remote Link is. The Remote Link EDH's two SUID fields are opaque locator values (e.g. a shard identifier and an object identifier within that shard) that an implementation resolves, by its own means, into the real remote bytes.

Header

Offset	Field	Size	Value	Hex
0	MH	1 B	SUID=0, DECF=11, Meta=1, MetaFmt=FDH, DataFmt=EDH, Locality=remote, RemoteFmt=EDH	77
1	MHE: Entry Count	8 B	3200000	000000000030D400
9	Meta FDH	2 B	Count64	003B
11	Data EDH: Field Count	1 B	Effective=3	02
12	Data EDH: Field Type[0]	2 B	UTF-8-16	0049
14	Data EDH: Field Type[1]	2 B	Count32	003A
16	Data EDH: Field Type[2]	2 B	ByteRef64	0017

Header total: 18 B

Meta Entry (lifetime ingestion counter. Shape given by the Meta FDH, Count64)

Offset	Field	Size	Value	Hex
18	Meta Entry	8 B	82400000000000	00004AF13EBAC000

Remote Link EDH (shape given by the Remote Link EDH; Data Entry Locality = 1, so no local data entries follow the header)

Offset	Field	Size	Value	Hex
26	Field Count	1 B	Effective=2	01
27	Field Type[0]	2 B	SUID	0004
29	Field Type[1]	2 B	SUID	0004

Remote Link Entry

Offset	Field	Size	Value	Hex
31	Entry: SUID[0] (shard)	8 B	65536	0000000000010000
39	Entry: SUID[1] (object)	8 B	4096	0000000000001000

Total: 47 B

77 000000000030D400 003B 02 0049 003A 0017
00004AF13EBAC000
01 0004 0004
0000000000010000 0000000000001000

Example 4: Simple SchemaType DTBNContainer

SchemaType defined as Size64, UTF-8-16, Count16, a file record of (file size, filename, format version). Size64 fits "size in bytes" exactly. The filename is text, so UTF-8-16 (not ByteArray16) is the semantically correct choice. "version" is an incrementing count, so it's Count16, not Size16. Schema TypeID 1AF0.

Header

Offset	Field	Size	Value	Hex
0	MH	1 B	SUID=0, DECF=10, Meta=0, DataFmt=FDH, Locality=local	40
1	MHE: Entry Count	2 B	3	0003
3	Data FDH	2 B	SchemaType 1AF0	1AF0

Header total: 5 B

Data Entries (file size, filename, version). SchemaType 1AF0 instance per row

#	size (B)	filename	version	Hex
1	240896	"report.pdf"	2	000000000003AD00 000A 7265706F72742E706466 0002
2	1310	"notes.txt"	1	000000000000051E 0009 6E6F7465732E747874 0001
3	4521984	"photo.png"	5	0000000000450000 0009 70686F746F2E706E67 0005

40 0003 1AF0
000000000003AD00 000A 7265706F72742E706466 0002
000000000000051E 0009 6E6F7465732E747874 0001
0000000000450000 0009 70686F746F2E706E67 0005

Example 5: Simple DTBNContainer with EntryTypes

EntryType fields, each carrying a different number of Uint8 values, heterogeneous per-entry shape.

Header

Offset	Field	Size	Value	Hex
0	MH	1 B	SUID=0, DECF=10, Meta=0, DataFmt=FDH, Locality=local	40
1	MHE: Entry Count	2 B	3	0003
3	Data FDH	2 B	EntryType	000C

Header total: 5 B

Data Entries (each is its own EntryType instance: Field Count byte, that many Uint8 TypeIDs, then that many values)

#	Field Count byte	Fields	Values	Hex
1	00 (Effective=1)	Uint8	72	00 002C 48
2	01 (Effective=2)	Uint8, Uint8	18, 95	01 002C 002C 12 5F
3	03 (Effective=4)	Uint8×4	3, 54, 201, 89	03 002C 002C 002C 002C 03 36 C9 59

40 0003 000C
00 002C 48
01 002C 002C 12 5F
03 002C 002C 002C 002C 03 36 C9 59

Example 6: Single Entry Format (DECF = 01)

Shows that DECF=01 needs no Entry-Count bytes at all (Count is implicitly 1). Here, a single Port number.

Offset	Field	Size	Value	Hex
0	MH	1 B	SUID=0, DECF=01, Meta=0, DataFmt=FDH, Locality=local	20
1	Data FDH	2 B	Port	005E
3	Entry: port	2 B	8080	1F90

Total: 5 B

20 005E 1F90

Example 7: DynamicType

Each Entry self-describes its own type via a TypeID prefix to allow heterogeneous entries. Here we have 3 entries: a status byte, a port number, and a short tag.

Header

Offset	Field	Size	Value	Hex
0	MH	1 B	SUID=0, DECF=10, Meta=0, DataFmt=FDH, Locality=local	40
1	MHE: Entry Count	2 B	3	0003
3	Data FDH	2 B	DynamicType	000B

Header total: 5 B

Data Entries (each self-prefixed with its own TypeID)

#	Type	Value	Hex
1	Uint8	200	002C C8
2	Port	8080	005E 1F90
3	UTF-8-16	"api"	0049 0003 617069

40 0003 000B
002C C8
005E 1F90
0049 0003 617069

Example 8: Undetermined-length stream, terminated by End

DECF other than 00/01 with Entry Count = 0 signals an open-ended stream; the last DynamicType Entry is an End marker instead of a value.

Header

Offset	Field	Size	Value	Hex
0	MH	1 B	SUID=0, DECF=11, Meta=0, DataFmt=FDH, Locality=local	60
1	MHE: Entry Count	8 B	0 (undetermined)	0000000000000000
9	Data FDH	2 B	DynamicType	000B

Header total: 11 B

Data Entries

#	Type	Value	Hex
1	Uint8	3	002C 03
2	Uint8	7	002C 07
3	End (terminator)	N/A	0012

60 0000000000000000 000B
002C 03
002C 07
0012

Example 9: Minimal SchemaType

Same mechanic as Example 4 but with two plain static fields instead of a mixed variable-length Entry. Easier to trace the offset with no length prefixes involved. SchemaType defined as Port, Count8, a (port, retry count) pair. Schema TypeID 1AF1.

Offset	Field	Size	Value	Hex
0	MH	1 B	SUID=0, DECF=01, Meta=0, DataFmt=FDH, Locality=local	20
1	Data FDH	2 B	SchemaType 1AF1	1AF1
3	Entry: Port	2 B	443	01BB
5	Entry: Count8 (retries)	1 B	3	03

Total: 6 B

20 1AF1 01BB 03

Example 10: Remote Locality with FDH Link

Example 3 already covers remote locality, but only with an EDH link; this shows the simpler FDH remote-link case (single-field link, no field Count byte): a byte offset into remote storage.

Offset	Field	Size	Value	Hex
0	MH	1 B	SUID=0, DECF=01, Meta=0, DataFmt=FDH, Locality=remote, RemoteFmt=FDH	22
1	Data FDH	2 B	Uint32	002E
3	Remote Link FDH	2 B	ByteRef64	0017
5	Remote Link Entry	8 B	65536	0000000000010000

Total: 13 B

22 002E 0017 0000000000010000

Example 11: Nested SchemaTypes

A SchemaType containing another SchemaType as one of its fields.

• SchemaType A (type 1AF2): Uint8, Uint8 - 2 fields
• SchemaType B (type 1AF3): Uint16, SchemaType A (1AF2) - 2 fields

Offset	Field	Size	Value	Hex
0	MH	1 B	SUID=0, DECF=01, Meta=0, DataFmt=FDH, Locality=local	20
1	Data FDH	2 B	SchemaType 1AF3	1AF3
3	Entry: Uint16	2 B	1440	05A0
5	Entry: SchemaType A field[0] (Uint8)	1 B	30	1E
6	Entry: SchemaType A field[1] (Uint8)	1 B	12	0C

Total: 7 B

20 1AF3 05A0 1E 0C

Example 12: SUID Present, with actual SUID bytes

Simplest possible structure, just to show the 8-byte SUID field populated.

Offset	Field	Size	Value	Hex
0	MH	1 B	SUID=1, DECF=00, Meta=0, DataFmt=FDH, Locality=local	80
1	MHE: SUID	8 B	20260805	00000000013527C5
9	Data FDH	2 B	Null/None	0010

Total: 11 B

80 00000000013527C5 0010

Example 13: UnknownType descriptor

Static-sizedness case: implementation-defined type 0x1234, size given by a 16-bit size field (prefix flag 010), value = 16 bytes.

Offset	Field	Size	Value	Hex
0	MH	1 B	SUID=0, DECF=01, Meta=0, DataFmt=FDH, Locality=local	20
1	Data FDH	2 B	UnknownType	0008
3	Entry: TypeID	2 B	0x1234	1234
5	Entry: flags (sizedness+prefix+reserved)	1 B	static, prefix=Size16	20
6	Entry: size (Size16 field)	2 B	16	0010

Total: 8 B

20 0008 1234 20 0010

Example 14: SchemaType definition

Registers a new SchemaType 1AF4 as (Uint16, Uint8), this is the definition itself, distinct from an instance of it.

Offset	Field	Size	Value	Hex
0	MH	1 B	SUID=0, DECF=01, Meta=0, DataFmt=FDH, Locality=local	20
1	Data FDH	2 B	SchemaType	0009
3	Entry: New TypeID	2 B	1AF4	1AF4
5	Entry: Field Count	1 B	Effective=2	01
6	Entry: Field Type[0]	2 B	Uint16	002D
8	Entry: Field Type[1]	2 B	Uint8	002C

Total: 10 B

20 0009 1AF4 01 002D 002C

Example 15: EntryType containing a SchemaType

An EntryType instance with 2 fields: a plain Uint8, then SchemaType 1AF1 (reused from Example 9: Port, Count8, a (port, retry count) pair), which expands to 2 more values.

Offset	Field	Size	Value	Hex
0	MH	1 B	SUID=0, DECF=01, Meta=0, DataFmt=FDH, Locality=local	20
1	FDH	2 B	EntryType	000C
3	EntryType Field Count	1 B	2	01
4	EntryType: Field Type[0]	2 B	Uint8	002C
6	EntryType: Field Type[1]	2 B	SchemaType 1AF1	1AF1
8	Entry: Uint8 value	1 B	5	05
9	Entry: SchemaType 1AF1 value (port, retries)	3 B	(443, 3)	01BB 03

Total: 12 B

20 000C 01 002C 1AF1 05 01BB 03

Example 16: DTBNContainer inside another DTBNContainer

The outer DTBNContainer's single Entry is a fully self-contained inner DTBNContainer (reusing Example 1's simplest form: Null type, 0 entries).

Offset	Field	Size	Value	Hex
0	MH	1 B	SUID=0, DECF=01, Meta=0, DataFmt=FDH, Locality=local	20
1	Data FDH	2 B	DTBNContainer	0000
3	Entry: nested MH	1 B	(Example 1)	00
4	Entry: nested Data FDH	2 B	Null/None	0010

Total: 6 B

20 0000 00 0010

Example 17: Nested DTBNContainer referenced via NestedOffset16

Two fields: a NestedOffset16 giving the byte offset (from root start) to where the nested DTBNContainer begins, and the nested DTBNContainer itself (Example 1's bytes again) placed right after.

Offset	Field	Size	Value	Hex
0	MH	1 B	SUID=0, DECF=01, Meta=0, DataFmt=EDH, Locality=local	24
1	Data EDH: Field Count	1 B	Effective=2	01
2	Data EDH: Field Type[0]	2 B	NestedOffset16	0025
4	Data EDH: Field Type[1]	2 B	DTBNContainer	0000
6	Entry: NestedOffset16	2 B	8 → points at offset 8, the row below	0008
8	Entry: nested MH	1 B	(Example 1)	00
9	Entry: nested Data FDH	2 B	Null/None	0010

Total: 11 B

24 01 0025 0000 0008 00 0010

Example 18: UnknownType, dynamically sized

Offset	Field	Size	Value	Hex
0	MH	1 B	SUID=0, DECF=01, Meta=0, DataFmt=FDH, Locality=local	20
1	Data FDH	2 B	UnknownType	0008
3	Entry: TypeID	2 B	0x5678	5678
5	Entry: flags (sizedness+prefix+reserved)	1 B	variable, prefix=Size32	B0

Total: 6 B

20 0008 5678 B0

Example 19: Minimal, Empty Entry-Point DTBNContainer with GTR

No metadata, all data local, single-Entry format used wherever the field allows it. Demonstrates §9.1's Canonical Entry Point: a top-level DTBNContainer whose one data entry is "GTR", DTBNContainer, where the nested GTR-DTBNContainer holds four hollow (empty) registries, one each for UnknownType, SchemaType, EnumType, SizeFixedType, in that order, as §9.1 requires.

GTR-DTBNContainer (nested structure)

Offset	Field	Size	Value	Hex
0	MH	1 B	SUID=0, DECF=01, Meta=0, DataFmt=EDH, Locality=local	24
1	Data EDH: Field Count	1 B	Effective=4	03
2	Data EDH: Field Type[0]	2 B	DTBNContainer	0000
4	Data EDH: Field Type[1]	2 B	DTBNContainer	0000
6	Data EDH: Field Type[2]	2 B	DTBNContainer	0000
8	Data EDH: Field Type[3]	2 B	DTBNContainer	0000
10	Entry[0]: nested MH (UnknownType registry)	1 B	hollow	00
11	Entry[0]: nested Data FDH	2 B	UnknownType	0008
13	Entry[1]: nested MH (SchemaType registry)	1 B	hollow	00
14	Entry[1]: nested Data FDH	2 B	SchemaType	0009
16	Entry[2]: nested MH (EnumType registry)	1 B	hollow	00
17	Entry[2]: nested Data FDH	2 B	EnumType	000A
19	Entry[3]: nested MH (SizeFixedType registry)	1 B	hollow	00
20	Entry[3]: nested Data FDH	2 B	SizeFixedType	000D

GTR-DTBNContainer total: 22 B

24 03 0000 0000 0000 0000
00 0008 00 0009 00 000A 00 000D

Entry-point DTBNContainer (outer structure)

Offset	Field	Size	Value	Hex
0	MH	1 B	SUID=0, DECF=01, Meta=0, DataFmt=EDH, Locality=local	24
1	Data EDH: Field Count	1 B	Effective=2	01
2	Data EDH: Field Type[0]	2 B	ByteArray8	0044
4	Data EDH: Field Type[1]	2 B	DTBNContainer	0000
6	Entry: tag length	1 B	3	03
7	Entry: tag bytes	3 B	"GTR"	475452
10	Entry: nested DTBNContainer	22 B	GTR-DTBNContainer (see table above)	(22 B, shown above)

Total: 32 B

24 01 0044 0000
03 475452
24 03 0000 0000 0000 0000
00 0008 00 0009 00 000A 00 000D

---

11. Appendices

11.1 Design Notes

• The easiest way to achieve interoperability, even without sharing whole DTBNContainers or implementation details, is simply by agreeing on TypeIDs.
• Field Count on Entries is capped at 256 fields because it neatly fits inside a byte, while (hopefully) providing ample space within a single Entry.
• Ideally, data is always accompanied by an DTBNContainer explaining everything about it; however, implementations aren't prohibited from embedding opaque data. UnknownType is one example, but the biggest example is remote data entries: what a Remote Link Entry points to is a complete mystery, all that BTDN cares about Remote Link Entries is that they somehow lead to the Data Entries.
• Meta entries and Remote Link Entries are confined to local, single-Entry forms on purpose because their data is explicitly local to their DTBNContainer.

11.2 Built-in Type Allocation Strategy

(This section is intended for maintainers of the built-in type table.)

The built-in TypeID space (0000–0FFF) is organized into 1024 blocks of 4 consecutive TypeIDs. Every block belongs to a single category and sizedness class. Multiple adjacent Blocks may be grouped into "block runs" when a larger category exists but we don't want to split it into several separately-named groups of 4 TypeIDs. For example, grouping the numeric types, sorted by Type Family, into a single 6-block run. Blocks and block runs are named using the format: [Category] - [Sizedness] [Sequential number]

A new type is sorted, in order, before it is allocated in the TypeID space, by: semantic category → sizedness → Type Family or standalone.

• Category: broad semantic grouping (Numeric, Reference, Structural, etc).
• Sizedness: Static or Variable, uniform within a block.
• Type Family: types with a close relation to each other (e.g. Uint8/16/32/64). Standalone: types that don't belong to a Type Family (e.g. Bool, Null, SUID).

Allocation rules:

• Standalone types fill any spare TypeID in an existing block of matching category and sizedness; if none exists, a new block is allocated. (This is how Port was added to the existing Network – Static 0 block alongside IPv4/IPv6, for example, no new block needed.)
• Type Families are block-aligned: a Type Family occupies its blocks exclusively, and other types cannot use its spare TypeIDs even when the family doesn't fill every slot in its block(s), because the family might grow into them later.
• Exception: once a Type Family is explicitly marked complete (maintainers commit that it will never grow further), any spare TypeIDs still unused in its blocks are released, and may then be allocated to standalone types of a matching category/sizedness, the same as any other spare slot. Until a family is marked complete this way, its spare slots stay reserved and off-limits.
• A Type Family never expands into spare TypeIDs outside its own allocated blocks. If no free TypeID remains within the family's allocation, a new block is allocated.
• New categories may be added as needed.

Types are Permanent once assigned to the built-in type table, and their binary representations become Permanent too.

SideNote: this permanence is also so the live draft can avoid replacing TypeIDs every time a new type is added, and so a new type can be placed next to similar ones. Blocks and block runs are an editorial organization of the built-in TypeID registry only. They have no binary representation and carry no semantic meaning beyond guiding allocation and maintenance of built-in TypeIDs.

11.3 Handling Very Large Type Spaces, a suggested approach

The type space is inherently limited to 65536 (64 KiB) values. For a type-heavy system that outgrows that, one possible approach is to logically partition the type space, so every partition gets its own independent, unused type space at the cost of harder interoperability when crossing partitions. A simple refinement is to reserve a portion of the type space as global across partitions, akin to the built-in types.

This is a suggestion, not Canonical guidance. The Canonical approach to type-space scaling is the single Global Type Registry described in §9.1.