Last active
September 23, 2024 13:53
-
-
Save chaudum/df70d63c1da5537338a424040cfc141f to your computer and use it in GitHub Desktop.
Binary format of Loki bloom blocks
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Binary format | |
| ## Tar Archive | |
| +--------------------------------------------------------------------------------------------------------+ | |
| | Compression (optional) | | |
| | +----------------------------------------------------------------------------------------------------+ | | |
| | | Tar Archive | | | |
| | | +------------------------------------------------------------------------------------------------+ | | | |
| | | | `series` file | | | | |
| | | +------------------------------------------------------------------------------------------------+ | | | |
| | | | `blooms` file | | | | |
| | | +------------------------------------------------------------------------------------------------+ | | | |
| | +----------------------------------------------------------------------------------------------------+ | | |
| +--------------------------------------------------------------------------------------------------------+ | |
| ## `series` file: | |
| +--------------------------------------------------------------------------------------------------------+ | |
| | BlockIndex | | |
| | +----------------------------------------------------------------------------------------------------+ | | |
| | | BlockOptions | | | |
| | | +------------------------------------------------------------------------------------------------+ | | | |
| | | | Schema | | | | |
| | | | +--------------------------------------------------------------------------------------------+ | | | | |
| | | | | Magic Number (4 bytes) | Version (1 byte) | Encoding (1 byte) | | | | | |
| | | | +--------------------------------------------------------------------------------------------+ | | | | |
| | | +------------------------------------------------------------------------------------------------+ | | | |
| | | | SeriesPageSize (8 bytes) | BloomPageSize (8 bytes) | BlockSize (8 bytes) | | | | |
| | | +------------------------------------------------------------------------------------------------+ | | | |
| | +----------------------------------------------------------------------------------------------------+ | | |
| | | Headers | | | |
| | | +------------------------------------------------------------------------------------------------+ | | | |
| | | | Count (uvarint) | | | | |
| | | +------------------------------------------------------------------------------------------------+ | | | |
| | | | SeriesPageHeaderWithOffset (repeated) | | | | |
| | | | +--------------------------------------------------------------------------------------------+ | | | | |
| | | | | Offset (uvarint) | Len (uvarint) | DecomressedLen (uvarint) | | | | | |
| | | | +--------------------------------------------------------------------------------------------+ | | | | |
| | | | | SeriesHeader | | | | | |
| | | | | +----------------------------------------------------------------------------------------+ | | | | | |
| | | | | | NumSeries (uvarint) | Bounds (uvarint, uvarint) | From (varint) | Through (varint) | | | | | | |
| | | | | +----------------------------------------------------------------------------------------+ | | | | | |
| | | | +--------------------------------------------------------------------------------------------+ | | | | |
| | | +------------------------------------------------------------------------------------------------+ | | | |
| | | | ... | | | | |
| | | +------------------------------------------------------------------------------------------------+ | | | |
| | | | ... | | | | |
| | | +------------------------------------------------------------------------------------------------+ | | | |
| | +----------------------------------------------------------------------------------------------------+ | | |
| | | Pages | | | |
| | | +------------------------------------------------------------------------------------------------+ | | | |
| | | | SeriesWithMeta | | | | |
| | | | +--------------------------------------------------------------------------------------------+ | | | | |
| | | | | Fingerprint (uvarint, delta-encoded) | | | | | |
| | | | +--------------------------------------------------------------------------------------------+ | | | | |
| | | | | NumOffsets (uvarint) | Offset (repeated) | ... | | | | | |
| | | | +--------------------------------------------------------------------------------------------+ | | | | |
| | | | | NumChunks (uvarint) | Chunk (repeated) | ... | | | | | |
| | | | +--------------------------------------------------------------------------------------------+ | | | | |
| | | | | NumFields (uvarint) | Field (repeated) | ... | | | | | |
| | | | +--------------------------------------------------------------------------------------------+ | | | | |
| | | +------------------------------------------------------------------------------------------------+ | | | |
| | | | ... | | | | |
| | | +------------------------------------------------------------------------------------------------+ | | | |
| | +----------------------------------------------------------------------------------------------------+ | | |
| | | HeaderOffset (8 bytes) | Checksum (4 bytes) | | | |
| | +----------------------------------------------------------------------------------------------------+ | | |
| +--------------------------------------------------------------------------------------------------------+ | |
| ## `blooms` file: | |
| +--------------------------------------------------------------------------------------------------------+ | |
| | BloomBlock | | |
| | +----------------------------------------------------------------------------------------------------+ | | |
| | | Schema | | | |
| | | +------------------------------------------------------------------------------------------------+ | | | |
| | | | Magic Number (4 bytes) | Version (1 byte) | Encoding (1 byte) | | | | |
| | | +------------------------------------------------------------------------------------------------+ | | | |
| | +----------------------------------------------------------------------------------------------------+ | | |
| | | Headers | | | |
| | | +------------------------------------------------------------------------------------------------+ | | | |
| | | | Count (uvarint) | | | | |
| | | +------------------------------------------------------------------------------------------------+ | | | |
| | | | BloomPageHeader (repeated) | | | | |
| | | | +--------------------------------------------------------------------------------------------+ | | | | |
| | | | | N (uvarint) | Offset (uvarint) | Len (uvarint) | DecomressedLen (uvarint) | | | | | |
| | | | +--------------------------------------------------------------------------------------------+ | | | | |
| | | +------------------------------------------------------------------------------------------------+ | | | |
| | | | ... | | | | |
| | | +------------------------------------------------------------------------------------------------+ | | | |
| | | | ... | | | | |
| | | +------------------------------------------------------------------------------------------------+ | | | |
| | +----------------------------------------------------------------------------------------------------+ | | |
| | | Pages | | | |
| | | +------------------------------------------------------------------------------------------------+ | | | |
| | | | Bloom | | | | |
| | | | +--------------------------------------------------------------------------------------------+ | | | | |
| | | | | Len (uvarint) | | | | | |
| | | | +--------------------------------------------------------------------------------------------+ | | | | |
| | | | | ScalableBloomFilter | | | | | |
| | | | | +----------------------------------------------------------------------------------------+ | | | | | |
| | | | | | xxxx | | | | | | |
| | | | | +----------------------------------------------------------------------------------------+ | | | | | |
| | | | +--------------------------------------------------------------------------------------------+ | | | | |
| | | +------------------------------------------------------------------------------------------------+ | | | |
| | | | ... | | | | |
| | | +------------------------------------------------------------------------------------------------+ | | | |
| | | | ... | | | | |
| | | +------------------------------------------------------------------------------------------------+ | | | |
| | +----------------------------------------------------------------------------------------------------+ | | |
| | | HeaderOffset (8 bytes) | Checksum (4 bytes) | | | |
| | +----------------------------------------------------------------------------------------------------+ | | |
| +--------------------------------------------------------------------------------------------------------+ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Appendix
Magic Numberis not actually part of theSchemastruct, but is encoded/decoded as part of it.HeadersandPagesare not an actual structs, but are containers for better visualisation.