MIT License
Copyright (c) 2019 Bastian Blokland
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
This week spend some time rewriting our binary serialization to operate on Span<byte> instead of the
good old BinaryWriter. Below are some findings and utilities that might be useful to other people.
Advantages of using Span<byte> vs BinaryWriter.
Because we can make more assumptions (like always being backed by real-memory and not some other stream) we can get much higher raw throughput:
Note: These benchmarks are using the BinSerialize utilities that are implemented below.
BenchmarkDotNet=v0.11.5, OS=macOS Mojave 10.14.5 (18F203) [Darwin 18.6.0]
Intel Core i9-8950HK CPU 2.90GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=3.0.100-preview5-011568
[Host] : .NET Core 3.0.0-preview5-27626-15 (CoreCLR 4.6.27622.75, CoreFX 4.700.19.22408), 64bit RyuJIT
Job-NVXXXM : .NET Core 3.0.0-preview5-27626-15 (CoreCLR 4.6.27622.75, CoreFX 4.700.19.22408), 64bit RyuJIT| Method | Mean | Error | StdDev | Ratio | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|
| BinaryWriter_Write | 682.7 ms | 6.840 ms | 6.399 ms | 1.00 | - | - | - | - |
| BinSerialize_Write | 123.5 ms | 2.074 ms | 1.838 ms | 0.18 | - | - | - | - |
| Method | Mean | Error | StdDev | Ratio | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|
| BinaryReader_Read | 422.9 ms | 5.032 ms | 4.461 ms | 1.00 | - | - | - | - |
| BinSerialize_Read | 105.8 ms | 2.445 ms | 2.287 ms | 0.25 | - | - | - | - |
| Method | Mean | Error | StdDev | Ratio | RatioSD | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|---|
| BinaryWriter_WriteReadString | 246.7 ms | 4.013 ms | 3.754 ms | 1.00 | 0.00 | 734000.0000 | - | - | 114.44 MB |
| BinSerialize_WriteReadString | 208.4 ms | 3.985 ms | 3.914 ms | 0.84 | 0.02 | 734000.0000 | - | - | 114.44 MB |
Note: Strings benchmark is using the naive way of allocating new strings, if you write into a buffer you re-use (or even on the stack) you can get huge gains.
Note: Source of these benchmarks can be found at the end of this file.
Because Span is just a struct containing a offset and length into memory, it basically costs nothing
to 'slice off' a section of a Span. We can use that to create a quite elegant way 'chaining'
individual writes without needing a separate 'position' variable.
For example a write method like this:
public static void WriteInt(ref Span<byte> span, int val)
{
// Write 4 bytes.
MemoryMarshal.Write(span, ref val);
// 'Advance' the span by 4 bytes.
span = span.Slice(sizeof(int));
}Allows for chaining of multiple writes:
byte[] buffer = new byte[128];
// Span will start as 'covering' the entire array.
var writeSpan = buffer.AsSpan();
WriteInt(ref writeSpan, 1337);
// Span now covers the array starting from byte 4 (because we wrote 4 bytes).
WriteInt(ref writeSpan, 42);
// Span now covers the array starting from byte 8.
// Knowing how much we wrote is a simple matter of subtracting from the array length.
var bytesWritten = buffer.Length - writeSpan.Length;Which allows for neat abstractions:
// Writing.
public static void WritePlayer(ref Span<byte> span, Player player)
{
BinSerialize.WriteInt(ref span, player.Id);
BinSerialize.WriteString(ref span, player.Name);
BinSerialize.WriteInt(ref span, player.Score);
}
public static void WriteTeam(ref Span<byte> span, Player[] team)
{
BinSerialize.WriteByte(ref span, (byte)team.Length);
foreach (var player in team)
WritePlayer(ref span, player);
}
// Reading.
public static Player ReadPlayer(ref ReadOnlySpan<byte> span) => new Player(
id: BinSerialize.ReadInt(ref span),
name: BinSerialize.ReadString(ref span),
score: BinSerialize.ReadInt(ref span));
public static Player[] ReadTeam(ref ReadOnlySpan<byte> span)
{
var length = BinSerialize.ReadByte(ref span);
var result = new Player[length];
for (var i = 0; i < length; i++)
result[i] = ReadPlayer(ref span);
return result;
}All the basic primitives are implemented in the BinSerialize class below.
While BinaryWriter can write into any Stream implementation, when it comes to writing into memory
you are stuck with MemoryStream. Span<byte> is more flexible as it can be backed by either a
byte[], bytes on the stack or even unmanaged memory.
Example of writing into stack memory:
Span<byte> buffer = stackalloc byte[8];
BinSerialize.WriteInt(ref buffer, 42);
BinSerialize.WriteInt(ref buffer, 1337);Example of writing into unmanaged heap memory:
// Allocate 8 bytes on the unmanaged heap.
IntPtr unmanagedMem = Marshal.AllocHGlobal(8);
byte* typedPointer = (byte*)unmanagedMem.ToPointer();
var writeSpan = new Span<byte>(typedPointer, length: 8);
BinSerialize.WriteInt(ref writeSpan, 42);
BinSerialize.WriteInt(ref writeSpan, 1337);
// Free the memory.
Marshal.FreeHGlobal(unmanagedMem);Because we know that its always backed by actual memory we can do some smart things to avoid having to write in multiple passes.
For example a common case is to include a header of what data will follow, but what if you want to check some condition to know if you want to write that data.
Imaging a transform where you want to serialize the fields if they are different from another transform:
public bool WriteDelta(ref Span<byte> span, TransformComp other)
{
// Header containing what fields will follow.
ref byte header = ref BinSerialize.ReserveByte(ref span);
// Write the fields (And update the header).
if (!this.Position.Equals(other.Position))
{
header |= 0b00000001;
this.Position.Write(ref span);
}
if (!this.Rotation.Equals(other.Rotation))
{
header |= 0b00000010;
this.Rotation.Write(ref span);
}
if (!this.Scale.Equals(other.Scale))
{
header |= 0b00000100;
this.Scale.Write(ref span);
}
return header != 0;
}This uses the ref locals feature from c# 7.0 to update a previous location.
This way we avoid doing the 'Equals' checks twice, or having to write into a temporary buffer first.
Disadvantages of using Span<byte> vs BinaryWriter.
Main disadvantage is that MemoryStream automatically expands its backing array when you reach the
end. While if you run out of a Span<byte> then thats it. Of course its possible to build some
abstraction around it that allows for requesting more memory but that will add more overhead and
will reduce the flexibility.
But luckily in most cases its perfectly possible to estimate how big the buffers need to be. And if
you pool the actual backing memory (for example with ArrayPool<byte>.Shared) its perfectly fine
to use quite oversized buffers.
Although not specific to Span<byte> i thought it might be usefull to mention some of the packing
means we ended up implementing.
Available through the BinSerialize.WritePackedUnsignedInteger method. This uses the will known
technique of storing a 7 bit integer followed by a single bit indicating if more data will follow.
More info on wikipedia.
This table shows the sizes for certain values:
| Value | Bytes |
|---|---|
| 0 to 127 | 1 byte |
| 128 to 16383 | 2 bytes |
| 16384 to 2097151 | 3 bytes |
| 2097152 to 268435455 | 4 bytes |
| more then 268435456 | 5 bytes |
As you can see its pretty good as a general purpose packing technique.
Available through the BinSerialize.WritePackedInteger method. With signed integers there is one
more piece of data we need to store, namely the sign bit. Normally the sign is on the most-significant
bit (which is unfortunate for us as it screws with our variable-length encoding, so storing -1 would
already take 5 bytes). To fix that we use something called ZigZagEncoding
which moves the sign to the least significant bit.
This table shows the sizes for certain values:
| Value | Bytes |
|---|---|
| less then -134217729 | 5 bytes |
| -134217728 to -1048577 | 4 bytes |
| -1048576 to -8193 | 3 bytes |
| -8192 to -65 | 2 bytes |
| -64 to 63 | 1 byte |
| 64 to 8191 | 2 bytes |
| 8192 to 1048575 | 3 bytes |
| 1048576 to 134217727 | 4 bytes |
| more then 134217728 | 5 bytes |
As you can see the size increase for the sign is now quite acceptable.
Best way we found to compress floats is to send them as a fraction of a well known range. For example
if we know that a value is always in the 0-1 range we can get pretty good precision already by sending
it as a 8 bit value. 0-1 as a 8 bit value has 0,00390625 precision already.
BinSerialize has the following methods to send ranges:
Write8BitRange(float min, float max, float value)(256steps between min and max)Write16BitRange(float min, float max, float value)(65535steps between min and max)
You might think that using 16 bit floats would be a nice general purpose compressor for floats,
at least thats what we thought. But 16 bit floats have quite limited precision already, the wikipedia
article has some nice tables showing the precision. And when you are developing a feature its non-trivial to
think if a 16 bit float will have enough precision. We ran into trouble quite a few times with values that looked
innocent enough actually got high-enough where we lost all decimal points. In practice its much nicer to use
ranges (as shown above) to compress floats and it gives a very easy way to calculate how much precision your
value will have.
Implementation of utilities for reading / writing binary data in LittleEndian.
Note: If you are targeting netstandard you'll need to take a dependency on System.Memory.
Note: We only support LittleEndian platforms, if you need mixed endianness support then this needs adjusting.
public static unsafe class BinSerialize
{
private static Encoding uft8 = Encoding.UTF8;
private static Decoder utf8decoder = Encoding.UTF8.GetDecoder();
public static void ThrowForBigEndian()
{
if (!BitConverter.IsLittleEndian)
throw new NotSupportedException("BigEndian systems are not supported at this time.");
}
/// <summary>
/// Read a packed integer.
/// </summary>
/// <remarks>
/// See <see cref="WritePackedInteger"/> for more information.
/// </remarks>
/// <param name="span">Span to read from.</param>
/// <returns>Unpacked integer.</returns>
public static int ReadPackedInteger(ref ReadOnlySpan<byte> span)
{
var zigzagged = ReadPackedUnsignedInteger(ref span);
return FromZigZagEncoding(zigzagged);
}
/// <summary>
/// Read a packed unsigned integer.
/// </summary>
/// <remarks>
/// See <see cref="WritePackedUnsignedInteger"/> for more information.
/// </remarks>
/// <param name="span">Span to read from.</param>
/// <returns>Unpacked unsigned integer.</returns>
public static uint ReadPackedUnsignedInteger(ref ReadOnlySpan<byte> span)
{
/* Read 7 bits of integer data and then the 8th bit indicates wether more data will follow.
More info: https://en.wikipedia.org/wiki/Variable-length_quantity */
uint result = 0;
var resultBitOffset = 0;
while (true)
{
var data = ReadByte(ref span);
// Mask of the first 7 bits of the data and then 'apply' it to the result.
result |= (uint)(data & 0b0111_1111) << resultBitOffset;
// Check the last bit to see if this was the end.
if ((data & 0b1000_0000) == 0)
break;
// Increment the offset so the next iteration points at the next bits of the result.
resultBitOffset += 7;
}
return result;
}
/// <summary>
/// Check how many many bytes it will take to write the given value as a packed integer.
/// </summary>
/// <remarks>
/// See <see cref="WritePackedInteger"/> for more information (including a size-table).
/// </remarks>
/// <param name="value">Value to check.</param>
/// <returns>Number of bytes it will take.</returns>
public static int GetSizeForPackedInteger(int value)
{
var zigzagged = ToZigZagEncoding(value);
return GetSizeForPackedUnsignedInteger(zigzagged);
}
/// <summary>
/// Check how many many bytes it will take to write the given value as a packed unsigned integer.
/// </summary>
/// <remarks>
/// See <see cref="WritePackedUnsignedInteger"/> for more information (including a size-table).
/// </remarks>
/// <param name="value">Value to check.</param>
/// <returns>Number of bytes it will take.</returns>
public static int GetSizeForPackedUnsignedInteger(uint value)
{
/* Check how many 7 bit values we need to store the integer, for more info see
'WritePackedUnsignedInteger' implementation. */
var bytes = 1;
while (value > 0b0111_1111)
{
value >>= 7;
bytes++;
}
return bytes;
}
/// <summary>
/// Pack a integer and write it.
/// Uses a variable-length encoding scheme.
/// </summary>
/// <remarks>
/// Size table:
/// less then -134217729 = 5 bytes
/// -134217728 to -1048577 = 4 bytes
/// -1048576 to -8193 = 3 bytes
/// -8192 to -65 = 2 bytes
/// -64 to 63 = 1 bytes
/// 64 to 8191 = 2 bytes
/// 8192 to 1048575 = 3 bytes
/// 1048576 to 134217727 = 4 bytes
/// more then 134217728 = 5 bytes
/// </remarks>
/// <param name="span">Span to write to.</param>
/// <param name="value">Value to pack and write.</param>
public static void WritePackedInteger(ref Span<byte> span, int value)
{
var zigzagged = ToZigZagEncoding(value);
WritePackedUnsignedInteger(ref span, zigzagged);
}
/// <summary>
/// Pack a unsigned integer and write it.
/// Uses a variable-length encoding scheme.
/// </summary>
/// <remarks>
/// Size table:
/// 0 to 127 = 1 bytes
/// 128 to 16383 = 2 bytes
/// 16384 to 2097151 = 3 bytes
/// 2097152 to 268435455 = 4 bytes
/// more then 268435456 = 5 bytes
/// </remarks>
/// <param name="span">Span to write to.</param>
/// <param name="value">Value to pack and write.</param>
public static void WritePackedUnsignedInteger(ref Span<byte> span, uint value)
{
/* Write 7 bits of integer data and then the 8th bit indicates wether more data will follow.
More info: https://en.wikipedia.org/wiki/Variable-length_quantity */
// As long as we have more data left then we can fit into 7 bits we need to 'split' it up.
while (value > 0b0111_1111)
{
// Write out the value and set the 8th bit to 1 to indicate more data will follow.
WriteByte(ref span, (byte)(value | 0b1000_0000));
// Shift the value by 7 to 'consume' the bits we've just written.
value >>= 7;
}
// Write out the last data (the 8th bit will always be 0 here to indicate the end).
WriteByte(ref span, (byte)value);
}
/// <summary>
/// Write a boolean.
/// </summary>
/// <remarks>
/// Will consume 1 byte.
/// </remarks>
/// <param name="span">Span to write to.</param>
/// <param name="val">Value to write.</param>
public static void WriteBool(ref Span<byte> span, bool val)
{
MemoryMarshal.Write(span, ref val);
// 'Advance' the span.
span = span.Slice(sizeof(bool));
}
/// <summary>
/// Write a unsigned 8 bit integer.
/// </summary>
/// <remarks>
/// Will consume 1 byte.
/// </remarks>
/// <param name="span">Span to write to.</param>
/// <param name="val">Value to write.</param>
public static void WriteByte(ref Span<byte> span, byte val)
{
MemoryMarshal.Write(span, ref val);
// 'Advance' the span.
span = span.Slice(sizeof(byte));
}
/// <summary>
/// Write a signed 8 bit integer.
/// </summary>
/// <remarks>
/// Will consume 1 byte.
/// </remarks>
/// <param name="span">Span to write to.</param>
/// <param name="val">Value to write.</param>
public static void WriteSByte(ref Span<byte> span, sbyte val)
{
MemoryMarshal.Write(span, ref val);
// 'Advance' the span.
span = span.Slice(sizeof(sbyte));
}
/// <summary>
/// Write a 16 bit signed integer.
/// </summary>
/// <remarks>
/// Will consume 2 bytes.
/// </remarks>
/// <param name="span">Span to write to.</param>
/// <param name="val">Value to write.</param>
public static void WriteShort(ref Span<byte> span, short val)
{
MemoryMarshal.Write(span, ref val);
// 'Advance' the span.
span = span.Slice(sizeof(short));
}
/// <summary>
/// Write a 16 bit unsigned integer.
/// </summary>
/// <remarks>
/// Will consume 2 bytes.
/// </remarks>
/// <param name="span">Span to write to.</param>
/// <param name="val">Value to write.</param>
public static void WriteUShort(ref Span<byte> span, ushort val)
{
MemoryMarshal.Write(span, ref val);
// 'Advance' the span.
span = span.Slice(sizeof(ushort));
}
/// <summary>
/// Write a 32 bit signed integer.
/// </summary>
/// <remarks>
/// Will consume 4 bytes.
/// </remarks>
/// <param name="span">Span to write to.</param>
/// <param name="val">Value to write.</param>
public static void WriteInt(ref Span<byte> span, int val)
{
MemoryMarshal.Write(span, ref val);
// 'Advance' the span.
span = span.Slice(sizeof(int));
}
/// <summary>
/// Write a 32 bit unsigned integer.
/// </summary>
/// <remarks>
/// Will consume 4 bytes.
/// </remarks>
/// <param name="span">Span to write to.</param>
/// <param name="val">Value to write.</param>
public static void WriteUInt(ref Span<byte> span, uint val)
{
MemoryMarshal.Write(span, ref val);
// 'Advance' the span.
span = span.Slice(sizeof(uint));
}
/// <summary>
/// Write a 64 bit signed integer.
/// </summary>
/// <remarks>
/// Will consume 8 bytes.
/// </remarks>
/// <param name="span">Span to write to.</param>
/// <param name="val">Value to write.</param>
public static void WriteLong(ref Span<byte> span, long val)
{
MemoryMarshal.Write(span, ref val);
// 'Advance' the span.
span = span.Slice(sizeof(long));
}
/// <summary>
/// Write a 64 bit unsigned integer.
/// </summary>
/// <remarks>
/// Will consume 8 bytes.
/// </remarks>
/// <param name="span">Span to write to.</param>
/// <param name="val">Value to write.</param>
public static void WriteULong(ref Span<byte> span, ulong val)
{
MemoryMarshal.Write(span, ref val);
// 'Advance' the span.
span = span.Slice(sizeof(ulong));
}
/// <summary>
/// Write a 32 bit floating-point number.
/// </summary>
/// <remarks>
/// Will consume 4 bytes.
/// </remarks>
/// <param name="span">Span to write to.</param>
/// <param name="val">Value to write.</param>
public static void WriteFloat(ref Span<byte> span, float val)
{
MemoryMarshal.Write(span, ref val);
// 'Advance' the span.
span = span.Slice(sizeof(float));
}
/// <summary>
/// Write a 64 bit floating-point number.
/// </summary>
/// <remarks>
/// Will consume 8 bytes.
/// </remarks>
/// <param name="span">Span to write to.</param>
/// <param name="val">Value to write.</param>
public static void WriteDouble(ref Span<byte> span, double val)
{
MemoryMarshal.Write(span, ref val);
// 'Advance' the span.
span = span.Slice(sizeof(double));
}
/// <summary>
/// Write a value as a fraction between a given minimum and maximum.
/// Uses 8 bits so we have '256' steps between min and max.
/// </summary>
/// <remarks>
/// Will consume 1 byte.
/// </remarks>
/// <param name="span">Span to write to.</param>
/// <param name="min">Minimum value of the range.</param>
/// <param name="max">Maximum value of the range.</param>
/// <param name="val">Value to write (within the range.</param>
public static void Write8BitRange(ref Span<byte> span, float min, float max, float val)
{
// Get a 0f - 1f fraction.
var frac = Fraction(min, max, val);
// Remap it to a byte and write it (+ .5f because we want round, not floor).
BinSerialize.WriteByte(ref span, (byte)((byte.MaxValue * frac) + .5f));
}
/// <summary>
/// Write a value as a fraction between a given minimum and maximum.
/// Uses 16 bits so we have '65535' steps between min and max.
/// </summary>
/// <remarks>
/// Will consume 2 bytes.
/// </remarks>
/// <param name="span">Span to write to.</param>
/// <param name="min">Minimum value of the range.</param>
/// <param name="max">Maximum value of the range.</param>
/// <param name="val">Value to write (within the range.</param>
public static void Write16BitRange(ref Span<byte> span, float min, float max, float val)
{
// Get a 0f - 1f fraction.
var frac = Fraction(min, max, val);
// Remap it to a ushort and write it (+ .5f because we want round, not floor).
BinSerialize.WriteUShort(ref span, (ushort)((ushort.MaxValue * frac) + .5f));
}
/// <summary>
/// Write a continuous block of bytes.
/// </summary>
/// <remarks>
/// Will consume as many bytes as are in the given block.
/// </remarks>
/// <param name="span">Span to write to.</param>
/// <param name="val">Block of bytes to write.</param>
public static void WriteBlock(ref Span<byte> span, ReadOnlySpan<byte> val)
{
val.CopyTo(span);
// 'Advance' the span.
span = span.Slice(val.Length);
}
/// <summary>
/// Check how many many bytes it will take to write the given string value.
/// </summary>
/// <remarks>
/// Size will be the length of the string as a 'packed unsigned integer' + the amount of
/// bytes when the characters are utf-8 encoded.
/// </remarks>
/// <param name="val">Value to get the size for.</param>
/// <returns>Number of bytes it will take.</returns>
public static int GetSizeForString(string val)
{
fixed (char* charPointer = val)
{
return GetSizeForString(charPointer, val.Length);
}
}
/// <summary>
/// Check how many many bytes it will take to write the given string.
/// </summary>
/// <remarks>
/// Size will be the length of the span as a 'packed unsigned integer' + the amount of
/// bytes when the characters are utf-8 encoded.
/// </remarks>
/// <param name="val">Value to get the size for.</param>
/// <returns>Number of bytes it will take.</returns>
public static int GetSizeForString(ReadOnlySpan<char> val)
{
fixed (char* charPointer = val)
{
return GetSizeForString(charPointer, val.Length);
}
}
/// <summary>
/// Check how many many bytes it will take to write the given string.
/// Make sure the data behind the pointer is pinned before calling this.
/// </summary>
/// <remarks>
/// Size will be the charCount as a 'packed unsigned integer' + the amount of
/// bytes when the characters are utf-8 encoded.
/// </remarks>
/// <param name="charPointer">Pointer to the first character.</param>
/// <param name="charPointer">How many characters are in the string.</param>
/// <returns>Number of bytes it will take.</returns>
public static int GetSizeForString(char* charPointer, int charCount)
{
var headerSize = GetSizeForPackedUnsignedInteger((uint)charCount);
var charsSize = uft8.GetByteCount(charPointer, charCount);
return headerSize + charsSize;
}
/// <summary>
/// Write a string as utf8.
/// </summary>
/// <remarks>
/// Prefixes the data with a 'packed unsigned integer' telling how many bytes will follow.
/// Format will match that of a <see cref="System.IO.BinaryWriter"/> that is using utf8 encoding.
/// </remarks>
/// <param name="span">Span to write to.</param>
/// <param name="val">Value to write.</param>
public static void WriteString(ref Span<byte> span, string val)
{
fixed (char* charPointer = val)
{
WriteString(ref span, charPointer, val.Length);
}
}
/// <summary>
/// Write a string as utf8.
/// </summary>
/// <remarks>
/// Prefixes the data with a 'packed unsigned integer' telling how many bytes will follow.
/// Format will match that of a <see cref="System.IO.BinaryWriter"/> that is using utf8 encoding.
/// </remarks>
/// <param name="span">Span to write to.</param>
/// <param name="val">Value to write.</param>
public static void WriteString(ref Span<byte> span, ReadOnlySpan<char> val)
{
fixed (char* charPointer = val)
{
WriteString(ref span, charPointer, val.Length);
}
}
/// <summary>
/// Write a string as utf8.
/// Make sure the data behind the pointer is pinned before calling this.
/// </summary>
/// <remarks>
/// Prefixes the data with a 'packed unsigned integer' telling how many bytes will follow.
/// Format will match that of a <see cref="System.IO.BinaryWriter"/> that is using utf8 encoding.
/// </remarks>
/// <param name="span">Span to write to.</param>
/// <param name="charPointer">Pointer to the first character.</param>
/// <param name="charPointer">How many characters are in the string.</param>
public static void WriteString(ref Span<byte> span, char* charPointer, int charCount)
{
// Write amount of bytes will follow.
var byteCount = uft8.GetByteCount(charPointer, charCount);
WritePackedUnsignedInteger(ref span, (uint)byteCount);
fixed (byte* spanPointer = span)
{
// Write chars as utf8.
var writtenBytes = uft8.GetBytes(charPointer, charCount, spanPointer, span.Length);
Debug.Assert(byteCount == writtenBytes, "Written bytes did not match encodings expected size");
}
// 'Advance' the span.
span = span.Slice(byteCount);
}
/// <summary>
/// Write a unmanaged struct.
/// </summary>
/// <remarks>
/// When using this make sure that 'T' has a explict memory-layout so its consistent
/// accross platforms.
/// In other words, only use this if you are 100% sure its safe to do so.
/// Will consume sizeof T.
/// </remarks>
/// <param name="span">Span to write to.</param>
/// <param name="val">Value to write.</param>
/// <typeparam name="T">Type of the struct to write.</typeparam>
public static void WriteStruct<T>(ref Span<byte> span, T val)
where T : unmanaged
{
MemoryMarshal.Write(span, ref val);
// 'Advance' the span.
var size = Unsafe.SizeOf<T>();
span = span.Slice(size);
}
/// <summary>
/// Read a boolean.
/// </summary>
/// <remarks>
/// Will consume 1 byte.
/// </remarks>
/// <param name="span">Span to read from.</param>
/// <returns>Read value.</returns>
public static bool ReadBool(ref ReadOnlySpan<byte> span)
{
var result = MemoryMarshal.Read<bool>(span);
// 'Advance' the span.
span = span.Slice(sizeof(bool));
return result;
}
/// <summary>
/// Read a unsigned 8 bit integer.
/// </summary>
/// <remarks>
/// Will consume 1 byte.
/// </remarks>
/// <param name="span">Span to read from.</param>
/// <returns>Read value.</returns>
public static byte ReadByte(ref ReadOnlySpan<byte> span)
{
var result = MemoryMarshal.Read<byte>(span);
// 'Advance' the span.
span = span.Slice(sizeof(byte));
return result;
}
/// <summary>
/// Read a signed 8 bit integer.
/// </summary>
/// <remarks>
/// Will consume 1 byte.
/// </remarks>
/// <param name="span">Span to read from.</param>
/// <returns>Read value.</returns>
public static sbyte ReadSByte(ref ReadOnlySpan<byte> span)
{
var result = MemoryMarshal.Read<sbyte>(span);
// 'Advance' the span.
span = span.Slice(sizeof(sbyte));
return result;
}
/// <summary>
/// Read a signed 16 bit integer.
/// </summary>
/// <remarks>
/// Will consume 2 bytes.
/// </remarks>
/// <param name="span">Span to read from.</param>
/// <returns>Read value.</returns>
public static short ReadShort(ref ReadOnlySpan<byte> span)
{
var result = MemoryMarshal.Read<short>(span);
// 'Advance' the span.
span = span.Slice(sizeof(short));
return result;
}
/// <summary>
/// Read a unsigned 16 bit integer.
/// </summary>
/// <remarks>
/// Will consume 2 bytes.
/// </remarks>
/// <param name="span">Span to read from.</param>
/// <returns>Read value.</returns>
public static ushort ReadUShort(ref ReadOnlySpan<byte> span)
{
var result = MemoryMarshal.Read<ushort>(span);
// 'Advance' the span.
span = span.Slice(sizeof(ushort));
return result;
}
/// <summary>
/// Read a signed 32 bit integer.
/// </summary>
/// <remarks>
/// Will consume 4 bytes.
/// </remarks>
/// <param name="span">Span to read from.</param>
/// <returns>Read value.</returns>
public static int ReadInt(ref ReadOnlySpan<byte> span)
{
var result = MemoryMarshal.Read<int>(span);
// 'Advance' the span.
span = span.Slice(sizeof(int));
return result;
}
/// <summary>
/// Read a unsigned 32 bit integer.
/// </summary>
/// <remarks>
/// Will consume 4 bytes.
/// </remarks>
/// <param name="span">Span to read from.</param>
/// <returns>Read value.</returns>
public static uint ReadUInt(ref ReadOnlySpan<byte> span)
{
var result = MemoryMarshal.Read<uint>(span);
// 'Advance' the span.
span = span.Slice(sizeof(uint));
return result;
}
/// <summary>
/// Read a signed 64 bit integer.
/// </summary>
/// <remarks>
/// Will consume 8 bytes.
/// </remarks>
/// <param name="span">Span to read from.</param>
/// <returns>Read value.</returns>
public static long ReadLong(ref ReadOnlySpan<byte> span)
{
var result = MemoryMarshal.Read<long>(span);
// 'Advance' the span.
span = span.Slice(sizeof(long));
return result;
}
/// <summary>
/// Read a unsigned 64 bit integer.
/// </summary>
/// <remarks>
/// Will consume 8 bytes.
/// </remarks>
/// <param name="span">Span to read from.</param>
/// <returns>Read value.</returns>
public static ulong ReadULong(ref ReadOnlySpan<byte> span)
{
var result = MemoryMarshal.Read<ulong>(span);
// 'Advance' the span.
span = span.Slice(sizeof(ulong));
return result;
}
/// <summary>
/// Read a 32 bit floating-point number.
/// </summary>
/// <remarks>
/// Will consume 4 bytes.
/// </remarks>
/// <param name="span">Span to read from.</param>
/// <returns>Read value.</returns>
public static float ReadFloat(ref ReadOnlySpan<byte> span)
{
var result = MemoryMarshal.Read<float>(span);
// 'Advance' the span.
span = span.Slice(sizeof(float));
return result;
}
/// <summary>
/// Read a 64 bit floating-point number.
/// </summary>
/// <remarks>
/// Will consume 4 bytes.
/// </remarks>
/// <param name="span">Span to read from.</param>
/// <returns>Read value.</returns>
public static double ReadDouble(ref ReadOnlySpan<byte> span)
{
var result = MemoryMarshal.Read<double>(span);
// 'Advance' the span.
span = span.Slice(sizeof(double));
return result;
}
/// <summary>
/// Read a value as a fraction between a given minimum and maximum.
/// Uses 8 bits so we have '256' steps between min and max.
/// </summary>
/// <remarks>
/// Will consume 1 byte.
/// </remarks>
/// <param name="span">Span to read from.</param>
/// <param name="min">Minimum value of the range.</param>
/// <param name="max">Maximum value of the range.</param>
/// <returns>Read value.</returns>
public static float Read8BitRange(ref ReadOnlySpan<byte> span, float min, float max)
{
// Read a byte.
var raw = BinSerialize.ReadByte(ref span);
// Remap it to the given range.
return Interpolate(min, max, (float)raw / byte.MaxValue);
}
/// <summary>
/// Read a value as a fraction between a given minimum and maximum.
/// Uses 16 bits so we have '65535' steps between min and max.
/// </summary>
/// <remarks>
/// Will consume 2 bytes.
/// </remarks>
/// <param name="span">Span to read from.</param>
/// <param name="min">Minimum value of the range.</param>
/// <param name="max">Maximum value of the range.</param>
/// <returns>Read value.</returns>
public static float Read16BitRange(ref ReadOnlySpan<byte> span, float min, float max)
{
// Read a ushort.
var raw = BinSerialize.ReadUShort(ref span);
// Remap it to the given range.
return Interpolate(min, max, (float)raw / ushort.MaxValue);
}
/// <summary>
/// Read a string.
/// </summary>
/// <remarks>
/// Can only be used for strings less then 128 kib as utf8, for bigger strings use a overload
/// where you pass a 'Span{char}' as the output buffer.
/// </remarks>
/// <param name="span">Span to read from.</param>
/// <returns>Read string.</returns>
public static string ReadString(ref ReadOnlySpan<byte> span)
{
const int MaxStackStringBytes = 128 * 1024; // 128 KiB.
// Read how many bytes will follow.
var byteCount = (int)ReadPackedUnsignedInteger(ref span);
// Check if the span contains the entire string.
if (span.Length < byteCount)
throw new ArgumentOutOfRangeException(nameof(span), "Given span is incomplete");
// Sanity check the size before allocating space on the stack.
if (byteCount >= MaxStackStringBytes)
throw new ArgumentException("Input contains a string with too many bytes to fit on the stack", nameof(span));
// Decode on the stack to avoid having to allocate a temporary buffer on the heap.
var maxCharCount = uft8.GetMaxCharCount(byteCount);
var charBuffer = stackalloc char[maxCharCount];
// Read chars as utf8.
int actualCharCount;
fixed (byte* bytePointer = span)
{
actualCharCount = utf8decoder.GetChars(bytePointer, byteCount, charBuffer, maxCharCount, flush: false);
}
// 'Advance' the span.
span = span.Slice(byteCount);
// Allocate the string.
return new string(charBuffer, startIndex: 0, length: actualCharCount);
}
/// <summary>
/// Read a string to a given output-buffer.
/// </summary>
/// <param name="span">Span to read from.</param>
/// <param name="chars">Buffer to write to.</param>
/// <returns>Amount of characters written</returns>
public static int ReadString(ref ReadOnlySpan<byte> span, Span<char> chars)
{
// Read amount of bytes will follow.
var byteCount = (int)ReadPackedUnsignedInteger(ref span);
// Check if input span contains the entire string.
if (span.Length < byteCount)
throw new ArgumentOutOfRangeException(nameof(span), "Given span is incomplete");
// No need to check if the output span has enough space as 'Encoding.GetChars' will
// already do that for us.
// Read chars as utf8.
int charsRead;
fixed (char* charPointer = chars)
fixed (byte* bytePointer = span)
{
charsRead = uft8.GetChars(bytePointer, byteCount, charPointer, chars.Length);
}
// 'Advance' the span.
span = span.Slice(byteCount);
return charsRead;
}
/// <summary>
/// Read a unmanaged struct.
/// </summary>
/// <remarks>
/// When using this make sure that 'T' has a explict memory-layout so its consistent
/// accross platforms.
/// In other words, only use this if you are 100% sure its safe to do so.
/// </remarks>
/// <param name="span">Span to read from.</param>
/// <typeparam name="T">Type of the struct to read.</typeparam>
/// <returns>Read value.</returns>
public static T ReadStruct<T>(ref ReadOnlySpan<byte> span)
where T : unmanaged
{
var result = MemoryMarshal.Read<T>(span);
// 'Advance' the span.
var size = Unsafe.SizeOf<T>();
span = span.Slice(size);
return result;
}
/// <summary>
/// Read a continuous block of bytes as a new byte-array.
/// </summary>
/// <remarks>
/// Will consume '<paramref name="byteCount"/>' amount of bytes.
/// </remarks>
/// <param name="span">Span to read from.</param>
/// <param name="byteCount">Amount of bytes to read.</param>
/// <returns>New byte-array containing the read bytes.</returns>
public static byte[] ReadBlock(ref ReadOnlySpan<byte> span, int byteCount)
{
var result = new byte[byteCount];
ReadBlock(ref span, result);
return result;
}
/// <summary>
/// Read a continuous block of bytes into given output span.
/// </summary>
/// <remarks>
/// Will consume length of '<paramref name="output"/>'.
/// </remarks>
/// <param name="span">Span to read from.</param>
/// <param name="output">Span to write to.</param>
public static void ReadBlock(ref ReadOnlySpan<byte> span, Span<byte> output)
{
span.Slice(0, output.Length).CopyTo(output);
// 'Advance' the span.
span = span.Slice(output.Length);
}
/// <summary>
/// 'Reserve' space for a boolean and return a ref to the space.
/// </summary>
/// <remarks>
/// Will consume 1 byte.
/// </remarks>
/// <param name="span">Span to reserver from.</param>
/// <returns>Reference to the reserved space.</returns>
public static ref bool ReserveBool(ref Span<byte> span)
{
ref bool result = ref Unsafe.As<byte, bool>(ref span[0]);
// Init to default, as otherwise it would be whatever data was at that memory.
result = default;
// 'Advance' the span.
span = span.Slice(sizeof(bool));
return ref result;
}
/// <summary>
/// 'Reserve' space for a unsigned 8 bit integer.
/// </summary>
/// <remarks>
/// Will consume 1 byte.
/// </remarks>
/// <param name="span">Span to reserver from.</param>
/// <returns>Reference to the reserved space.</returns>
public static ref byte ReserveByte(ref Span<byte> span)
{
ref byte result = ref span[0];
// Init to default, as otherwise it would be whatever data was at that memory.
result = default;
// 'Advance' the span.
span = span.Slice(sizeof(byte));
return ref result;
}
/// <summary>
/// 'Reserve' space for a signed 8 bit integer.
/// </summary>
/// <remarks>
/// Will consume 1 byte.
/// </remarks>
/// <param name="span">Span to reserver from.</param>
/// <returns>Reference to the reserved space.</returns>
public static ref sbyte ReserveSByte(ref Span<byte> span)
{
ref sbyte result = ref Unsafe.As<byte, sbyte>(ref span[0]);
// Init to default, as otherwise it would be whatever data was at that memory.
result = default;
// 'Advance' the span.
span = span.Slice(sizeof(sbyte));
return ref result;
}
/// <summary>
/// 'Reserve' space for a signed 16 bit integer.
/// </summary>
/// <remarks>
/// Will consume 2 bytes.
/// </remarks>
/// <param name="span">Span to reserver from.</param>
/// <returns>Reference to the reserved space.</returns>
public static ref short ReserveShort(ref Span<byte> span)
{
ref short result = ref Unsafe.As<byte, short>(ref span[0]);
// Init to default, as otherwise it would be whatever data was at that memory.
result = default;
// 'Advance' the span.
span = span.Slice(sizeof(short));
return ref result;
}
/// <summary>
/// 'Reserve' space for a unsigned 16 bit integer.
/// </summary>
/// <remarks>
/// Will consume 2 bytes.
/// </remarks>
/// <param name="span">Span to reserver from.</param>
/// <returns>Reference to the reserved space.</returns>
public static ref ushort ReserveUShort(ref Span<byte> span)
{
ref ushort result = ref Unsafe.As<byte, ushort>(ref span[0]);
// Init to default, as otherwise it would be whatever data was at that memory.
result = default;
// 'Advance' the span.
span = span.Slice(sizeof(ushort));
return ref result;
}
/// <summary>
/// 'Reserve' space for a signed 32 bit integer.
/// </summary>
/// <remarks>
/// Will consume 4 bytes.
/// </remarks>
/// <param name="span">Span to reserver from.</param>
/// <returns>Reference to the reserved space.</returns>
public static ref int ReserveInt(ref Span<byte> span)
{
ref int result = ref Unsafe.As<byte, int>(ref span[0]);
// Init to default, as otherwise it would be whatever data was at that memory.
result = default;
// 'Advance' the span.
span = span.Slice(sizeof(int));
return ref result;
}
/// <summary>
/// 'Reserve' space for a unsigned 32 bit integer.
/// </summary>
/// <remarks>
/// Will consume 4 bytes.
/// </remarks>
/// <param name="span">Span to reserver from.</param>
/// <returns>Reference to the reserved space.</returns>
public static ref uint ReserveUInt(ref Span<byte> span)
{
ref uint result = ref Unsafe.As<byte, uint>(ref span[0]);
// Init to default, as otherwise it would be whatever data was at that memory.
result = default;
// 'Advance' the span.
span = span.Slice(sizeof(uint));
return ref result;
}
/// <summary>
/// 'Reserve' space for a signed 64 bit integer.
/// </summary>
/// <remarks>
/// Will consume 8 bytes.
/// </remarks>
/// <param name="span">Span to reserver from.</param>
/// <returns>Reference to the reserved space.</returns>
public static ref long ReserveLong(ref Span<byte> span)
{
ref long result = ref Unsafe.As<byte, long>(ref span[0]);
// Init to default, as otherwise it would be whatever data was at that memory.
result = default;
// 'Advance' the span.
span = span.Slice(sizeof(long));
return ref result;
}
/// <summary>
/// 'Reserve' space for a unsigned 64 bit integer.
/// </summary>
/// <remarks>
/// Will consume 8 bytes.
/// </remarks>
/// <param name="span">Span to reserver from.</param>
/// <returns>Reference to the reserved space.</returns>
public static ref ulong ReserveULong(ref Span<byte> span)
{
ref ulong result = ref Unsafe.As<byte, ulong>(ref span[0]);
// Init to default, as otherwise it would be whatever data was at that memory.
result = default;
// 'Advance' the span.
span = span.Slice(sizeof(ulong));
return ref result;
}
/// <summary>
/// 'Reserve' space for a unmanaged struct.
/// </summary>
/// <remarks>
/// When using this make sure that 'T' has a explict memory-layout so its consistent
/// accross platforms.
/// In other words, only use this if you are 100% sure its safe to do so.
/// Will consume sizeof T.
/// </remarks>
/// <param name="span">Span to reserver from.</param>
/// <typeparam name="T">Type of the unmanaged struct.</typeparam>
/// <returns>Reference to the reserved space.</returns>
public static ref T ReserveStruct<T>(ref Span<byte> span)
where T : unmanaged
{
ref T result = ref Unsafe.As<byte, T>(ref span[0]);
// Init to default, as otherwise it would be whatever data was at that memory.
result = default;
// 'Advance' the span.
var size = Unsafe.SizeOf<T>();
span = span.Slice(size);
return ref result;
}
private static uint ToZigZagEncoding(int val)
{
/* We encode the integer in such a way that the sign is on the least significant bit,
known as zig-zag encoding:
https://en.wikipedia.org/wiki/Variable-length_quantity#Zigzag_encoding */
return (uint)((val << 1) ^ (val >> 31));
}
private static int FromZigZagEncoding(uint zigzagged)
{
/* We encode integers in such a way that the sign is on the least significant bit,
known as zig-zag encoding:
https://en.wikipedia.org/wiki/Variable-length_quantity#Zigzag_encoding */
return (int)(zigzagged >> 1) ^ -(int)(zigzagged & 1);
}
private static float Interpolate(float min, float max, float frac) =>
min + ((max - min) * Clamp01(frac));
private static float Fraction(float min, float max, float val) =>
(min == max) ? 0f : Clamp01((val - min) / (max - min));
private static float Clamp01(float val) => val < 0f ? 0f : (val > 1f ? 1f : val);
}xUnit tests covering the implementation.
public class BinSerializeFacts
{
[Fact]
public void UnalignedWritesCanBeRead()
{
var buffer = new byte[64];
var writeSpan = new Span<byte>(buffer);
// Write 32 integers that are not aligned to 4 bytes.
BinSerialize.WriteByte(ref writeSpan, 137);
BinSerialize.WriteInt(ref writeSpan, 133337);
BinSerialize.WriteByte(ref writeSpan, 137);
BinSerialize.WriteInt(ref writeSpan, 133337);
var readSpan = new ReadOnlySpan<byte>(buffer);
Assert.Equal(137, BinSerialize.ReadByte(ref readSpan));
Assert.Equal(133337, BinSerialize.ReadInt(ref readSpan));
Assert.Equal(137, BinSerialize.ReadByte(ref readSpan));
Assert.Equal(133337, BinSerialize.ReadInt(ref readSpan));
}
[Theory]
[InlineData(0)]
[InlineData(int.MinValue)]
[InlineData(int.MaxValue)]
[InlineData(-1337)]
[InlineData(-137)]
[InlineData(1337)]
[InlineData(137)]
public void PackedIntegerCanBeSerialized(int val)
{
var buffer = new byte[5];
var writeSpan = new Span<byte>(buffer);
BinSerialize.WritePackedInteger(ref writeSpan, val);
var readSpan = new ReadOnlySpan<byte>(buffer);
Assert.Equal(val, BinSerialize.ReadPackedInteger(ref readSpan));
}
[Theory]
[InlineData(uint.MinValue)]
[InlineData(uint.MaxValue)]
[InlineData((uint)int.MaxValue + 1337)]
public void PackedUnsignedIntegerCanBeSerialized(uint val)
{
var buffer = new byte[5];
var writeSpan = new Span<byte>(buffer);
BinSerialize.WritePackedUnsignedInteger(ref writeSpan, val);
var readSpan = new ReadOnlySpan<byte>(buffer);
Assert.Equal(val, BinSerialize.ReadPackedUnsignedInteger(ref readSpan));
}
[Theory]
[InlineData(true)]
[InlineData(false)]
public void BoolCanBeSerialized(bool val)
{
var buffer = new byte[1];
var writeSpan = new Span<byte>(buffer);
BinSerialize.WriteBool(ref writeSpan, val);
var readSpan = new ReadOnlySpan<byte>(buffer);
Assert.Equal(val, BinSerialize.ReadBool(ref readSpan));
}
[Theory]
[InlineData(byte.MinValue)]
[InlineData(byte.MaxValue)]
[InlineData(137)]
public void ByteCanBeSerialized(byte val)
{
var buffer = new byte[1];
var writeSpan = new Span<byte>(buffer);
BinSerialize.WriteByte(ref writeSpan, val);
var readSpan = new ReadOnlySpan<byte>(buffer);
Assert.Equal(val, BinSerialize.ReadByte(ref readSpan));
}
[Theory]
[InlineData(0)]
[InlineData(sbyte.MinValue)]
[InlineData(sbyte.MaxValue)]
[InlineData(13)]
[InlineData(-13)]
public void SByteCanBeSerialized(sbyte val)
{
var buffer = new byte[1];
var writeSpan = new Span<byte>(buffer);
BinSerialize.WriteSByte(ref writeSpan, val);
var readSpan = new ReadOnlySpan<byte>(buffer);
Assert.Equal(val, BinSerialize.ReadSByte(ref readSpan));
}
[Theory]
[InlineData(0)]
[InlineData(short.MinValue)]
[InlineData(short.MaxValue)]
[InlineData(1337)]
[InlineData(-1337)]
public void ShortCanBeSerialized(short val)
{
var buffer = new byte[2];
var writeSpan = new Span<byte>(buffer);
BinSerialize.WriteShort(ref writeSpan, val);
var readSpan = new ReadOnlySpan<byte>(buffer);
Assert.Equal(val, BinSerialize.ReadShort(ref readSpan));
}
[Theory]
[InlineData(ushort.MinValue)]
[InlineData(ushort.MaxValue)]
[InlineData(1337)]
public void UShortCanBeSerialized(ushort val)
{
var buffer = new byte[2];
var writeSpan = new Span<byte>(buffer);
BinSerialize.WriteUShort(ref writeSpan, val);
var readSpan = new ReadOnlySpan<byte>(buffer);
Assert.Equal(val, BinSerialize.ReadUShort(ref readSpan));
}
[Theory]
[InlineData(0)]
[InlineData(int.MinValue)]
[InlineData(int.MaxValue)]
[InlineData(133337)]
[InlineData(-133337)]
public void IntCanBeSerialized(int val)
{
var buffer = new byte[4];
var writeSpan = new Span<byte>(buffer);
BinSerialize.WriteInt(ref writeSpan, val);
var readSpan = new ReadOnlySpan<byte>(buffer);
Assert.Equal(val, BinSerialize.ReadInt(ref readSpan));
}
[Theory]
[InlineData(uint.MinValue)]
[InlineData(uint.MaxValue)]
[InlineData(133337)]
public void UIntCanBeSerialized(uint val)
{
var buffer = new byte[4];
var writeSpan = new Span<byte>(buffer);
BinSerialize.WriteUInt(ref writeSpan, val);
var readSpan = new ReadOnlySpan<byte>(buffer);
Assert.Equal(val, BinSerialize.ReadUInt(ref readSpan));
}
[Theory]
[InlineData(0)]
[InlineData(long.MinValue)]
[InlineData(long.MaxValue)]
[InlineData(13333337)]
[InlineData(-13333337)]
public void LongCanBeSerialized(long val)
{
var buffer = new byte[8];
var writeSpan = new Span<byte>(buffer);
BinSerialize.WriteLong(ref writeSpan, val);
var readSpan = new ReadOnlySpan<byte>(buffer);
Assert.Equal(val, BinSerialize.ReadLong(ref readSpan));
}
[Theory]
[InlineData(ulong.MinValue)]
[InlineData(ulong.MaxValue)]
[InlineData(13333337)]
public void ULongCanBeSerialized(ulong val)
{
var buffer = new byte[8];
var writeSpan = new Span<byte>(buffer);
BinSerialize.WriteULong(ref writeSpan, val);
var readSpan = new ReadOnlySpan<byte>(buffer);
Assert.Equal(val, BinSerialize.ReadULong(ref readSpan));
}
[Theory]
[InlineData(0f)]
[InlineData(float.MinValue)]
[InlineData(float.MaxValue)]
[InlineData(1337.23f)]
[InlineData(-1337.62f)]
public void FloatCanBeSerialized(float val)
{
var buffer = new byte[4];
var writeSpan = new Span<byte>(buffer);
BinSerialize.WriteFloat(ref writeSpan, val);
var readSpan = new ReadOnlySpan<byte>(buffer);
Assert.Equal(val, BinSerialize.ReadFloat(ref readSpan));
}
[Theory]
[InlineData(0f, 1f, 0f, 0f)]
[InlineData(0f, 1f, 1f, 1f)]
[InlineData(0f, 1f, -.1f, 0f)]
[InlineData(0f, 1f, 1.1f, 1f)]
[InlineData(-1f, 1f, -1.1f, -1f)]
[InlineData(0f, 255f, 128f, 128f)]
[InlineData(0f, 255f, 255f, 255f)]
[InlineData(50f, 100f, 75f, 75.1f)] // 75.1f, due to only having 8 bit precision.
[InlineData(-1f, 1f, 0f, 0f)]
[InlineData(0f, 1f, .25f, .25f)]
public void _8BitRangeCanBeSerialized(float min, float max, float val, float expectedVal)
{
var buffer = new byte[1];
var writeSpan = new Span<byte>(buffer);
BinSerialize.Write8BitRange(ref writeSpan, min, max, val);
var readSpan = new ReadOnlySpan<byte>(buffer);
Assert.Equal(expectedVal, BinSerialize.Read8BitRange(ref readSpan, min, max), precision: 2);
}
[Theory]
[InlineData(0f, 1f, 0f, 0f)]
[InlineData(0f, 1f, 1f, 1f)]
[InlineData(0f, 1f, -.1f, 0f)]
[InlineData(0f, 1f, 1.1f, 1f)]
[InlineData(-1f, 1f, -1.1f, -1f)]
[InlineData(0f, 65_535f, 32_767f, 32_767f)]
[InlineData(0f, 65_535f, 65_535f, 65_535f)]
[InlineData(-1f, 1f, 0f, 0f)]
[InlineData(0f, 1f, .25f, .25f)]
public void _16BitRangeCanBeSerialized(float min, float max, float val, float expectedVal)
{
var buffer = new byte[2];
var writeSpan = new Span<byte>(buffer);
BinSerialize.Write16BitRange(ref writeSpan, min, max, val);
var readSpan = new ReadOnlySpan<byte>(buffer);
Assert.Equal(expectedVal, BinSerialize.Read16BitRange(ref readSpan, min, max), precision: 4);
}
[Theory]
[InlineData(0)]
[InlineData(double.MinValue)]
[InlineData(double.MaxValue)]
[InlineData(1337.0023)]
[InlineData(-1337.2323)]
public void DoubleCanBeSerialized(double val)
{
var buffer = new byte[8];
var writeSpan = new Span<byte>(buffer);
BinSerialize.WriteDouble(ref writeSpan, val);
var readSpan = new ReadOnlySpan<byte>(buffer);
Assert.Equal(val, BinSerialize.ReadDouble(ref readSpan));
}
[Theory]
[InlineData("")]
[InlineData("Test String")]
[InlineData("🇯🇵 🇰🇷 🇩🇪 🇨🇳 🇺🇸 🇫🇷 🇪🇸 🇮🇹 🇷🇺 🇬🇧")]
[InlineData("Test\nString\n")]
public void StringCanBeSerialized(string val)
{
var buffer = new byte[128];
var writeSpan = new Span<byte>(buffer);
BinSerialize.WriteString(ref writeSpan, val);
var readSpan = new ReadOnlySpan<byte>(buffer);
Assert.Equal(val, BinSerialize.ReadString(ref readSpan));
}
[StructLayout(LayoutKind.Sequential, Pack = 1)]
private struct TestStruct
{
public int A;
public float B;
public byte C;
public uint D;
}
[Fact]
public void StructCanBeSerialized()
{
var testStruct = default(TestStruct);
testStruct.A = 13337;
testStruct.B = 1337f;
testStruct.C = 137;
testStruct.D = 17;
var buffer = new byte[13];
var writeSpan = new Span<byte>(buffer);
BinSerialize.WriteStruct(ref writeSpan, testStruct);
var readSpan = new ReadOnlySpan<byte>(buffer);
var readStruct = BinSerialize.ReadStruct<TestStruct>(ref readSpan);
Assert.Equal(testStruct.A, readStruct.A);
Assert.Equal(testStruct.B, readStruct.B);
Assert.Equal(testStruct.C, readStruct.C);
Assert.Equal(testStruct.D, readStruct.D);
}
[Fact]
public void BlockCanBeSerialized()
{
// Get random bytes to serialize.
var random = new Random(Seed: 1337);
var data = new byte[64];
random.NextBytes(data);
// Write the data.
var buffer = new byte[128];
var writeSpan = new Span<byte>(buffer);
BinSerialize.WriteBlock(ref writeSpan, data);
var readSpan = new ReadOnlySpan<byte>(buffer);
var readBlock = BinSerialize.ReadBlock(ref readSpan, byteCount: data.Length);
Assert.NotNull(readBlock);
Assert.Equal(data.Length, readBlock.Length);
Assert.True(data.AsSpan().SequenceEqual(readBlock.AsSpan()));
}
[Fact]
public void BoolCanBeReserved()
{
var buffer = new byte[1];
var writeSpan = new Span<byte>(buffer);
ref bool reserved = ref BinSerialize.ReserveBool(ref writeSpan);
reserved = true;
var readSpan = new ReadOnlySpan<byte>(buffer);
Assert.True(BinSerialize.ReadBool(ref readSpan));
}
[Fact]
public void ByteCanBeReserved()
{
var buffer = new byte[1];
var writeSpan = new Span<byte>(buffer);
ref byte reserved = ref BinSerialize.ReserveByte(ref writeSpan);
reserved = 137;
var readSpan = new ReadOnlySpan<byte>(buffer);
Assert.Equal(137, BinSerialize.ReadByte(ref readSpan));
}
[Fact]
public void SByteCanBeReserved()
{
var buffer = new byte[1];
var writeSpan = new Span<byte>(buffer);
ref sbyte reserved = ref BinSerialize.ReserveSByte(ref writeSpan);
reserved = -17;
var readSpan = new ReadOnlySpan<byte>(buffer);
Assert.Equal(-17, BinSerialize.ReadSByte(ref readSpan));
}
[Fact]
public void ShortCanBeReserved()
{
var buffer = new byte[2];
var writeSpan = new Span<byte>(buffer);
ref short reserved = ref BinSerialize.ReserveShort(ref writeSpan);
reserved = -1337;
var readSpan = new ReadOnlySpan<byte>(buffer);
Assert.Equal(-1337, BinSerialize.ReadShort(ref readSpan));
}
[Fact]
public void UShortCanBeReserved()
{
var buffer = new byte[2];
var writeSpan = new Span<byte>(buffer);
ref ushort reserved = ref BinSerialize.ReserveUShort(ref writeSpan);
reserved = 1337;
var readSpan = new ReadOnlySpan<byte>(buffer);
Assert.Equal(1337, BinSerialize.ReadUShort(ref readSpan));
}
[Fact]
public void IntCanBeReserved()
{
var buffer = new byte[4];
var writeSpan = new Span<byte>(buffer);
ref int reserved = ref BinSerialize.ReserveInt(ref writeSpan);
reserved = -133337;
var readSpan = new ReadOnlySpan<byte>(buffer);
Assert.Equal(-133337, BinSerialize.ReadInt(ref readSpan));
}
[Fact]
public void UIntCanBeReserved()
{
var buffer = new byte[4];
var writeSpan = new Span<byte>(buffer);
ref uint reserved = ref BinSerialize.ReserveUInt(ref writeSpan);
reserved = 133337;
var readSpan = new ReadOnlySpan<byte>(buffer);
Assert.Equal((uint)133337, BinSerialize.ReadUInt(ref readSpan));
}
[Fact]
public void LongCanBeReserved()
{
var buffer = new byte[8];
var writeSpan = new Span<byte>(buffer);
ref long reserved = ref BinSerialize.ReserveLong(ref writeSpan);
reserved = -1333333337;
var readSpan = new ReadOnlySpan<byte>(buffer);
Assert.Equal(-1333333337, BinSerialize.ReadLong(ref readSpan));
}
[Fact]
public void ULongCanBeReserved()
{
var buffer = new byte[8];
var writeSpan = new Span<byte>(buffer);
ref ulong reserved = ref BinSerialize.ReserveULong(ref writeSpan);
reserved = 1333333337;
var readSpan = new ReadOnlySpan<byte>(buffer);
Assert.Equal(1333333337, BinSerialize.ReadLong(ref readSpan));
}
[Fact]
public void StructCanBeReserved()
{
var buffer = new byte[13];
var writeSpan = new Span<byte>(buffer);
ref TestStruct reserved = ref BinSerialize.ReserveStruct<TestStruct>(ref writeSpan);
reserved.A = 13337;
reserved.B = 1337f;
reserved.C = 137;
reserved.D = 17;
var readSpan = new ReadOnlySpan<byte>(buffer);
var readStruct = BinSerialize.ReadStruct<TestStruct>(ref readSpan);
Assert.Equal(reserved.A, readStruct.A);
Assert.Equal(reserved.B, readStruct.B);
Assert.Equal(reserved.C, readStruct.C);
Assert.Equal(reserved.D, readStruct.D);
}
[Theory]
[InlineData(0)]
[InlineData(int.MinValue)]
[InlineData(int.MaxValue)]
[InlineData(-133337)]
[InlineData(-1337)]
[InlineData(133337)]
[InlineData(137)]
public void PackedIntegerWriteCanBeEstimated(int val)
{
var expectedBytes = BinSerialize.GetSizeForPackedInteger(val);
var buffer = new byte[64];
var writeSpan = new Span<byte>(buffer);
BinSerialize.WritePackedInteger(ref writeSpan, val);
var writtenBytes = buffer.Length - writeSpan.Length;
Assert.Equal(writtenBytes, expectedBytes);
}
[Theory]
[InlineData(uint.MinValue)]
[InlineData(uint.MaxValue)]
[InlineData((uint)int.MaxValue + 1337)]
public void PackedUnsignedIntegerWriteCanBeEstimated(uint val)
{
var expectedBytes = BinSerialize.GetSizeForPackedUnsignedInteger(val);
var buffer = new byte[64];
var writeSpan = new Span<byte>(buffer);
BinSerialize.WritePackedUnsignedInteger(ref writeSpan, val);
var writtenBytes = buffer.Length - writeSpan.Length;
Assert.Equal(writtenBytes, expectedBytes);
}
[Theory]
[InlineData(-134217729, 5)]
[InlineData(-134217728, 4)]
[InlineData(-1048577, 4)]
[InlineData(-1048576, 3)]
[InlineData(-8193, 3)]
[InlineData(-8192, 2)]
[InlineData(-65, 2)]
[InlineData(-64, 1)]
[InlineData(0, 1)]
[InlineData(63, 1)]
[InlineData(64, 2)]
[InlineData(8191, 2)]
[InlineData(8192, 3)]
[InlineData(1048575, 3)]
[InlineData(1048576, 4)]
[InlineData(134217727, 4)]
[InlineData(134217728, 5)]
public void PackedIntegerWriteIsExpectedSize(int val, int expectedSize) =>
Assert.Equal(expectedSize, BinSerialize.GetSizeForPackedInteger(val));
[Theory]
[InlineData(0, 1)]
[InlineData(127, 1)]
[InlineData(128, 2)]
[InlineData(16383, 2)]
[InlineData(16384, 3)]
[InlineData(2097151, 3)]
[InlineData(2097152, 4)]
[InlineData(268435455, 4)]
[InlineData(268435456, 5)]
public void PackedUnsignedIntegerWriteIsExpectedSize(uint val, int expectedSize) =>
Assert.Equal(expectedSize, BinSerialize.GetSizeForPackedUnsignedInteger(val));
[Theory]
[InlineData("")]
[InlineData("Test String")]
[InlineData("🇯🇵 🇰🇷 🇩🇪 🇨🇳 🇺🇸 🇫🇷 🇪🇸 🇮🇹 🇷🇺 🇬🇧")]
[InlineData("Test\nString\n")]
public void StringWriteCanBeEstimated(string val)
{
var expectedBytes = BinSerialize.GetSizeForString(val);
var buffer = new byte[128];
var writeSpan = new Span<byte>(buffer);
BinSerialize.WriteString(ref writeSpan, val);
var writtenBytes = buffer.Length - writeSpan.Length;
Assert.Equal(writtenBytes, expectedBytes);
}
[Theory]
[InlineData("")]
[InlineData("Test String")]
[InlineData("🇯🇵 🇰🇷 🇩🇪 🇨🇳 🇺🇸 🇫🇷 🇪🇸 🇮🇹 🇷🇺 🇬🇧")]
[InlineData("Test\nString\n")]
public void StringFormatMatchesBinaryWriter(string val)
{
var memStreamBuffer = new byte[128];
int memStreamBytesWritten;
using (var memStream = new MemoryStream())
{
using (var binaryWriter = new BinaryWriter(memStream, Encoding.UTF8, leaveOpen: true))
binaryWriter.Write(val);
memStreamBytesWritten = (int)memStream.Length;
}
var buffer = new byte[128];
var writeSpan = new Span<byte>(buffer);
BinSerialize.WriteString(ref writeSpan, val);
var writtenBytes = buffer.Length - writeSpan.Length;
Assert.Equal(memStreamBytesWritten, writtenBytes);
Assert.True(
memStreamBuffer.AsSpan().Slice(memStreamBytesWritten).SequenceEqual(buffer.AsSpan().Slice(writtenBytes)));
}
}Benchmarks implemented using BenchmarkDotnet.
using System;
using System.IO;
using BenchmarkDotNet.Attributes;
[MemoryDiagnoser]
public class BinSerializeWriteBenchmark : IDisposable
{
private const int Capacity = 268_435_456; // 256 megs.
private const int IterationSize = sizeof(byte) + sizeof(int) + sizeof(long) + sizeof(float);
private const int Iterations = Capacity / IterationSize;
private readonly MemoryStream memoryStream;
private readonly BinaryWriter binaryWriter;
private readonly Memory<byte> memoryBuffer;
public BinSerializeWriteBenchmark()
{
this.memoryStream = new MemoryStream(Capacity);
this.binaryWriter = new BinaryWriter(this.memoryStream);
this.memoryBuffer = new byte[Capacity];
}
[IterationCleanup]
public void IterationCleanup()
{
// Reset the memory stream.
this.memoryStream.Position = 0;
this.memoryStream.SetLength(0);
// Write zero's in the buffer of the memory stream.
Array.Clear(this.memoryStream.GetBuffer(), index: 0, length: 0);
// Write zero's in the memory-buffer.
this.memoryBuffer.Span.Clear();
}
[Benchmark(Baseline = true)]
public void BinaryWriter_Write()
{
for (int i = 0; i < Iterations; i++)
{
this.binaryWriter.Write(0x89);
this.binaryWriter.Write(1337);
this.binaryWriter.Write(13337L);
this.binaryWriter.Write(1337f);
}
}
[Benchmark]
public void BinSerialize_Write()
{
var span = this.memoryBuffer.Span;
for (int i = 0; i < Iterations; i++)
{
BinSerialize.WriteByte(ref span, 0x89);
BinSerialize.WriteInt(ref span, 1337);
BinSerialize.WriteLong(ref span, 13337L);
BinSerialize.WriteFloat(ref span, 1337f);
}
}
public void Dispose()
{
this.binaryWriter.Dispose();
}
}| Method | Mean | Error | StdDev | Ratio | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|
| BinaryWriter_Write | 682.7 ms | 6.840 ms | 6.399 ms | 1.00 | - | - | - | - |
| BinSerialize_Write | 123.5 ms | 2.074 ms | 1.838 ms | 0.18 | - | - | - | - |
using System;
using System.IO;
using BenchmarkDotNet.Attributes;
[MemoryDiagnoser]
public class BinSerializeReadBenchmark : IDisposable
{
private const int Capacity = 268_435_456; // 256 megs.
private const int IterationSize = sizeof(byte) + sizeof(int) + sizeof(long) + sizeof(float);
private const int Iterations = Capacity / IterationSize;
private readonly Random random = new Random();
private readonly byte[] backingArray;
private readonly MemoryStream memoryStream;
private readonly BinaryReader binaryReader;
public BinSerializeReadBenchmark()
{
this.backingArray = new byte[Capacity];
this.memoryStream = new MemoryStream(this.backingArray, 0, Capacity, writable: true, publiclyVisible: true);
this.binaryReader = new BinaryReader(this.memoryStream);
}
[IterationCleanup]
public void IterationCleanup()
{
// Reset the memory stream.
this.memoryStream.Position = 0;
this.memoryStream.SetLength(Capacity);
// Initialize with random data.
this.random.NextBytes(this.backingArray);
}
[Benchmark(Baseline = true)]
public void BinaryReader_Read()
{
for (int i = 0; i < Iterations; i++)
{
this.binaryReader.ReadByte();
this.binaryReader.ReadInt32();
this.binaryReader.ReadInt64();
this.binaryReader.ReadSingle();
}
}
[Benchmark]
public void BinSerialize_Read()
{
var span = new ReadOnlySpan<byte>(this.backingArray);
for (int i = 0; i < Iterations; i++)
{
BinSerialize.ReadByte(ref span);
BinSerialize.ReadInt(ref span);
BinSerialize.ReadLong(ref span);
BinSerialize.ReadFloat(ref span);
}
}
public void Dispose()
{
this.binaryReader.Dispose();
}
}| Method | Mean | Error | StdDev | Ratio | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|
| BinaryReader_Read | 422.9 ms | 5.032 ms | 4.461 ms | 1.00 | - | - | - | - |
| BinSerialize_Read | 105.8 ms | 2.445 ms | 2.287 ms | 0.25 | - | - | - | - |
using System;
using System.IO;
using System.Text;
using BenchmarkDotNet.Attributes;
[MemoryDiagnoser]
public class BinSerializeStringsBenchmark : IDisposable
{
private const string RefString = "🇯🇵 🇰🇷 🇩🇪 🇨🇳 🇺🇸 🇫🇷 🇪🇸 🇮🇹 🇷🇺 🇬🇧";
private const int Iterations = 1_000_000;
private readonly MemoryStream memoryStream;
private readonly BinaryReader binaryReader;
private readonly BinaryWriter binaryWriter;
private readonly Memory<byte> memoryBuffer;
public BinSerializeStringsBenchmark()
{
var capacity = BinSerialize.GetSizeForString(RefString) * Iterations;
this.memoryStream = new MemoryStream(capacity);
this.binaryReader = new BinaryReader(this.memoryStream, Encoding.UTF8, leaveOpen: true);
this.binaryWriter = new BinaryWriter(this.memoryStream, Encoding.UTF8, leaveOpen: true);
this.memoryBuffer = new byte[capacity];
}
[IterationCleanup]
public void IterationCleanup()
{
// Reset the memory stream.
this.memoryStream.Position = 0;
this.memoryStream.SetLength(0);
// Write zero's in the buffer of the memory stream.
Array.Clear(this.memoryStream.GetBuffer(), index: 0, length: 0);
// Write zero's in the memory-buffer.
this.memoryBuffer.Span.Clear();
}
[Benchmark(Baseline = true)]
public void BinaryWriter_WriteReadString()
{
for (int i = 0; i < Iterations; i++)
this.binaryWriter.Write(RefString);
// Set stream back to beginning.
this.memoryStream.Position = 0;
for (int i = 0; i < Iterations; i++)
this.binaryReader.ReadString();
}
[Benchmark]
public void BinSerialize_WriteReadString()
{
Span<byte> writeSpan = this.memoryBuffer.Span;
for (int i = 0; i < Iterations; i++)
BinSerialize.WriteString(ref writeSpan, RefString);
ReadOnlySpan<byte> readSpan = this.memoryBuffer.Span;
for (int i = 0; i < Iterations; i++)
BinSerialize.ReadString(ref readSpan);
}
public void Dispose()
{
this.binaryReader.Dispose();
this.binaryWriter.Dispose();
this.memoryStream.Dispose();
}
}| Method | Mean | Error | StdDev | Ratio | RatioSD | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|---|
| BinaryWriter_WriteReadString | 246.7 ms | 4.013 ms | 3.754 ms | 1.00 | 0.00 | 734000.0000 | - | - | 114.44 MB |
| BinSerialize_WriteReadString | 208.4 ms | 3.985 ms | 3.914 ms | 0.84 | 0.02 | 734000.0000 | - | - | 114.44 MB |
Added MIT license.