Skip to content

Instantly share code, notes, and snippets.

@MakMukhi

MakMukhi/Optimize_read_path.md

Last active February 28, 2019 04:09
Show Gist options
  • Save MakMukhi/968cfd41ea8318df8564c95c431329e9 to your computer and use it in GitHub Desktop.
Save MakMukhi/968cfd41ea8318df8564c95c431329e9 to your computer and use it in GitHub Desktop.
Download ZIP
gRPC-Go: Optimize read path
Raw
Optimize_read_path.md

gRPC-Go: Optimize read path

Status: (Final)

Authors: [email protected]

Reviewers: dfawley@, menghanl@, yuxuanli@, psrini@

Last Updated: 2018-02-25

Objective

Read path of stream can be optimized to get rid of several memory allocations and one memmove operation for every message.

Background

gRPC receives messages in form of one or more data frames. The framer API in Go is such that for every data frame received the bytes must be copied out. Currently, we resort to allocating a new byte slice( the size of data contained by a given frame) and copy bytes into that slice. This slice is then added to the recvBuffer of a stream. This recvBuffer can be looked at as an infinite buffer made out of a chan and a slice.

Read operation is such that the gRPC application layer first reads 5 bytes (length of a message header in gRPC message encoding) out of this recvBuffer. This header contains among other information the length of a following message. The application layer then allocates a byte slice of that length and passes it to the transport layer to fill it up.

For both these reads from application layer (header and message), the transport layer fills up the provided byte slice by copying data out of the recvBuffer. Therefore, for every message smaller than 16K (maximum size a data frame can be of). There will be two allocations(one by gRPC application layer and one by gRPC transport layer) and two memmove operations(data frame to recvBuffer and recvBuffer to application layer allocated memory). Note, that the memmove operations here do not account for the message header.

For messages that are greater than 16K, there will be several allocations done by the transport layer (equal to the number of data frames the message takes, for example a 1 MB message will take 1MB/16KB frames) and multiple memoves.

Overview

The allocation done by gRPC layer can be made redundant if the transport layer is made aware of gRPC's message encoding. The transport layer then, upon receiving a data frame, can read the first 5 bytes and decipher the length of byte slice that needs to be allocated, allocate it and start copying data into this memory. The read operation by gRPC application layer will now change such that instead of providing a memory to be filled it accepts memory from transport layer which is one message.

For messages smaller than 16KB this will lead to 1 less memory allocation (by the gRPC layer), and one less copy (from recvBuffer to gRPC layer allocated memory).

For messages bigger than 16KB this will save multiple (message size/ 16KB) allocations (by the transport layer for each data frame) and multiple copies (from recvBuffer to gRPC layer allocated memory).

Detailed Design

gRPC message encoding:

Every message after being serialized is added with another 5 bytes of header.

The first byte of this header contains the payload format of the message(i.e. if it's compressed or not.). The next 4 bytes contain the size of the message.

This message(header and message) is then passed to the transport layer which breaks it up into 16KB chunks, if need be, and creates a data frame for each and writes them on the wire.

On the receiving end, the transport layer reads each data frame and copies the bytes out it into a freshly allocated memory. This memory is then stowed away in the corresponding stream's data structure.

When the gRPC application layer makes a read call on the receiving end, it starts with making a read call of 5 bytes. Extracts the length of the following message from the last 4 bytes and allocates new memory. The next read call made by the gRPC application layer reads the message into this new memory.

Read and stream-level flow control:

Reading mechanism in gRPC also involves stream-level flow control. A read by the application layer may trigger window updates sent to the sender.

Owing to a previous optimization, currently every read operation by the application layer triggers a proactive window update by the transport to the receiving side. So for example, if a message worth 128KB were to be read by a stream but may not have been read yet or even received by the transport yet, the transport layer will still send a proactive window update for this message since the application has already committed to reading it. This is an important optimization should be preserved.

Data frame containing multiple messages:

Although, the current implementation doesn't do this, this isn't explicitly disallowed by the gRPC protocol. In fact, we should look into doing this an optimization in a future effort. This means that we can't expect every data frame to contain a full message header.

Implementation:

The stream data structure will be added with another method func (s *Stream) consume([]byte) which will take bytes from data frame and execute logic similar to the gRPC application layer. Special consideration must be taken to account for:

  1. Frame containing multiple messages.
  2. Frame containing partial header.
  3. And the obvious one, frame containing partial message.

When a complete message is received it will be added to the recvBuffer of this stream.

This method will be called by the reader goroutine on the client-side and HandleStreams goroutine on the server-side.

Application reads which will be executed by a stream's goroutine may occur either when:

  1. One or more messages have already been received. This is the simpler case in which the application is returned one message and a stream-level window update is sent for that message.

  2. No full message has been received. In this case we might further have 2 situations:

    1. A full header for the message has been received, in which case we can send a proactive window update of the size of the message that we are in the middle of receiving.
    2. A paritial header has been received, in this case the a flag on the stream data structure should be set, such that later when the whole header frame is received a proactive window update can be sent.

    When the complete message is received it is returned to the stream, which meanwhile waits on a chan like it does in the current implementation.

Caveats

Special attention should be paid to the following:

  1. Data frame containing multiple gRPC messages and possible partial message headers.
  2. Proactive window updates upon application reads.
  3. Synchronization between a stream's goroutine that calls read and reader/HandleStreams goroutine which calls consume must be kept minimum. The events that will require synchronization is: telling reader/HandleStreams goroutine that a stream has committed to reading, upon a read by a stream finding if a full header frame has been received and if so what's the length of the message currently being read for a window update needs to be sent out for it, updating window update related fields.
  4. This will result in an API change of transport.Stream. But since we consider this to be an internal API, we should be able to do this.

Testing Plan

The current transport and gRPC end to end tests should be sufficient to test this change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment