Skip to content

Instantly share code, notes, and snippets.

@tbg
Last active December 14, 2022 20:22
Show Gist options
  • Save tbg/02cd5d7148c9918193a48fe23ef748dd to your computer and use it in GitHub Desktop.
Save tbg/02cd5d7148c9918193a48fe23ef748dd to your computer and use it in GitHub Desktop.
WriteBatch memcpy

WriteBatch gets created internally in a storage.Batch // instance 1

https://github.com/cockroachdb/cockroach/blob/f089b43803b1d59d4752238e5b4bb37cfe0fbf8d/pkg/kv/kvserver/replica_proposal.go#L713-L717

The impl actually copies, womp womp (the batch reuses the original memory but still): // instance 2

https://github.com/cockroachdb/cockroach/blob/5bafe901cd6fd4ceee00ee922e7f13ed95c4134b/pkg/storage/pebble_batch.go#L567-L576

Aside: If we kept the batch around with the proposal, we could conceivably use its backing memory. We could also apply the batch directly, possibly saving overhead on the leaseholder.

This is now sitting on a kvserverpb.RaftCommand:

https://github.com/cockroachdb/cockroach/blob/f089b43803b1d59d4752238e5b4bb37cfe0fbf8d/pkg/kv/kvserver/replica_proposal.go#L761-L766

We then marshal the RaftCommand, i.e. allocate and write: // instance 3

https://github.com/cockroachdb/cockroach/blob/97054a0e76049cd8f78d8b534ab1e2107be9f2ed/pkg/kv/kvserver/replica_raft.go#L437-L447

But we're not done yet! Because raft needs a raftpb.Entry:

https://github.com/cockroachdb/cockroach/blob/6ed91486435892122ba1fb9f1bc64cc339f160c3/pkg/kv/kvserver/replica_proposal_buf.go#L534-L536

When we've given this to raft and it hands it back to us for appending to the log, we need to marshal this entry:

https://github.com/cockroachdb/cockroach/blob/3d8819a85e9b1c668031e7ad294acae9944730bc/pkg/kv/kvserver/logstore/logstore.go#L237-L241 // instance 4

but we're not done there, because we'll write this via an inline put, i.e. the data goes here

https://github.com/cockroachdb/cockroach/blob/3e927d23bdc321ced8f860c2f1f8187c92d25000/pkg/storage/mvcc.go#L1714-L1715

and we still need to marshal that: // instance 5

https://github.com/cockroachdb/cockroach/blob/3e927d23bdc321ced8f860c2f1f8187c92d25000/pkg/storage/mvcc.go#L1275-L1289

(No allocation here usually - it's pooled - but we're still copying it around).

Are done yet? No! We have to send the entry to followers as well. It will be contained in a message here (concretely, in the Entries slice of a raftpb.Message within:

https://github.com/cockroachdb/cockroach/blob/4490ef18fbbba2158619acd1062e234dd8447536/pkg/kv/kvserver/raft_transport.go#L442-L441

and of course to be send over the wire it will need to be marshaled again (inside of gRPC), and the result copied into a network buffer. // instance 6+7

Also, we usually have two followers, not just one. // instance 8+9

We haven't applied the command yet... that will need to happen too!

https://github.com/cockroachdb/cockroach/blob/b57ea75ab556c37739539087bb664b9721cb817f/pkg/kv/kvserver/replica_application_decoder.go#L65-L74

calls through to unmarshal: // instance 10

https://github.com/cockroachdb/cockroach/blob/37326191e4bce7afe83ea6d9b4aaf35bc6bf4d6f/pkg/kv/kvserver/raftlog/entry.go#L133-L132.

There are additional copies if the raftentry cache forces us to ever fetch the entry from disk, since then we need to unmarshal first the raftpb.Entry and then from it the kvserverpb.RaftCommand.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment