Skip to content

Instantly share code, notes, and snippets.

@raulk

raulk/notes.md

Last active July 8, 2019 19:01
Show Gist options
  • Save raulk/8db7e4cc7658ae9c9483aea1eed6b398 to your computer and use it in GitHub Desktop.
Save raulk/8db7e4cc7658ae9c9483aea1eed6b398 to your computer and use it in GitHub Desktop.
Download ZIP
research spike: stream closure in libp2p
Raw
notes.md

research doc: usable stream closure in libp2p

libp2p streams are full-duplex. Each party can read and write simultaneously on the conduit, and the underlying transport guarantees (or not) delivery, while the multiplexer also participates in congestion control. Yamux, for example, applies stream-scoped congestion control to curtail head-of-line blocking in some circumstances. Mplex is more simplistic and relies purely on TCP congestion control.

There's an active debate in go-libp2p to rethink the single Close() method on streams. The proposal is to make explicit which tip of the duplex is being shutdown via explicit CloseWrite() and CloseRead() methods.

These are notes I've gathered from researching how such closures are handled in other domains.

Connection close in TCP

In TCP, each party in the connection keeps read and write state independently (window). A FIN packet signals that the sending peer has no more data to write. The local stack of the sending peer will not accept any further sends after a closure, but receives are still allowed in this state. The connection is effectively in a half-closed state here.

The receiving stack will ACK the FIN immediately, but will not send a FIN of its own until the user has called close on the socket. The receiving end can still write data on the socket between ACK'ing the FIN and sending its own FIN.

This section of RFC793 (Transmission Control Protocol) sums up the choreography and expectations:

3.5. Closing a Connection

CLOSE is an operation meaning "I have no more data to send." The notion of closing a full-duplex connection is subject to ambiguous interpretation, of course, since it may not be obvious how to treat the receiving side of the connection. We have chosen to treat CLOSE in a simplex fashion. The user who CLOSEs may continue to RECEIVE until he is told that the other side has CLOSED also. Thus, a program could initiate several SENDs followed by a CLOSE, and then continue to RECEIVE until signaled that a RECEIVE failed because the other side has CLOSED. We assume that the TCP will signal a user, even if no RECEIVEs are outstanding, that the other side has closed, so the user can terminate his side gracefully. A TCP will reliably deliver all buffers SENT before the connection was CLOSED so a user who expects no data in return need only wait to hear the connection was CLOSED successfully to know that all his data was received at the destination TCP. Users must keep reading connections they close for sending until the TCP says no more data.

https://tools.ietf.org/html/rfc793

Conclusions:

  • close() in POSIX signals we have no more data to send, sending a FIN on the wire, and rejecting any calls to send() thereafter.
  • However, receive()s are still allowed until the local stack receives the other party’s FIN.
  • While that happens, TCP guarantees delivery of any packets in between.
  • The party receiving the unexpected FIN signals so to the local application, either by failing on receive() or by different means, e.g. epoll, kqueue.
  • The app can continue writing data, and TCP will guarantee its delivery, until it, in turn, calls close() which sends a FIN, and at that point the connection is closed both ways (once acked).

Go SDK

TCPConn has Close(), CloseRead() and CloseWrite() methods:

  • CloseRead() calls shutdown(SHUT_RD) (half-duplex close). It has no impact on the wire; it closes the file descriptor but in some systems it may be reopened if the remote sends more data.
  • CloseWrite() calls shutdown(SHUT_WR) (half-duplex close). It sends a FIN, and therefore the full connection termination sequence.
  • Close() calls close() and initiates a full-duplex active close.

https://github.com/golang/go/blob/f686a28/src/net/fd_unix.go#L182-L199 https://www.gnu.org/software/libc/manual/html_node/Closing-a-Socket.html

From The Linux Programming Interface (pp. 1273, "Calling shutdown() on a TCP Socket"):

The SHUT_RD operation can't be meaningfully used with TCP sockets. This is because most TCP implementations don't provide the expected behaviour for SHUT_RD, and the effect of SHUT_RD varies across implementations. On Linux and a few other implementations, following a SHUT_RD [...], a read() returns end-of-file, as we expect from the description of SHUT_RD [...]. However, if the peer application subsequently writes data on its socket, then it is possible to read that data on the local socket.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment