SCM_RIGHTS API quirks

SCM_RIGHTS.md

As tested on Linux:

• An SCM_RIGHTS ancillary message is "attached" to the range of data bytes sent in the same sendmsg() call.
• However, as always, recvmsg() calls on the receiving end don't necessarily map 1:1 to sendmsg() calls. Messages can be coalesced or split.
• The recvmsg() call that receives the first byte of the ancillary message's byte range also receives the ancillary message itself.
• To prevent multiple ancillary messages being delivered at once, the recvmsg() call that receives the ancillary data will be artifically limited to read no further than the last byte in the range, even if more data is available in the buffer after that byte, and even if that later data is not actually associated with any ancillary message.
• However, if the recvmsg() that received the first byte does not provide enough buffer space to read the whole message, the next recvmsg() will be allowed to read past the end of the mesage range and even into a new ancillary message's range, returning the ancillary data for the later message.
• Regular read()s will show the same pattern of potentially ending early even though they cannot receive ancillary messages at all. This can mess things up when using edge triggered I/O if you assumed that a short read() indicates no more data is available.
• A single SCM_RIGHTS message may contain up to SCM_MAX_FD (253) file descriptors.
• If the recvmsg() does not provide enough ancillary buffer space to fit the whole descriptor array, it will be truncated to fit, with the remaining descriptors being discarded and closed. You cannot split the list over multiple calls.

To add two interesting cases:

1. Suppose you sendmsg() twice, both 5 bytes - first message with FDs, second without. You then call recvmsg() asking for 10 bytes. On Linux, you will receive 5 bytes. On BSD (probably all of them including XNU, though I haven't tried any but Free and Net, but the socket logic is much the same between them) you will receive 10 bytes.
2. Suppose you sendmsg() twice, both 5 bytes, first message without FDs, second message with FDs. Then if you recvmsg() asking for 10 bytes, on Linux you'll receive 10 bytes and you'll get the FDs with that, but on BSD you'll get 5 bytes without the FDs.

So Linux will limit what you receive to the end of the first message that has ancillary data while BSD will limit what you receive by the beginning of the next message that has ancillary data.