SCM_RIGHTS API quirks

SCM_RIGHTS.md

As tested on Linux:

• An SCM_RIGHTS ancillary message is "attached" to the range of data bytes sent in the same sendmsg() call.
• However, as always, recvmsg() calls on the receiving end don't necessarily map 1:1 to sendmsg() calls. Messages can be coalesced or split.
• The recvmsg() call that receives the first byte of the ancillary message's byte range also receives the ancillary message itself.
• To prevent multiple ancillary messages being delivered at once, the recvmsg() call that receives the ancillary data will be artifically limited to read no further than the last byte in the range, even if more data is available in the buffer after that byte, and even if that later data is not actually associated with any ancillary message.
• However, if the recvmsg() that received the first byte does not provide enough buffer space to read the whole message, the next recvmsg() will be allowed to read past the end of the mesage range and even into a new ancillary message's range, returning the ancillary data for the later message.
• Regular read()s will show the same pattern of potentially ending early even though they cannot receive ancillary messages at all. This can mess things up when using edge triggered I/O if you assumed that a short read() indicates no more data is available.
• A single SCM_RIGHTS message may contain up to SCM_MAX_FD (253) file descriptors.
• If the recvmsg() does not provide enough ancillary buffer space to fit the whole descriptor array, it will be truncated to fit, with the remaining descriptors being discarded and closed. You cannot split the list over multiple calls.

@ClosetMonkey The more I read about SCM_RIGHTS the more confused I become.

Me too.

@kentonv you MUST check if you received an SCM_RIGHTS message and, if so, close the file descriptors [...] You MUST check whether you received two and close the second one

Thanks for creating this gist! What I'm puzzled about: How do I know how many file descriptors I received?

The formula would essentially be the inverse of CMSG_LEN(), divided by sizeof(int). But I can't find a macro for this.

One possibility is to open up CMSG_LEN()'s definition, e.g. with the glibc header files the inverse would be len - CMSG_ALIGN (sizeof (struct cmsghdr)), whoops, there goes portability I'm afraid.

Another possibility is to guess the value in a loop (or binary search), passing the guesses to CMSG_LEN() and comparing them to the actual received ancillary data length.

Am I missing something?

@egmontkob Unfortunately, you do indeed need to divide the number of bytes that the kernel indicates you actually received, by the size of an int. Here's my code.

https://github.com/capnproto/capnproto/blob/7c8802fb9bec8818f289a44b0ec22419a845b249/c++/src/kj/async-io-unix.c++#L665-L680

@ClosetMonkey This is a very old interface and clearly it wouldn't pass muster by modern standards, but a lot of things are built on it now so it can't really go away. Instead we say, this is an old weird API and if you want to use it you'd better be careful.

I am a bit surprised that MacOS gets away with not closing the file descriptors for you, that feels like it might be a vulnerability of some sort, though I guess on single-user desktops, resource exhaustion vulnerabilities aren't considered to be a huge deal.

@kentonv Thanks for your response!

It's not the division I was worried about, rather having to subtract CMSG_ALIGN (sizeof (struct cmsghdr)) which looked presumably non-portable for me, I mean, is it guaranteed that that's what CMSG_LEN() adds to the payload length?

Your approach of writing CMSG_LEN(0) instead is definitely nicer, the nicest so far. You still rely on the (pretty reasonable) assumption that the function is linear with a slope of sizeof(int), but don't make an assumption on the overhead's size. Nice!

I really think there should be a macro doing this. I'm not sure if glibc / freebsd / etc. developers would be open to this idea; or if it should be brought up with POSIX / Austin Group or who else exactly.

recvmmsg (note the extra m) should ease the issue regarding recvmsg stopping at the end of the byte range. It's usually needed for performance anyway.

@vinipsmaker told me to share this tweet by Andreas Kling (Ladybug browser leader developer):

We are hitting this kernel bug with file descriptor passing on macOS/XNU and it's pretty terrible and it's known since 2011!?
The obvious workaround is to use mach_msg and port rights, which is what WebKit does, and what we will do as well. But.. uhh..
https://openradar.me/9477351

https://x.com/awesomekling/status/1846424613951099317
https://xcancel.com/awesomekling/status/1846424613951099317