libc_portability.md

Improve Libc Portability

• Proposal: SE-NNNN
• Authors: Zac Bowling (Google)
• Review Manager: TBD
• Status: Awaiting implementation

During the review process, add the following fields as needed:

• Implementation: apple/swift#NNNNN
• Decision Notes: Rationale, Additional Commentary
• Bugs: SR-NNNN, SR-MMMM
• Previous Revision: 1
• Previous Proposal: SE-XXXX

Introduction

There are number of inconsistencies and issues with how you access libc across each platform within Swift.

libc is currently accessible either by the native Darwin module on Darwin platforms or the Glibc module for all other platforms. These modules provide nearly the same API across each platform but are exposed under different names. This situation thus requires you to gate importing each module on some type of platform check which. More often #if os(Linux) is used which not inclusive of all possible platforms the code will likely run on. This check is also insufficient for checking the specific libc you are targeting has the features you may want at compile time.

The Glibc swift module is currently compiled against the libc on the machine that built the toolchain which may not necessarily match the libc on the machine where the toolchain runs or the the target sysroot/SDK you point your swiftc at compile time. In contrast, the Darwin module is always up-to-date with the target SDK since the Darwin modulemap is included inside the SDK folder.

A side effect of the Glibc module being compiled ahead of time is that it bakes in absolute paths of the system to the system headers which breaks when cross-compiling against relocatable SDK paths.

In some libc implementations, some features provided are hidden behind feature macros that have to be defined before before you include the libc headers when you use them. Some of these macros can even change function signatures and behaviors of libc functions. (See feature_test_macros(7) on Glibc).

The Glibc module name is also unfortunately named for users on FreeBSD, Android, Haiku, and Fuchsia where the libc is not specifically GNU's glibc but some other libc. The module name is also incorrect on Linux when using another libc like musl, dietlibc, or uClibc instead of GNU's libc.

Swift-evolution thread: Discussion thread topic for that proposal

Motivation

We want to smooth over some of these inconsistencies that make Swift code less portable when consuming libc in Swift. We want to leave some backward compatibility with code that makes use of the Darwin and Glibc modules today but encourage developers to migrate code to the solution proposed here.

Proposed solution

Create a PlatformLibC module

We create a PlatformLibC module that replaces the Darwin and Glibc. This module would simply re-export the symbols from both of these modules. Developers should be able to include this module to get all portable libc methods they need without platform checks.

JIT the (G)libc module against the target SDK at compile time

Instead of compiling the Glibc module as part of the stdlib, we move the module to compile on-demand for each target SDK passed into Swiftc. We can also update the paths used by the importer as the SDK base for the module map. There is some additional glue code, outside of the just the Glibc.modulemap, that would have to be compiled and cached for each non-Darwin target.

Since we are JITing the module, we allow developer to be able to set feature flags on the libc module that the clang importer will set when compiling against the target's libc headers. By default we can implicitly set sensible defaults based on the target's triple (like _GNU_SOURCE on Linux libcs).

Detailed design

TODO: We write some code that does the things

Proposed design:

1. Create a PlatformLibC module that re-exports Darwin or Glibc for each arch.
2. Move the glibc.modulemap and any extra glibc swift glue code from the platform stdlib package into the lib/swift folder so that it can be compiled for the target SDK on demand.
3. Update the compiler frontend to compile and cache the libc module for us for each new -sdk passed in
   1. swiftc should apply the same kind of transforms we do in gyb (maybe we can use the standard C++ processor on the file) on the glibc.modulemap
   2. compile the rest of the glibc module with the extra glue code and store /tmp like the module cache.
   3. add the cached new glibc module from /tmp to the linker options for specific toolchains jitting this module.
   4. Add args to the swiftc frontend for defining libc define options for the SDK

Source compatibility

Darwin should be unaffected.

Unfortunately the current glibc module on Linux (or anything else) is already inherently brittle given it's being based on the libc of whomever built the Glibc module into the toolchain. The glibc module should remain compatibile for non-Darwin developers that have previously built their own toolchain for their own system. Developers that rely on the glibc module to behave like it did from a foreign system's libc different enough than their own (without obtaining and passing in that same foreign system's sysroot into the -sdk arg) may experience some source breakage. This might be people who obtained a pre-built toolchain from Swift.org for example and ran it on a different OS than it was compiled on. Any breakage suffered is likely just exposing undetected errors in the target's build.

Effect on ABI stability

Each libc implementation has it's own ABI stability guarantees and this proposal doesn't change that.

The ABI stability of any targeted libc is entirely the responsibility of the developer consuming libc on their target platform to check and ensure that the methods and features they are using from their libc is ABI stable for all the targets they want to deploy their binaries on.

Effect on API resilience

This adds a new public API (the PlatformLibC module) and deprecates usage of the Darwin and Glibc modules. This modules API is entirely determined by the target platform's libc and any additional code that is added to the glue code we have added to build the module.

Alternatives considered

Instead of JITing the Glibc module, implementing a VFS layer for the clang importer was considered so that we could relocate the target sysroot for SDK for the glibc module map. This would require changes in clang and would still not solve being able to update the glibc module to match the target's libc features.