NNNN-cross-import-overlays.md

Introduce cross-import overlays to factor out cross-cutting APIs

• Proposal: SE-NNNN
• Authors: Becca Royal-Gordon, Varun Gandhi
• Review Manager: TBD
• Implementation: In master behind -Xfrontend -enable-cross-import-overlays
• Status: Prototyped behind a feature flag

Introduction

Cross-import overlays allow Swift to automatically import additional “overlay” modules based on the combination of imports in a particular source file. They allow one library or framework to seamlessly offer tailored APIs for interoperating with another, without imposing additional dependencies or code on clients who don’t need it.

Swift-evolution threads: pitch 1, pitch 2

Motivation

Felicity is a framework engineer for the (hypothetical) hot new ride-sharing startup, Karr™. Karr uses Swift pervasively across all platforms, and it gives its teams wide latitude to choose their own styles and dependencies, so there’s a lot of diversity in the frameworks and packages they use. For example, some teams write in a more imperative style; others use RxSwift; still others use Combine. Some backend teams use Vapor; others use Kitura; others use raw NIO. And the list goes on.

This freedom causes endless headaches for Felicity. Her job is to maintain KarrKit, a shared framework which encapsulates a lot of business logic and glue code, and which has been growing endlessly as each team contributes support for their pet toolkit. Development compiles have become opportunities for coffee breaks; WMO release builds are now often run overnight. CPU cores in Karr’s build cluster are sitting idle while they wait for KarrKit to build before anything else can. The framework is now taking up a hundred megabytes on disk. And clients of KarrKit are suffering for this too—just importing all of this code, most of which they don’t use, slows down their own compiles.

Felicity could address this by breaking up KarrKit into individual modules. Each module would be much smaller, take much less time to build, could be built in parallel, and could load much faster and only when needed. But Karr has a lot of existing code with an import KarrKit at the top of the file; breaking up KarrKit would require a company-wide effort to import the right set of KarrKit modules for each file. There’s no way to let the compiler automatically figure out which subset of modules each file needs.

Right?

——

The story of KarrKit is perhaps overly dramatic, but it demonstrates a basic truth: adding support for framework A to framework B can impose substantial technical costs on the builds of both framework B and its clients, which need to build and load the extra code. And yet splitting that support into a separate framework AB imposes cognitive costs on B and its clients, as they need to discover these modules, teach and learn about them, separately document them, etc.

One place where we already acknowledge this need because it’s so stark is in testing. Swift supports separate test modules, and SwiftPM supports separate test-only dependencies, because you often need things in testing that you don’t need in production. But even here, our support is incomplete: You ought to be able to write test helpers (like custom XCTAssert*() functions or pre-made mock types) that automatically become available when you import one of your dependent frameworks with XCTest or your chosen test framework. Today, test support is either built into the same module and shipped to end users, or it’s in a separate framework that you have to import explicitly.

Another place Apple specifically ran into this problem is with SwiftUI support. It has now been announced that macOS 11, iOS/tvOS 14, and watchOS 7 will offer SwiftUI views for specialized tasks like displaying maps, SpriteKit scenes, and Sign In With Apple buttons. But there’s no good place to implement these views in current Swift. If they’re placed in new frameworks, this would create a “parallel universe” of SwiftUI frameworks (e.g. import MapKit for AppKit/UIKit, but import MapUI for SwiftUI), . If they’re included in SwiftUI, the framework will, over time, become bloated with specialty views that most users don’t use. But if they’re included directly in individual frameworks like MapKit and SpriteKit, then these frameworks would have to start importing and depending upon SwiftUI. This not only burdens UIKit/AppKit-only apps with unnecessary SwiftUI support, it also posed serious difficulties on watchOS.[1]

But this is a more general problem than just testing and SwiftUI. It’s why ReactiveSwift, ReactiveCocoa, and ReactiveMapKit are three separate things. It’s why you need to import GRDBCombine separately from import GRDB. It’s why SwiftyJSON has a separate Alamofire-SwiftyJSON module. It’s why you have to think at all about half of the projects published under the RxSwiftCommunity organization. Solving this problem for the package ecosystem will require SwiftPM support—which is out of scope for a Swift language evolution proposal like this one—but the package manager can’t tackle this problem until the language has a way to solve it.

But today, the only alternative is to import and support all possible dependencies in a single, monolithic module. There is no way to get the usability of a monolithic module with the nice factoring of separate modules.

[1]

For example, on watchOS, WatchKit imports MapKit for WKInterfaceMap, and SwiftUI imports WatchKit to interoperate with WKInterface; if MapKit tried to import SwiftUI to offer a SwiftUI view, this would create a circular dependency where none of the modules could be built without having already built the others. There were, in fact, several different modules with this problem, and we saw no straightforward way to solve it without at least some source compatibility breaks.

Proposed solution

We propose allowing Swift and Clang modules (frameworks and libraries) to declare cross-import overlay modules. A cross-import overlay module wraps its declaring module, re-exporting the declaring module’s contents but also adding extra APIs to it. The compiler imports the cross-import overlay module in place of the declaring module only when an additional bystanding module is imported into the same file. The APIs added to the cross-import overlay will typically relate to the bystanding module, but there is no technical requirement forcing this.

For instance, KarrKit (declaring module) could declare a cross-import with Combine (bystanding module) which would import _KarrKit_Combine (cross-import overlay module). If it did, the compiler would essentially rewrite this:

import KarrKit
import Combine

Into this:

import _KarrKit_Combine
import Combine

While making KarrKit an alias for _KarrKit_Combine in that source file.

Declaring a cross-import overlay

The declaring module declares a cross-import overlay by placing a special YAML file at a particular location in their framework bundle or, for libraries, next to the directory or file that defines the library’s module. For instance, a cross-import overlay for KarrKit and Combine would be declared by a YAML file at:

<directory where framework is installed>/
    KarrKit.framework/
        Modules/
            KarrKit.swiftcrossimport/
                Combine.swiftoverlay

With the following contents:

—
version: 1
modules:
- name: _KarrKit_Combine

Because these files must be installed into the framework bundle or next to the library, third-party developers cannot make system modules declare cross-import overlays without modifying the SDK. In essence, this means that the declaring framework’s distributor controls which cross-import overlays it has.

Requirements for the bystanding module

The bystanding module is a completely passive participant in the cross-import. It can be any module in any search path, it does not contain any files declaring the cross-import overlay, and its author is not necessarily aware that the cross-import overlay exists. Third-party developers can use system modules from the SDK as bystanding modules.

Requirements for the cross-import overlay module

The cross-import overlay module is in most respects an ordinary Swift module. It has its own module name, can be a framework or library, and can be installed in any directory in the search paths. (This contrasts with the existing “Swift overlays” that add Swift functionality to system C/Objective-C frameworks; those share the name of their underlying module, must be installed in particular places, and take advantage of special module loading quirks to wrap Clang modules specifically).

To function correctly, a cross-import overlay module should use an @_exported import of the declaring module and a regular import of the bystanding module:

// _KarrKit_Combine cross-import overlay

@_exported import KarrKit
import Combine

// Add new public APIs here

To hide the cross-import overlay from autocompletion, its module name should begin with an _. We recommend using _<declaring module>_<bystanding module> as the overlay name, but this is just a convention—the compiler doesn’t ascribe any special meaning to this name.

Compiler cross-importing behavior

For each source file, the compiler computes the set of modules that are visible in that file, including via @_exported imports from Swift modules. Then it looks for combinations of these modules that have cross-import overlays. If it finds any, it implicitly imports those cross-import overlays.

This set includes Objective-C modules imported directly by Swift modules, but does not include additional modules imported by those Objective-C modules. (Including transitive Objective-C imports would tend to cause unexpected cross-import overlays to be imported.)

Name lookup behavior

Scoped imports and qualified lookups for the declaring module match declarations in any cross-import overlays loaded by that file. For instance, both of these code samples are valid even if RideEventPublisher is declared in _KarrKit_Combine:

import class KarrKit.RideEventPublisher
import Combine

import KarrKit
import Combine

func fn(_: KarrKit.RideEventPublisher) { ... }

If a cross-import overlay module and its declaring module both declare the same name with the same overload signature, the overlay’s declaration shadows the declaring module’s.

Detailed design

Cross-import overlay declaration format

Only top-level modules can be the declaring, bystanding, or overlay module of a cross-import overlay. Where we refer to a “module name” in this discussion, we mean a top-level module name, not a submodule name.

When loading a top-level module from disk, the compiler determines its module definition path. It then looks for a swiftcrossimport directory at a sibling path to the module definition path. If it finds the swiftcrossimport directory, it then looks for swiftoverlay files at certain paths inside it. These files declare the module’s cross-import overlays.

Module definition path

The module definition path is the path to the file or directory on disk which contains the definition for a given module.

For Clang (C and Objective-C) modules, the module definition path is the path to the “modulemap” file which declares the module. If there is no “modulemap” file which explicitly declares the module, the module does not have a module definition path and cannot declare any cross-import overlays.

include/
    KarrKit.h
    module.modulemap    <= module definition path

For “fat” Swift modules (i.e. ones with a “swiftmodule” directory directly in the search path which contains different files for each platform), the module definition path is the path to the “swiftmodule” directory.

lib/
    swift/
        KarrKit.swiftmodule/    <= module definition path
            x86_64-apple-macos.swiftinterface
            x86_64-apple-macos.swiftdoc

For “thin” Swift modules (i.e. ones with a set of “swiftmodule”, “swiftinterface”, and “swiftdoc” files directly in the search path), the module definition path is the path to the “swiftinterface” or “swiftmodule” file for the module.

lib/
    swift/
        KarrKit.swiftmodule    <= module definition path
        KarrKit.swiftdoc

Note that, for either a Swift framework or a Clang framework with an explicit module map, the module definition path will always be directly inside the “Modules” directory of the framework bundle. This is a happy coincidence, not a special case.

The swiftcrossimport directory

Once the module definition path is computed, the last path component is removed. Then, within that directory, the compiler looks (non-recursively) for entries matching:

1. <declaring module name>.swiftcrossimport/*.swiftoverlay
2. <declaring module name>.swiftcrossimport/<target module triple>/*.swiftoverlay
3. <declaring module name>.swiftcrossimport/<target-variant module triple>/*.swiftoverlay, if a target-variant has been specified

The target module triple and target-variant module triple are derived from the -target and -target-variant command-line switches, transformed into the ”normalized module triple” format used by the “fat” swiftmodule format.[1] On ABI-stable platforms, this normalization removes OS version numbers and maps synonyms to canonical forms to produce a single, known name for that stable ABI.

[1] There is no proposal documenting this normalization, but the swift::getTargetSpecificModuleTriple() implementation in the compiler’s lib/Basic/Platform.cpp is intended to be self-documenting.)

The swiftoverlay files

Within each of the three directories searched in the swiftcrossimport directory, directory entries whose names end in .swiftoverlay are processed as cross-import overlay declarations. The base name of the file is computed and taken to be the name of the bystanding module.

If necessary, the compiler parses these files to determine the names of the overlay modules. Each file must be in YAML 1.2 format and must contain a top-level mapping with the following keys:

• version: A file format version number. In this proposal, all swiftoverlay files should be given version 1, and the compiler will reject swiftoverlay files whose version is not in the range 1.0 ..< 2.0.
• modules: A sequence of zero or more mappings, each containing the following keys:
  • name: The name of a top-level module to be loaded as a cross-import overlay.

If there is more than one module mapping in modules, all of the modules listed are cross-import overlays and will be imported.

Edge cases

• swiftcrossimport directories whose base name is not that of a top-level module, or which are not located in the same directory as that module’s module definition path, will be ignored. (For instance, a directory at AppKit.framework/Modules/Foundation.swiftcrossimport will never be used, since AppKit.framework/Modules/module.modulemap does not define the Foundation module.)

• Within the directories searched in a swiftcrossimport directory, directory entries which do not end with .swiftoverlay are ignored. Entries which end with .swiftoverlay but are not readable files are invalid.

• Within a swiftoverlay file, unrecognized keys are ignored. Files with no version or modules keys, or modules with no name key, are invalid.

• If swiftoverlay files with the same name are present in both the swiftcrossimport directory itself and the module-triple-specific subdirectories, the compiler takes the union of all of their “modules” lists.

• The compiler doesn’t eagerly load swiftoverlay files, so it may not diagnose invalid files if it doesn’t need to read them.

Cross-importing

After a file’s import declarations have been processed, but before any scoped imports are validated, we will insert an additional step, called cross-importing.

Cross-importing is an iterative algorithm. Its steps are (italicized terms are defined in more detail below):

1. Compute the cross-import set of the file (i.e. the set of all modules visible from this file which participate in cross-importing).

2. Consider each pair of modules in the cross-import set. If the first declares one or more cross-import overlays with the second as bystander:

   1. Load the list of cross-import overlay modules for that pair of modules.
   2. Register each overlay module in the source file’s cross-import overlay lookup table.
   3. Add an implicit overlay import to the source file for any of the overlay modules which have not been imported.

3. If any cross-import overlay modules were implicitly imported, the cross-import set has changed; repeat all steps.

Cross-import sets

For our purposes, an “import” is a tuple of (imported-module, import-attributes, import-scope).

The cross-import set is a set of imports which includes:

1. The list of top-level modules imported into a given source file (with substitutions based on the cross-import overlay lookup table—see that section below), along with the attributes and scopes of the imports;
2. The set of top-level modules @_exported imported in other source files in the same module (with substitutions based on the cross-import overlay lookup table), along with the attributes and scopes of the imports;
3. The set of implicit imports for the current module (e.g. Swift), with default attributes and scopes; and
4. The set of top-level modules @_exported imported by each Swift module in the cross-import set (recursively), with the attributes and scopes of the import from #1–#3 that caused them to be included in the transitive import set.

Imports with the @testable or @_private attributes are not included in the cross-import set; nor are their transitive imports.

Note that the modules re-exported by a Swift module are included in the cross-import set, but the modules re-exported by an Objective-C module are not.[2] For example, a Swift file containing import UIKit does not import cross-import overlays for CoreAnimation, even though several UIKit headers import CoreAnimation headers. You would need to explicitly write import CoreAnimation to get its cross-import overlays.

[2] Rationale: C and Objective-C modules re-export everything they import, so in practice, including them in the cross-import set would make the set very large with many unexpected entries. Swift @_exported imports are much rarer and are most often used by overlays which should be transparent to the cross-import logic.

Cross-import overlay lookup table

Whenever both the declaring and bystanding module of a cross-import overlay are in a source file’s cross-import set, the overlay module is registered in that source file’s cross-import overlay lookup table.

This table maps a declaring module to a list of overlay modules which are “on top” of it in that file. Registering a cross-import overlay has two effects:

1. Any imports of one of the declaring modules directly in the source file are disabled, so the compiler behaves as though the declaring module was not directly imported into that source file. (Overlays should @_exported import the declaring module, so it should still be transitively imported and included in the cross-import set.)

2. Name lookups from that source file into the underlying module will be dispatched to the overlay modules first, and will see the declaring module’s declaration only through the overlay modules’ re-exports. (That is, in a source file which has both KarrKit and Combine in its cross-import set, even an explicit, qualified reference to KarrKit.RideEventPublisher should look in _KarrKit_Combine first.)

Implicit overlay imports

When the compiler discovers a pair of imports in the transitive import set that should load a cross-import overlay, it computes a new import of the overlay according to the following rules:

• The scope of the import is the same as the scope of the declaring module’s import.
• The import is @_exported if both the declaring and bystanding imports are @_exported.
• The import is @_implementationOnly if either the declaring or bystanding imports are @_implementationOnly.

If this import is not in the cross-import set, it then adds the import to the source file’s import list for that overlay, as though the user wrote such an import.

Edge cases

• The source file’s parent module is never included in the cross-import set. That is, a module never tries to cross-import overlays which it declares or bystands.

• If the source file’s parent module is in any cross-import overlay lists, it is removed from those lists. That is, even though a cross-import overlay module imports the declaring and bystanding modules, it never tries to import itself.

• Submodule imports are treated as though they were imports of the top-level module’s Swift overlay, or if there is none, its Clang module.

• Because Swift overlays @_exported import the underlying Clang module, both the overlay and underlying modules will be included in the cross-import set.

• Module interface (swiftmodule) files are essentially “pre-cross-imported”:

  • When emitting a module interface, the compiler emits explicit import declarations for each cross-import overlay, leaves out any import declarations for their declaring modules, and qualifies types located in an overlay with the overlay’s name.
  • When loading a module interface (swiftinterface file), the compiler skips the cross-importing step.

• A cross-import overlay can be the declaring or bystanding module of another cross-import overlay; this can be leveraged to effectively create a cross-import overlay which requires more than one bystanding module to be loaded.

Tooling

Currently, SourceKit hides modules whose names start with _ from code completion. When a cross-import overlay module’s name starts with an _, SourceKit will generally behave as though its declarations are part of the overlay’s declaring module. For example, the generated interface of a module will also include declarations from the cross-import overlays the module declares. Comments around these declarations will explain which modules need to be imported to make them visible.

Requirements for cross-import overlay modules

Cross-import overlay modules must:

• Use an @_exported import of their declaring module. (Otherwise, name lookup and transitive cross-importing of the declaring module may break down.)
• Have a distinct module name from their declaring module. (Otherwise, one of them will shadow the other.)
• Be written in Swift. (Otherwise, name lookup behavior may be negatively impacted.)

Implementation guidelines for cross-import overlay modules

Cross-import overlay modules should:

• Have a module name which starts with an _, and preferably have a module name of the form _<declaring module name>_<bystanding module name>. (Otherwise, the cross-import overlay will be more visible.)
• Not @_exported import the bystanding module or any other modules. (Otherwise, the declarations of those modules will also be accessible when qualifying with the declaring module’s name.)
• Contain only APIs which pertain to both the declaring and bystanding module; generally useful APIs should be moved to the declaring module. (Otherwise, these APIs will be unnecessarily locked away unless an irrelevant second module is also imported.)

These recommendations are not mandatory, but cross-import overlays work best when they are followed.

Source compatibility

By itself, this feature does not affect source compatibility. However, it is possible that some cross-import overlays will declare APIs which conflict with other APIs from explicitly-imported modules. We do not usually consider such conflicts to violate Swift’s source stability promises.

Effect on ABI stability

This change does not affect the ABI of existing code.

Effect on API resilience

The declarations in a cross-import overlay are mangled with the overlay module’s name, not the declaring module’s name. This means that cross-import overlays have the following resilience rules:

• Renaming a cross-import overlay module is ABI-breaking.

• Moving a declaration from a declaring module to one of its cross-import overlays is ABI-breaking.

• Moving a declaration from a cross-import overlay module to its declaring module is ABI-breaking. (However, the compiler-internal @_originallyImportedIn attribute could be used to make this change safely.)

• Modifying the contents of a .swiftcrossimport directory is not ABI-breaking, but may break source compatibility.

Alternatives considered

Conditional imports

At first glance, it might seem like the goals of this proposal could be accomplished with some kind of “conditional imports” feature:

// This class is only available to clients who have also imported Combine.
class RideEventPublisher: Publisher where import Combine {
  ...
}

This would work for very simple cases, but we believe it would break down in complex ones with dependencies between the modules. If you wanted to put an API into LibraryA with where import LibraryB, but LibraryB imported—even as an implementation detail—LibraryA, this would form a circular dependency, making it impossible to build either module without having already built the other. Moving the declaration to the other module would remove this technical barrier, but this location may not fit the user’s mental model, making it hard to locate documentation. It can also create large, monolithic modules with lots of loosely-related functionality grouped together, creating both technical challenges (long compile times) and organizational challenges (many teams stepping on each others’ toes in a single module).

We think that placing the cross-cutting declarations in a separate module creates a much more scalable solution. Ultimately, if you have a case simple enough for a hypothetical conditional import feature to handle, you can probably just always import the dependency and always offer the APIs that use it. If you have enough cross-cutting APIs for that solution to start chafing, you probably have enough to need a separate module anyway.

Express the overlay relationship differently

The directory of files the compiler reads to discover cross-import overlays seems like an awkward choice, but we settled on this solution because it has a lot of desirable traits. To explain, let’s look at some specific alternatives we rejected:

1. Use a naming convention alone, e.g. _DeclaringLibrary_BystandingLibrary. To determine if a pair of modules had a cross-import overlay, the compiler would need to either speculatively stat O(n^2 * k) paths where cross-import overlays could potentially reside, or enumerate each directory in the include paths to build a table of available modules. Neither solution would have acceptable performance.

2. Put information about the cross-import overlay in the overlay itself. The compiler would not be able to tell which overlays it would need to load based only on the visible imports, so it would need to discover and load cross-import information from all overlays, with performance penalties similar to a naming convention.

3. List all cross-import overlays in a per-import-path or global file. This would make it unnecessarily difficult to collect several modules with cross-import overlays into a single directory—you would need to use a tool to combine the contents of the files. (A global file would effectively restrict cross-import overlays to the vendor only.)

4. Put information about the cross-import overlays in the serialized module file itself. Retroactively adding a cross-import overlay to a module built without knowledge of it would require use of a custom tool to edit the module file. The present solution requires only the ability to create directories and copy files—things build and install systems know how to do. It also allows Clang modules to declare cross-import overlays.

5. List all of the cross-import overlays declared by a single module in one file. Retroactively adding two cross-import overlays to a module would require combining the files with a special tool. Again, the present solution allows us to install overlays simply by copying files.

6. Put the cross-import files inside the .swiftmodule directory. Clang modules and thin Swift modules would need a different path. The present solution always puts this information at the same path inside a framework or alongside a library.

7. Use a file format other than YAML. We want a format which has some structure to allow for future expansion (not just a text file containing a module name), which can be viewed and edited with a text editor (not binary), which supports comments (not JSON), and which won’t require a lot of unnecessary design or implementation effort (simple and LLVM-supported). YAML fits the bill.

By contrast, in the design we propose:

• Cross-import information is essentially sharded by declaring module, and the cross-import set can be computed incrementally, so the compiler only needs to build tables of information involving the modules it actually loads.
• Swift and Clang modules declare cross-import overlays in the same format and often at the same paths.
• No special tooling is required to write, inspect, update, or install cross-import overlay metadata, even if information about cross-imports with several different bystanders needs to be installed into a single cross-import overlay.

As ungainly as it is, the design we settled on is much more practical than a lot of “pure” alternatives.

Future directions

Add SwiftPM support to this proposal

While this proposal alone is enough to solve the use case of SwiftUI support in Apple’s frameworks, extending it to third-party code—and particularly to the package ecosystem—will require support from the Swift Package Manager as well. As a thumbnail sketch, the package manager would need to:

• Allow packages to provide cross-import overlays with particular library products as declaring modules and other library products or system modules as bystanders.
• Generate the necessary metadata and add it to the declaring modules.
• Compute from a target’s declared list of dependencies which cross-import overlays it might need from those available in the package graph, and implicitly add them to the dependency list.

SwiftPM has a separate evolution process from the Swift language, so it is not appropriate to include SwiftPM support directly in this proposal, but we think it’s a good idea and will work with SwiftPM’s maintainers to prepare a proposal on SwiftPM support.