Scala 3 introduces a new macro system that replaces the experimental Scala 2 system with one that should avoid unsoundness, be simpler to use, and simpler for the compiler team to evolve and maintain.
Perhaps the most significant feature missing from Scala 3 today is the lack of support for expressive compile-time metaprogramming. By this, I mean the ability to generate and/or reshape classes, traits, and objects at compile-time. In Scala 2, the community used macro annotations for this purpose. Scala 3 addressed a few of the use-cases of macro annotations, but we still lack a feature that offers anything close to the full range of expressive capability of macro annotations.
This document proposes a replacement for Scala 2 macro annotations that should:
- offer nearly same expressive power,
- address all, or most, major use cases,
- be simpler to maintain, and
- be safer and easier to use.
On the implementation side this replacement should:
- be integrated with the compiler (instead of a plugin),
- be simpler to maintain,
- expose no compiler internals, and
- re-use the existing macro system and its semantics as much as possible.
Most importantly, this feature should fit cleanly into the existing Scala 3 metaprogramming model and existing language features.
There are two key design principles this feature should follow:
- User code as written must not be altered, unless the code is explicitly marked as modifiable.
- Nothing extraneous should be required to be generated by a macro.
Due to various limitations, macro annotations in Scala 2 are unable to adhere to either design principle.
Generally, macro annotations are hard for authors to make correct, difficult for
users to introspect, and effectively unrestricted in their capabilities. This is
because at their core they are simply a function of AST => AST. This means
they can:
- alter the inheritance hierarchy of a class/trait/object,
- change order, types, or names of class parameters,
- modify generic type parameters,
- and more.
Instead, this proposal seeks to cleanly integrate with the more restrictive metaprogramming model in Scala 3. Features would be limited to:
- adding new declarations (fields, methods, types, classes, etc) at compile-time
- into existing named traits/classes/objects/packages
- as a function of:
- content of the enclosing trait/class/object
- static parameters
- overriding existing declarations when explicitly allowed.
By extending the semantics of import and export we can achieve most of the
stated design goals.
Today import and export have the exact same syntax, but dual functionality.
Both take a path to a stable identifier followed by selectors that determine
which symbols should be processed and how. The syntax could be extended to allow
import and export to accept a path to a macro and along with static
arguments to the macro, instantiate the macro, and then process the resulting
object as if it were any other object.
The import keyword makes visible selected symbols from the object or package
specified by some path. The syntax could be extended to allow import to import
symbols into the current scope from a macro synthesized object.
Let's look at an example to get a feel for how this could work:
def someMacro(b: Expr[Boolean])(using Quotes, cx: ImportDecl & EnclosingTemplate): Expr[cx.Decls] = {
if (b.value)
cx.decls('{
object freshTermName {
def fizzle: Boolean = true
}
})
else
cx.decls('{
object freshTermName {
def swizzle: Double = -1.0d
}
})
}
trait Tum {
def tim: String
}
class Foo extends Tum {
import ${someMacro(true)}.*
def tim = fizzle.toString // only fizzle is visible in this scope
}
class Bar extends Tum {
import ${someMacro(false)}.*
def tim = swizzle.toString // only swizzle is visible in this scope
}In this example, classes Foo and Bar both have a method called tim. But
the implementation of that method can depend on the symbols that become visible
after evaluating the import macro. Because the macro evaluates differently in
the two classes, the symbols visible and their contents, is different.
Unlike regular macros, those called with import would take as a given both a
Quotes and a context. The context serves three purposes:
- It allows the compiler to pass a concise easy-to-use description of the macro's surroundings.
- It constrains the macro to the semantics of the invoking statement.
- It allows the macro to specify to both the compiler and users the contexts the macro is valid in.
In the above example we see the macro accept two marker traits: ImportDecl and
EnclosingTemplate. These traits constrain where the macro is valid. The
EnclosingTemplate parameter describes the content of the surrounding object
the macro was called from, including any methods, fields, constructors, type
parameters, etc, as well as the full AST of the object.
The context would also be used to ensure that an AST of the appropriate shape is returned. This is to maintain some level of type-safety so that only object-shaped ASTs are returned by an import/export macro. Unfortunately, object declarations themselves do not have a well-specified type in Scala 3. As a solution, we use the context to ensure that an AST of the appropriate shape is returned.
A deeper discussion of contexts follows in a later section.
Of course, normal import rules would apply to the synthesized object. Private
and protected members would be hidden, etc.
Import macros couldn't change the shape of an object, yet they would be very useful in many situations, especially those where you don't want to alter the shape of an object.
However, fully expressive metaprogramming should be able to change an object's shape. This is the domain of export macros.
The export keyword currently adds forwarders to selected members of an object.
The syntax could be extended to allow export to add forwarders to members of
objects instantiated at compile-time by a macro. Let's look at an example to get
a feel for how this could work:
def someMacro(b: Expr[Boolean])(using Quotes, cx: ExportDecl & EnclosingTemplate): Expr[cx.Decls] = {
if (b.value)
cx.decls('{
object freshTermName {
def fizzle: Boolean = true
}
})
else
cx.decls('{
object freshTermName {
def swizzle: Double = -1.0d
}
})
}
class Foo {
export $someMacro(true).*
}
class Bar {
export $someMacro(false).*
}
new Foo.fizzle // valid
new Foo.swizzle // invalid!
new Bar.swizzle // valid
new Bar.fizzle // invalid!In this example, the class Foo would have the method fizzle added, whereas
the class Bar would have the method swizzle added.
As we saw with the import macro, this macro also takes givens of a Quotes
and a context. The ExportDecl trait constrains the macro to be used only in
export statements.
By default, normal export rules would apply to forwarding declarations of the
synthesized object. Relaxations of these rules are discussed in later sections.
Not every import/export macro will be valid in every context. For example, macros should be able to specify:
- if they work with
import, orexport, or both; - if they work inside an object/class/trait template, or outside a template at package-level (aka "top-level"), or both.
(Scala 3 has introduced top-level declarations. This means that supporting top-level import/export macros should be possible and readily achievable.)
A macro may only be valid, or intended to be used, in one context and should only accept the contexts of its intended call site(s). The compiler would then generate an error, perhaps with an annotation present to supply a customized error message, if the macro was called from an incompatible location.
The restrictions can be described by two pairs of traits. One (or both) trait of each pair would need to be specified in the macro signature.
The traits ImportDecl and ExportDecl constrain the macro to either import or
export semantics, respectively.
The traits EnclosingPackage and EnclosingTemplate constrain the macro to
either package instantiation sites where it would introduce top-level
declarations or class/trait/object template sites. The traits would also
describe the surrounding package, or the surrounding class/trait/object and its
contents, respectively.
Examples:
def onlyExportsInObjects()(using Quotes, cx: ExportDecl & EnclosingTemplate): Expr[cx.Decls]
def onlyImportsAtPackageLevel()(using Quotes, cx: ImportDecl & EnclosingPackage): Expr[cx.Decls]
def anyPlaceAnyWhere()(
using Quotes,
cx: (ImportDecl | ExportDecl) & (EnclosingTemplate | EnclosingPackage)
): Expr[cx.Decls]The syntax for calling a macro, referred to as a Splice in Scala 3, in the
context of an import or export is slightly changed and simplified from its
use in other contexts.
In Scala 3 a Splice looks like:
${...}
This is because expression-oriented contexts require the additional syntax in
order to disambiguate the call. The rules of paths are stricter, and not
expression oriented. This means we can simplify the syntax in the import and
export contexts.
The syntax proposed is:
$path.to.macro(..args..).<selector>
For example:
import $some.deep.path().*
export $visibleMacro(1, false, "str", Array("a", "b", "c")).*To import or export a macro, the path would be required to start with a $,
followed by the path to the macro function, and finish with required parenthesis
containing the arguments, if any, to the macro.
For the POC, arguments would be restricted to the set of literal values supported by annotations. This would be relaxed to support more literal types in time.
While the aforementioned syntax may feel a bit odd, users of Ammonite should be comfortable with it since Ammonite already does something very similar with its "magic imports".
For example import $file.ScriptFile._ is special syntax interpreted by
Ammonite to load the symbols from another script file, named ScriptFile.sc, and
brings them into the current file.
Given the popularity of Ammonite, this should make the syntax readily acceptable, with a large user-base already accustomed to the functionality.
The syntax also gives Ammonite a chance to replace it's "magic imports" with standard Scala syntax.
The prior example could become import $file("ScriptFile").*, where file is
now simply a macro that Ammonite makes visible to all scripts.
@kitlangton Thank you for the feedback!
On ergonomics/complexity:
I didn't originally distinguish between import and export, but after considering the feature for a while I think there are a few reasons we need to.
First, we need some way to insert definitions into a companion object vs just into the current context. The
declsmethod on ExportDecl has a slightly different signature that accepts a second object that will be inserted into the companion object of the enclosing class (EnclosingTemplate being the mechanism of enforcement of the fact there is an enclosing class or trait for the compiler and user). I've yet to think of a use case for inserting random imports into a companion object. Can we come up with one?Second, I think library authors will need some control over the "interface" of the synthesized definitions, since I think it will in general be more difficult to create a safe and usable export macro than an import one. The authors of export macros will need to be very careful about the names of definitions, and may only use a few. Import macros can be less careful and have just a few names users are supposed to import.
On optimizing the SIP:
I very much agree. In fact I would pull out more than just that to a separate document.
I ended up having a conversation with Oron Port a while back. His use-cases all required mechanical rewriting of existing code in order to reduce boilerplate. None of the proposals in this document addressed that use case and, in general, I'm not sure it's a use that can be easily and ergonomically supported. But I have the feeling that mechanical transformations need a much more thorough consideration and certainly a different proposal. I also have some thoughts on how they could be implemented.
Odersky also recently stated that macro annotations are either being worked on, or will be, by someone at EPFL. That gives me pause on my proposed approach for macro annotations as well.
On the extensions:
export inlineandexport overrideare absolutely performance and ergonomic inspired extensions. We can leave them for another document, but I think they highlight something that we should give some thought to: composition and re-usability.inline traitserves a few purposes. One is to be a facility for abstracting away import/export macros that appear repeatedly in a codebase. In some cases, macros will be created that expect other macros to have been previously instantiated. Having such code wrapped up for new users to safely re-use is key I think for general ergonomics regarding macros.The second purpose is to easily allow the content of a class to be the input to the macro and the class itself the target of the macro logic.
I also just really liked the symmetry of the language having
inline traitandinline def.Implementation:
Since writing this proposal I've been hacking some evenings trying to implement import/export macros in dotty. While it has been slow going for me I see a pretty clear implementation path. The repo is here: https://github.com/littlenag/dotty/tree/inline-trait
Next Steps:
I'll update this document and leave just the import/export macro content. The rest I'll move someplace else.