Scala 3 introduces a new macro system that replaces the experimental Scala 2 system with one that should avoid unsoundness, be simpler to use, and simpler for the compiler team to evolve and maintain.
Perhaps the most significant feature missing from Scala 3 today is the lack of support for expressive compile-time metaprogramming. By this, I mean the ability to generate and/or reshape classes, traits, and objects at compile-time. In Scala 2, the community used macro annotations for this purpose. Scala 3 addressed a few of the use-cases of macro annotations, but we still lack a feature that offers anything close to the full range of expressive capability of macro annotations.
This document proposes a replacement for Scala 2 macro annotations that should:
- offer nearly same expressive power,
- address all, or most, major use cases,
- be simpler to maintain, and
- be safer and easier to use.
On the implementation side this replacement should:
- be integrated with the compiler (instead of a plugin),
- be simpler to maintain,
- expose no compiler internals, and
- re-use the existing macro system and its semantics as much as possible.
Most importantly, this feature should fit cleanly into the existing Scala 3 metaprogramming model and existing language features.
There are two key design principles this feature should follow:
- User code as written must not be altered, unless the code is explicitly marked as modifiable.
- Nothing extraneous should be required to be generated by a macro.
Due to various limitations, macro annotations in Scala 2 are unable to adhere to either design principle.
Generally, macro annotations are hard for authors to make correct, difficult for
users to introspect, and effectively unrestricted in their capabilities. This is
because at their core they are simply a function of AST => AST. This means
they can:
- alter the inheritance hierarchy of a class/trait/object,
- change order, types, or names of class parameters,
- modify generic type parameters,
- and more.
Instead, this proposal seeks to cleanly integrate with the more restrictive metaprogramming model in Scala 3. Features would be limited to:
- adding new declarations (fields, methods, types, classes, etc) at compile-time
- into existing named traits/classes/objects/packages
- as a function of:
- content of the enclosing trait/class/object
- static parameters
- overriding existing declarations when explicitly allowed.
By extending the semantics of import and export we can achieve most of the
stated design goals.
Today import and export have the exact same syntax, but dual functionality.
Both take a path to a stable identifier followed by selectors that determine
which symbols should be processed and how. The syntax could be extended to allow
import and export to accept a path to a macro and along with static
arguments to the macro, instantiate the macro, and then process the resulting
object as if it were any other object.
The import keyword makes visible selected symbols from the object or package
specified by some path. The syntax could be extended to allow import to import
symbols into the current scope from a macro synthesized object.
Let's look at an example to get a feel for how this could work:
def someMacro(b: Expr[Boolean])(using Quotes, cx: ImportDecl & EnclosingTemplate): Expr[cx.Decls] = {
if (b.value)
cx.decls('{
object freshTermName {
def fizzle: Boolean = true
}
})
else
cx.decls('{
object freshTermName {
def swizzle: Double = -1.0d
}
})
}
trait Tum {
def tim: String
}
class Foo extends Tum {
import ${someMacro(true)}.*
def tim = fizzle.toString // only fizzle is visible in this scope
}
class Bar extends Tum {
import ${someMacro(false)}.*
def tim = swizzle.toString // only swizzle is visible in this scope
}In this example, classes Foo and Bar both have a method called tim. But
the implementation of that method can depend on the symbols that become visible
after evaluating the import macro. Because the macro evaluates differently in
the two classes, the symbols visible and their contents, is different.
Unlike regular macros, those called with import would take as a given both a
Quotes and a context. The context serves three purposes:
- It allows the compiler to pass a concise easy-to-use description of the macro's surroundings.
- It constrains the macro to the semantics of the invoking statement.
- It allows the macro to specify to both the compiler and users the contexts the macro is valid in.
In the above example we see the macro accept two marker traits: ImportDecl and
EnclosingTemplate. These traits constrain where the macro is valid. The
EnclosingTemplate parameter describes the content of the surrounding object
the macro was called from, including any methods, fields, constructors, type
parameters, etc, as well as the full AST of the object.
The context would also be used to ensure that an AST of the appropriate shape is returned. This is to maintain some level of type-safety so that only object-shaped ASTs are returned by an import/export macro. Unfortunately, object declarations themselves do not have a well-specified type in Scala 3. As a solution, we use the context to ensure that an AST of the appropriate shape is returned.
A deeper discussion of contexts follows in a later section.
Of course, normal import rules would apply to the synthesized object. Private
and protected members would be hidden, etc.
Import macros couldn't change the shape of an object, yet they would be very useful in many situations, especially those where you don't want to alter the shape of an object.
However, fully expressive metaprogramming should be able to change an object's shape. This is the domain of export macros.
The export keyword currently adds forwarders to selected members of an object.
The syntax could be extended to allow export to add forwarders to members of
objects instantiated at compile-time by a macro. Let's look at an example to get
a feel for how this could work:
def someMacro(b: Expr[Boolean])(using Quotes, cx: ExportDecl & EnclosingTemplate): Expr[cx.Decls] = {
if (b.value)
cx.decls('{
object freshTermName {
def fizzle: Boolean = true
}
})
else
cx.decls('{
object freshTermName {
def swizzle: Double = -1.0d
}
})
}
class Foo {
export $someMacro(true).*
}
class Bar {
export $someMacro(false).*
}
new Foo.fizzle // valid
new Foo.swizzle // invalid!
new Bar.swizzle // valid
new Bar.fizzle // invalid!In this example, the class Foo would have the method fizzle added, whereas
the class Bar would have the method swizzle added.
As we saw with the import macro, this macro also takes givens of a Quotes
and a context. The ExportDecl trait constrains the macro to be used only in
export statements.
By default, normal export rules would apply to forwarding declarations of the
synthesized object. Relaxations of these rules are discussed in later sections.
Not every import/export macro will be valid in every context. For example, macros should be able to specify:
- if they work with
import, orexport, or both; - if they work inside an object/class/trait template, or outside a template at package-level (aka "top-level"), or both.
(Scala 3 has introduced top-level declarations. This means that supporting top-level import/export macros should be possible and readily achievable.)
A macro may only be valid, or intended to be used, in one context and should only accept the contexts of its intended call site(s). The compiler would then generate an error, perhaps with an annotation present to supply a customized error message, if the macro was called from an incompatible location.
The restrictions can be described by two pairs of traits. One (or both) trait of each pair would need to be specified in the macro signature.
The traits ImportDecl and ExportDecl constrain the macro to either import or
export semantics, respectively.
The traits EnclosingPackage and EnclosingTemplate constrain the macro to
either package instantiation sites where it would introduce top-level
declarations or class/trait/object template sites. The traits would also
describe the surrounding package, or the surrounding class/trait/object and its
contents, respectively.
Examples:
def onlyExportsInObjects()(using Quotes, cx: ExportDecl & EnclosingTemplate): Expr[cx.Decls]
def onlyImportsAtPackageLevel()(using Quotes, cx: ImportDecl & EnclosingPackage): Expr[cx.Decls]
def anyPlaceAnyWhere()(
using Quotes,
cx: (ImportDecl | ExportDecl) & (EnclosingTemplate | EnclosingPackage)
): Expr[cx.Decls]The syntax for calling a macro, referred to as a Splice in Scala 3, in the
context of an import or export is slightly changed and simplified from its
use in other contexts.
In Scala 3 a Splice looks like:
${...}
This is because expression-oriented contexts require the additional syntax in
order to disambiguate the call. The rules of paths are stricter, and not
expression oriented. This means we can simplify the syntax in the import and
export contexts.
The syntax proposed is:
$path.to.macro(..args..).<selector>
For example:
import $some.deep.path().*
export $visibleMacro(1, false, "str", Array("a", "b", "c")).*To import or export a macro, the path would be required to start with a $,
followed by the path to the macro function, and finish with required parenthesis
containing the arguments, if any, to the macro.
For the POC, arguments would be restricted to the set of literal values supported by annotations. This would be relaxed to support more literal types in time.
While the aforementioned syntax may feel a bit odd, users of Ammonite should be comfortable with it since Ammonite already does something very similar with its "magic imports".
For example import $file.ScriptFile._ is special syntax interpreted by
Ammonite to load the symbols from another script file, named ScriptFile.sc, and
brings them into the current file.
Given the popularity of Ammonite, this should make the syntax readily acceptable, with a large user-base already accustomed to the functionality.
The syntax also gives Ammonite a chance to replace it's "magic imports" with standard Scala syntax.
The prior example could become import $file("ScriptFile").*, where file is
now simply a macro that Ammonite makes visible to all scripts.
Firstly, this is wonderful! A reasonable substitute for annotation macros is, IMHO, necessary for Scala 3's success. This ability to synthesize new methods was the most common, practical, and sane use-case for annotation macros, and you've rightly identified it as the feature to re-enable.
Here's some random feedback! :)
On ergonomics/complexity:
The ImportDecl/ExportDecl distinction seems unnecessary to me. I could be missing context, but it adds extra complexity to the development/usage of these macros that seems unneeded. Why shouldn't the user be able to decide by simply switching between import/export keywords? But perhaps this is my lack of imagination as to the issues this might cause.
On optimizing the SIP:
The core
import/exportmacro feature is so important that I feel it's worth clearly prioritizing it. The override/companion/inline/trait features seem to be extensions/conveniences atop this important core ability. It may worth be eliding them for an initial pitch. Or, at least, explicitly qualifying them as extensions.Okay! Take that with a grain of salt :) Thanks for writing this up! I really hope the Scala 3 team can be convinced of the importance of this feature.
P.S.: We had our own ideas for a hacky, incremental annotation macro replacement here (scala/scala3#14891 (comment)), but first class support would be much much better.