Expressive Metaprogramming for Scala 3

Overview

Scala 3 introduces a new macro system that replaces the experimental Scala 2 system with one that should avoid unsoundness, be simpler to use, and simpler for the compiler team to evolve and maintain.

Perhaps the most significant feature missing from Scala 3 today is the lack of support for expressive compile-time metaprogramming. By this, I mean the ability to generate and/or reshape classes, traits, and objects at compile-time. In Scala 2, the community used macro annotations for this purpose. Scala 3 addressed a few of the use-cases of macro annotations, but we still lack a feature that offers anything close to the full range of expressive capability of macro annotations.

This document proposes a replacement for Scala 2 macro annotations that should:

offer nearly same expressive power,
address all, or most, major use cases,
be simpler to maintain, and
be safer and easier to use.

On the implementation side this replacement should:

be integrated with the compiler (instead of a plugin),
be simpler to maintain,
expose no compiler internals, and
re-use the existing macro system and its semantics as much as possible.

Most importantly, this feature should fit cleanly into the existing Scala 3 metaprogramming model and existing language features.

Desired Metaprogramming Capabilities

There are two key design principles this feature should follow:

User code as written must not be altered, unless the code is explicitly marked as modifiable.
Nothing extraneous should be required to be generated by a macro.

Due to various limitations, macro annotations in Scala 2 are unable to adhere to either design principle.

Generally, macro annotations are hard for authors to make correct, difficult for users to introspect, and effectively unrestricted in their capabilities. This is because at their core they are simply a function of AST => AST. This means they can:

alter the inheritance hierarchy of a class/trait/object,
change order, types, or names of class parameters,
modify generic type parameters,
and more.

Instead, this proposal seeks to cleanly integrate with the more restrictive metaprogramming model in Scala 3. Features would be limited to:

adding new declarations (fields, methods, types, classes, etc) at compile-time
into existing named traits/classes/objects/packages
as a function of:
- content of the enclosing trait/class/object
- static parameters
overriding existing declarations when explicitly allowed.

Import/Export Macros

By extending the semantics of import and export we can achieve most of the stated design goals.

Today import and export have the exact same syntax, but dual functionality. Both take a path to a stable identifier followed by selectors that determine which symbols should be processed and how. The syntax could be extended to allow import and export to accept a path to a macro and along with static arguments to the macro, instantiate the macro, and then process the resulting object as if it were any other object.

Import Macros

The import keyword makes visible selected symbols from the object or package specified by some path. The syntax could be extended to allow import to import symbols into the current scope from a macro synthesized object.

Let's look at an example to get a feel for how this could work:

def someMacro(b: Expr[Boolean])(using Quotes, cx: ImportDecl & EnclosingTemplate): Expr[cx.Decls] = {
  if (b.value)
    cx.decls('{
      object freshTermName {
        def fizzle: Boolean = true
      }
    })
  else
    cx.decls('{
      object freshTermName {
        def swizzle: Double = -1.0d
      }
    })
}

trait Tum {
  def tim: String
}

class Foo extends Tum {

  import ${someMacro(true)}.*

  def tim = fizzle.toString // only fizzle is visible in this scope
}

class Bar extends Tum {

  import ${someMacro(false)}.*

  def tim = swizzle.toString // only swizzle is visible in this scope
}

In this example, classes Foo and Bar both have a method called tim. But the implementation of that method can depend on the symbols that become visible after evaluating the import macro. Because the macro evaluates differently in the two classes, the symbols visible and their contents, is different.

Unlike regular macros, those called with import would take as a given both a Quotes and a context. The context serves three purposes:

It allows the compiler to pass a concise easy-to-use description of the macro's surroundings.
It constrains the macro to the semantics of the invoking statement.
It allows the macro to specify to both the compiler and users the contexts the macro is valid in.

In the above example we see the macro accept two marker traits: ImportDecl and EnclosingTemplate. These traits constrain where the macro is valid. The EnclosingTemplate parameter describes the content of the surrounding object the macro was called from, including any methods, fields, constructors, type parameters, etc, as well as the full AST of the object.

The context would also be used to ensure that an AST of the appropriate shape is returned. This is to maintain some level of type-safety so that only object-shaped ASTs are returned by an import/export macro. Unfortunately, object declarations themselves do not have a well-specified type in Scala 3. As a solution, we use the context to ensure that an AST of the appropriate shape is returned.

A deeper discussion of contexts follows in a later section.

Of course, normal import rules would apply to the synthesized object. Private and protected members would be hidden, etc.

Import macros couldn't change the shape of an object, yet they would be very useful in many situations, especially those where you don't want to alter the shape of an object.

However, fully expressive metaprogramming should be able to change an object's shape. This is the domain of export macros.

Export Macros

The export keyword currently adds forwarders to selected members of an object. The syntax could be extended to allow export to add forwarders to members of objects instantiated at compile-time by a macro. Let's look at an example to get a feel for how this could work:

def someMacro(b: Expr[Boolean])(using Quotes, cx: ExportDecl & EnclosingTemplate): Expr[cx.Decls] = {
  if (b.value)
    cx.decls('{
      object freshTermName {
        def fizzle: Boolean = true
      }
    })
  else
    cx.decls('{
      object freshTermName {
        def swizzle: Double = -1.0d
      }
    })
}

class Foo {
  export $someMacro(true).*
}

class Bar {
  export $someMacro(false).*
}

new Foo.fizzle  // valid
new Foo.swizzle // invalid!

new Bar.swizzle // valid
new Bar.fizzle  // invalid!

In this example, the class Foo would have the method fizzle added, whereas the class Bar would have the method swizzle added.

As we saw with the import macro, this macro also takes givens of a Quotes and a context. The ExportDecl trait constrains the macro to be used only in export statements.

By default, normal export rules would apply to forwarding declarations of the synthesized object. Relaxations of these rules are discussed in later sections.

Context Restrictions

Not every import/export macro will be valid in every context. For example, macros should be able to specify:

if they work with import, or export, or both;
if they work inside an object/class/trait template, or outside a template at package-level (aka "top-level"), or both.

(Scala 3 has introduced top-level declarations. This means that supporting top-level import/export macros should be possible and readily achievable.)

A macro may only be valid, or intended to be used, in one context and should only accept the contexts of its intended call site(s). The compiler would then generate an error, perhaps with an annotation present to supply a customized error message, if the macro was called from an incompatible location.

The restrictions can be described by two pairs of traits. One (or both) trait of each pair would need to be specified in the macro signature.

The traits ImportDecl and ExportDecl constrain the macro to either import or export semantics, respectively.

The traits EnclosingPackage and EnclosingTemplate constrain the macro to either package instantiation sites where it would introduce top-level declarations or class/trait/object template sites. The traits would also describe the surrounding package, or the surrounding class/trait/object and its contents, respectively.

Examples:

def onlyExportsInObjects()(using Quotes, cx: ExportDecl & EnclosingTemplate): Expr[cx.Decls]

def onlyImportsAtPackageLevel()(using Quotes, cx: ImportDecl & EnclosingPackage): Expr[cx.Decls]

def anyPlaceAnyWhere()(
  using Quotes, 
  cx: (ImportDecl | ExportDecl) & (EnclosingTemplate | EnclosingPackage)
): Expr[cx.Decls]

Simplified Splice Syntax

The syntax for calling a macro, referred to as a Splice in Scala 3, in the context of an import or export is slightly changed and simplified from its use in other contexts.

In Scala 3 a Splice looks like:

${...}

This is because expression-oriented contexts require the additional syntax in order to disambiguate the call. The rules of paths are stricter, and not expression oriented. This means we can simplify the syntax in the import and export contexts.

The syntax proposed is:

$path.to.macro(..args..).<selector>

For example:

import $some.deep.path().*
export $visibleMacro(1, false, "str", Array("a", "b", "c")).*

To import or export a macro, the path would be required to start with a $, followed by the path to the macro function, and finish with required parenthesis containing the arguments, if any, to the macro.

For the POC, arguments would be restricted to the set of literal values supported by annotations. This would be relaxed to support more literal types in time.

Ammonite

While the aforementioned syntax may feel a bit odd, users of Ammonite should be comfortable with it since Ammonite already does something very similar with its "magic imports".

For example import $file.ScriptFile._ is special syntax interpreted by Ammonite to load the symbols from another script file, named ScriptFile.sc, and brings them into the current file.

Given the popularity of Ammonite, this should make the syntax readily acceptable, with a large user-base already accustomed to the functionality.

The syntax also gives Ammonite a chance to replace it's "magic imports" with standard Scala syntax.

The prior example could become import $file("ScriptFile").*, where file is now simply a macro that Ammonite makes visible to all scripts.

littlenag/expressive_metaprogramming.md