RFC for macros in golo

Macros for Golo are becoming a reality 😄

See https://github.com/yloiseau/golo-lang/tree/wip/macro-quote

By the way, some choices need to be made, hence this RFC.

Syntax elements

Currently I have added several special syntax elements:

quote { golo code } is replaced in the Ir tree by CodeBuilder calls that create a tree equivalent to the enclosed code. This allows functions (macro) to easily return an Ir tree.
unquote(expr) or ~var that is the reverse of quote, and allows to re-inject values in the tree built by quote
macro foo = |params| {...} to define a macro. Regular functions can do the work, but it makes the compilation step easier. I'll try to get rid of this one since this step is not dry yet.
&foo(args), &foo(args) { block } or &foo { block } to call the macro. Hard to get rid of this one (see expansion step).

Definition step

A macros is just regular function that takes Ir nodes as arguments and returns an Ir node (a new one or its argument, transformed or not).

The quote/unquote keywords allow to build easily simple Ir trees to return. More complex trees can be build using the gololang.macros.CodeBuilder module.

For instance, code like

function macroTest = {
  return quote {
    println(1 + 2)
  }
}

generates a Ir equivalent to:

function macroTest= {
  return functionInvocation()
          : name("println")
          : arg(
             binaryOperation(PLUS(), constant(1), constant(2))
          )
}

Since macro are just regular functions, hence static methods, they even can be written in Java, provided the class is available to the compiler.

Currently, only statements can be quoted. It is not possible to write code like

return quote {
  function foo = |arg| { ... }
}

return quote {
  struct Foo = {bar, baz}
}

since in the grammar, the quote keyword is followed by a Block.

However, macros returning top-level statements (currently only functions, but structs and augments are coming soon) can be written by using the CodeBuilder facilities. For instance, let the macro in macros.golo

module TopLevelMacros

import gololang.macros.CodeBuilder
import gololang.macros.CodeBuilder$Operations

function myMacro = -> publicFunction()
  : name("foo")
  : param("x", "y")
  : block(
    returns(plus(refLookup("x"), refLookup("y")))
  )

called in test.golo like

module TopLevelTest

&TopLevelMacros.myMacro()

function main = |args| {
  println(foo(1, 2))
}

results in an Ir equivalent to:

module Test

function foo = |x, y| -> x + y

function main = |args| {
  println(foo(1, 2))
}

and golo compile macros.golo && golo golo --files test.golo prints 3 (this is actually working code 😄)

I don't plan to allow quote to contains such top-level statements (yet).

Compilation step

More work is needed here. Macro must be available at compile time (see expansion step), thus they currently must be defined in a separate module that must be compiled before the compilation of the module using it, and the produced .class must be in the CLASSPATH (more precisely in CLASSPATH_PREFIX) of the golo command when golo compile or golo golo the module using it. That is why I introduced the macro keyword, to make a macro only compilation step and inject the produced classes in the class loader dynamically. Using this, macros will be compiled in a different class than normal code before the compilation of the code itself.

For instance

module Foo

macro myMacro = |params| { ... }

function foo = {
  &myMacro(args)
}

will produce a Foo.Macros class containing only the myMacro method that will be injected in the compiler class loader to then create the Foo class containing only the foo method with the macro expanded. This is not currently working yet.

However creating two modules macros.golo and test.golo containing

module Macros

function myMacro = |params| { ... }

and

module Test

import Macros

function foo = {
  &myMacro(args)
}

function main = |args| { ... }

respectively, and running it with golo compile macros.golo && golo golo --files test.golo or golo compile macros.golo test.golo && golo run --module Test works. The order of file in the second command is relevant.

Maybe adding a command line option to explicitly add macro containing modules to the compiler could also be useful.

See this other gist for more discussion on this aspect https://gist.github.com/yloiseau/8a8ce445cf9393239bec

Expansion step

When compiling “normal” code, the Ir tree is walked by a visitor that:

look for macro invocation nodes (marked with the & prefix)
find the corresponding method and invoke it (thus the need for availability at compile time)
visit the result to expand macros recursively
replace the macro invocation node in the parent node by the result node

I'd like to add an other kind of macros, that take the expansion context as a first argument, much like a method invocation takes the object itself as first argument, e.g. the module itself for a top-level macro call. This would allow the macro to modify directly the context, instead of just returning a node that will replace the macro invocationi (think JS that change the DOM tree...) the context parameter being automagically added to the call (just like method call).

The question here is how to define this kind of macros:

just a regular macro with a special invocation mark (like method that are just regular functions but called differently), for instance &:

macro foo = |context, arg1, arg2, block| { ... }

&:foo (a, b) { ... }

Pros: no special declaration, different use possible (normal call with an explicit different context), possible to use existing functions.

Cons: unexpected result if called normally, nothing besides doc tells the user that this macro must be called with special mark.

special declaration and normal invocation, e.g.

contextual macro foo = |context, arg1, arg2, block| { ... }

&foo (a, b) { ... }

Pros: compile time check of the presence of the fist parameter (like for augment), can't be called without a context.

Cons: add another keyword, can't use regular function as macro.

everything implicit: the context parameter is not even specified in the macro signature, and it is automagically added to the reference table when calling the macro (similar to the this behavior in JS)

Pros: no special syntax.

Cons: explicit is better than implicit, the behavior may be difficult to predict depending on the implementation (see JS this 😄).

everything explicit: the macro specify a parameter for the context, the context is explicitly passed as argument in the macro call, e.g.

macro foo = |context, arg1, arg2, block| { ... }

&foo (__CONTEXT__, a, b) { ... }

Pros: everything is explicit.

Cons: more verbose, must provide a “special” variable that reference the context, e.g. __CONTEXT__ here, which can be replaced dynamically by the Ir visitor, that would be only a symbol (search for Ref{name=__CONTEXT__} and replace it by the actual node).

I personally like the first solution, or a mix of 1 and 4. Any thoughts?

Some use cases

(all this is working code 😄)

Ever wanted a repeat until construct in Golo?

function repeatUntil = |cond, repeatBlock| -> quote {
  unquote(repeatBlock)
  while not ~cond {
    unquote(repeatBlock)
  }
}

&repeatUntil (a < 0) {
  println(a)
  a = a - 1
}

Ever wanted string interpolation in Golo?

function str = |s| {
  let matcher = Pattern.compile("\\$\\{([^}]+)\\}"): matcher(s: getValue())
  let pattern = constant(matcher: replaceAll("%s"))
  let func = functionInvocation(): name("String.format")
    : arg(pattern)
  matcher: reset()
  while (matcher: find()) {
    func: arg(refLookup(matcher: group(1)))
  }
  return func
}

function main = |args| {
  let a = 42
  let b = "world"
  println(&str("Hello ${b}, the answer is ${a}"))
}

(ping @danielpetisme)

To be done

pre-compilations step and class injection to allow “on the fly” macro compilation when doing golo golo --files macros.golo test.golo
command line option to inject macro class in the compiler
top level macros to generate structs, augmentations and so on
special macro working directly on its context
tests, doc, code cleaning, predefined macros library, …

yloiseau/RFC-macros.md