Metaprogramming in Elixir - with a practical example

What is metaprogramming?

Writing code that writes code
In Elixir this could mean:
- Write code outside of functions to define functions or modules
- Implement a macro to manipulate AST at compile time
- Manipulate and evaluate AST at runtime

What is an Elixir AST

Internally, any Elixir code is expressed in an Abstract Syntax Tree which is an Elixir term.

Elixir code is represented as follows:

Any simple value like numbers, atoms and strings are represented by themselves
Lists are represented by themselves
Tuples of size two are represented by themselves
All other pieces of Elixir code are represented as a three-tuple
- First, a name of the function, variable or whatever is represented. This is either a simple value or a three-tuple
- Second, a list of context data
- Third, arguments

Find the AST (quoted expression) of any Elixir code by using quote:

quote do: sum(1, 2)
# {:sum, [], [1, 2]}

Examples:

# Simple value
iex(3)> quote do: 1
1

# Variable
iex(4)> quote do: a
{:a, [], Elixir}

# Anonymous function
iex(5)> quote do: fn x -> x + 1 end
{
  :fn,
  [],
  [
    {
      :->,
      [],
      [
        [{:x, [], Elixir}],
        {
          :+,
          [context: Elixir, import: Kernel],
          [
            {:x, [], Elixir},
            1
          ]
        }
      ]
    }
  ]
}

# Function definition
iex(7)> quote do: def sum(a, b), do: a + b
{
  :def,
  [context: Elixir, import: Kernel],
  [
    {:sum, [context: Elixir], [{:a, [], Elixir}, {:b, [], Elixir}]},
    [
      do: {
        :+,
        [context: Elixir, import: Kernel],
        [
          {:a, [], Elixir},
          {:b, [], Elixir}
        ]
      }
    ]
  ]
}

# Map construction
iex(8)> quote do: %{a: 1}
{:%{}, [], [a: 1]}

# An entire module
iex(9)> quote do
...(9)> defmodule MyModule do
...(9)> def my_fun(x) do
...(9)> IO.inspect(x)
...(9)> :ok
...(9)> end
...(9)> end
...(9)> end
{
  :defmodule,
  [context: Elixir, import: Kernel],
  [
    {:__aliases__, [alias: false], [:MyModule]},
    [
      do: {
        :def,
        [context: Elixir, import: Kernel],
        [
          {:my_fun, [context: Elixir], [{:x, [], Elixir}]},
          [do: {:__block__, [], [{{:., [], [{:__aliases__, [alias: false], [:IO]}, :inspect]}, [], [{:x, [], Elixir}]}, :ok]}]
        ]
      }
    ]
  ]
}

quote with unquote

Unquote is used to inject variables into a quoted expression:

iex(1)> a = 42
42

iex(2)> quote do: a + 5
{:+, [context: Elixir, import: Kernel], [{:a, [], Elixir}, 5]}

iex(3)> quote do: unquote(a) + 5
{:+, [context: Elixir, import: Kernel], [42, 5]}

Defining functions at compile time

defmodule HTTP do
  @status_codes %{
    200 => "ok",
    400 => "invalid",
    404 => "not found"
  }

  for {status, description} <- @status_codes do
    def description(unquote(status)) do
      unquote(description)
    end
  end
end

IO.puts "The description for code 400 is '#{HTTP.description(400)}'"
# The description for code 400 is 'invalid'

The compiled beam file is equivalent to what would be compiled from

defmodule http do
  def description(200), do: "ok"
  def description(400), do: "invalid"
  def description(404), do: "not found"
end

Implementing a macro

Macros transform the Elixir AST

Elixir code -> Turn into Elixir AST -> Macro expansion -> BEAM code

defmodule Math do
  defmacro human(term) do
    human_quoted(term)
  end

  defp human_quoted({operator, _context, [arg1, arg2]}) when operator in [:+, :-, :*, :/] do
    expression =
      [
        human_quoted(arg1),
        human_quoted(operator),
        human_quoted(arg2)
      ]
      |> Enum.join(" ")
    "(#{expression})"
  end
  defp human_quoted(number) when is_number(number) do
    "#{number}"
  end
  defp human_quoted(:+), do: "plus"
  defp human_quoted(:-), do: "minus"
  defp human_quoted(:*), do: "times"
  defp human_quoted(:/), do: "divided by"
end

defmodule Demo do
  require Math

  def run do
    Math.human(1 + 2 * 3 / 4 - 5)
    |> IO.puts
  end
end

Demo.run
# ((1 plus ((2 times 3) divided by 4)) minus 5)

Notice that the compiled(!) Demo module is equivalent to:

defmodule Demo do
  def run do
    "((1 plus ((2 times 3) divided by 4)) minus 5)"
    |> IO.puts
  end
end

Runtime AST manipulation with a practical example

Use case: Needs a pub sub hub with pattern subscription

Want to do something like this:

Hub.subscribe("some channel", %{age: age} when age > 42)

Hub.publish("some channel", %Person{name: "John", age: 15}) # Will not be published to the above process
Hub.publish("some channel", %Person{name: "Jane", age: 55}) # Will be published to the above process

Advantages:

Will not send messages between processes when not needed
We now at publish time if a message is handled by a subscriber

Disadvantage:

Will have to match a pattern for each message for each subscriber

The challenge

Normally we write a pattern where it is used to pattern match:

case message do
  when %{age: age} = message when age > 42 ->
    handle_message(message)
  _ ->
    :skip
end

We can not directly save a pattern in a variable:

iex(5)> pattern = %{age: age} when age > 42
** (CompileError) iex:5: undefined function when/2

Solution

First we save the pattern as an AST:

iex(5)> pattern = quote do: %{age: age} when age > 42
{:when, [],
 [{:%{}, [], [age: {:age, [], Elixir}]},
  {:>, [context: Elixir, import: Kernel], [{:age, [], Elixir}, 42]}]}

Then we inject the pattern in another AST (using quote and unquote)

def pattern_match?(pattern, term) do
  quoted_term = Macro.escape(term)

  ast = quote do
    case unquote(quoted_term) do
      unquote(pattern) -> true
      _ -> false
    end
  end

  {result, _} = Code.eval_quoted(ast)
  result
end

And that's it!

Hub available at github and hex.

Metaprogramming tips

Function creation

Is awesome for creating functions from a data source
Works even with files and web as data source
Gives super fast runtime evaluation
Use it as much as possible! (where it makes sense)
Example.from Elixir unicode support: https://github.com/elixir-lang/elixir/blob/master/lib/elixir/unicode/unicode.ex#L394

Macros

Macros receives AST as input and must return AST
Only write a macro if needed
Implement macros as much as possible with normal functions doing the heavy lifting
Example from Hub: https://github.com/vesta-merkur/hub/blob/master/lib/hub.ex#L49

Compile time module creation

Don't go there

This is definitely feasable to do, but at almost any cost, try to avoid it.

using

The __using__ macro is invoked on use MyModule
Typically used to setup a module for a specific use case
Example from ExUnit.Case: https://github.com/elixir-lang/elixir/blob/master/lib/ex_unit/lib/ex_unit/case.ex#L197

DSL

Rare use case where a DSL is a good choice. If it is, macros are excellent at it.
Typically implemented by defining the __using__ macro, which is invoked on use MyModule
Example: ExUnit

lasseebert/elixir_metaprogramming_201705.md