Improved `assert_compile_time_raise` based on this comment by Andrea Leopardi: http://andrealeopardi.com/posts/compile-time-work-with-elixir-macros/#comment-2347206739

test_helper.exs

ExUnit.start()

defmodule CompileTimeAssertions do
  defmacro assert_compile_time_raise(expected_exception, expected_message, fun) do
    # At compile-time, the fun is in AST form and thus cannot raise.
    # At run-time, we will evaluate this AST, and it may raise.
    fun_quoted_at_runtime = Macro.escape(fun)

    quote do
      assert_raise unquote(expected_exception), unquote(expected_message), fn ->
        Code.eval_quoted(unquote(fun_quoted_at_runtime))
      end
    end
  end
end

This is much later, but this discussion has been very helpful, thanks!