New API proposal for (block/closure) compilers in Rubinius

ast.rb

require_relative './new_api'

class MyLang::AST::Function < RBX::AST::Node
  def initialize(arguments, body)
    @arguments = arguments
    @body = body
    @scope = RBX::Scope.new
  end

  def bytecode(g)
    pos(g)

    fn_g = RBX::Generator.for_block(g, scope)

    fn_c = RBX::BlockCompiler.new(arguments, body)
    fn_c.compile_arguments(fn_g)
    fn_c.compile_body(fn_g)
    fn_c.finalize(fn_g)

    g.create_block fn_g
  end

  private
  attr :arguments, :body, :scope
end

new_api.rb

RBX = Rubinius::ToolSet.current::TS # just a convenience alias

class RBX::Scope
  include RBX::Compiler::LocalVariables

  attr_accessor :parent

  def nest_scope(scope)
    scope.parent = self
  end

  def search_local(name)
    if variable = variables[name]
      variable.nested_reference
    elsif reference = @parent.search_local(name)
      reference.depth += 1
      reference
    end
  end

  def new_local(name)
    variables[name] ||= RBX::Compiler::LocalVariable.new allocate_slot
  end

  def new_nested_local(name)
    new_local(name).nested_reference
  end

  def assign_local_reference(var)
    if variable = variables[var.name]
      var.variable = variable.reference
    elsif reference = @parent.search_local(var.name)
      reference.depth += 1
      var.variable = reference
    else
      variable = new_local var.name
      var.variable = variable.reference
    end
  end
end

class RBX::BlockCompiler
  def initialize(arguments, body)
    @arguments = arguments
    @body = body
  end

  def compile_arguments(g)
    g.required_args = required_args
    g.post_args = post_args
    g.total_args = total_args
    g.cast_for_multi_block_arg unless arguments.count.zero?

    arguments.each_with_index do |a, i|
      g.shift_array
      local = g.state.scope.new_local(a.to_s)
      g.set_local local.slot
      g.pop
    end
    g.pop unless arguments.empty?
  end

  def compile_body(g)
    g.state.push_block

    body.each_with_index do |expression, idx|
      expression.bytecode(g)
      g.pop unless idx == body.size - 1
    end

    g.state.pop_block
    g.ret
  end

  def finalize(g)
    g.local_names = g.state.scope.local_names
    g.local_count = g.state.scope.local_count
    g.pop_state
    g.close
  end

  private
  attr_reader :arguments, :body

  def required_args
    arguments.count
  end

  def post_args
    arguments.count
  end

  def total_args
    arguments.count
  end
end

class RBX::Generator
  def self.for_block(g, scope)
    blk_g = g.class.new
    blk_g.name = g.state.name || :__block__
    blk_g.file = g.file
    blk_g.for_block = true

    g.state.scope.nest_scope scope
    blk_g.push_state scope

    blk_g.state.push_super g.state.super
    blk_g.state.push_eval g.state.eval
    blk_g.state.push_name blk_g.name

    blk_g
  end
end

RATIONALE.md

For language developers on the Rubinius platform, the APIs to the Rubinius generator and other compiler parts are generally very smooth and easy to use. An exception to this is when trying to generate a function/closure. Messing with the low-level APIs turns out to be unwieldy, as they tend to expose quite a lot of detail about Ruby-specific semantics.

The goal of this proposal is to provide a higher-level API optimized for a more general use case.

The file ast.rb contains the user code, and the file new_api.rb contains the support code to make it work (what would end up being implemented in some rubinius gem).

One thing that could be confusing about this is for_block. A "block" is a somewhat odd name for the thing that Ruby creates. It's usually called a closure. Further, while a "block" in Ruby tends to be used to reference a lexical closure that isn't mobile, we know that a method can take that, turn it into a Proc, and pass it anywhere. So we have this really weird "block", "lambda", "Proc", and sometimes "closure" nomenclature.

Ideally, we wouldn't propagate this confusion. :)

In the AST there is already the concept of "scope" and all executable contexts are uniformly CompiledCode instances. I think combining the scope and code into an object may be a useful API. It reduces to asking, "What kind of scope am I in, open or closed?" and "What is the code (ie actual set of instructions) that will execute in this scope?"

Thoughts?