rubykaigi2019-ujihisa-play-with-local-vars.md

Play with local vars

While waiting for my talk, guess what does it output without actually running it.

---

require 'continuation'
def foo(x = (return callcc {|c| return c }; x))
  'ruby'
end
p foo['vim']

Play with local vars

I assume you know about

• Ruby syntax
• parse.y
• YARV ISeq
• eval
• Binding
• AST Transformation
• TracePoint
• Continuation

I don't assume you know about

• Local variables

Play with local vars

Abstract

This 40min talk is only about Ruby's local variables. I'm not going to talk about anything else.

I'll demonstrate, or more precisely, play with Ruby local variables, including a discussion about the following common pitfall:

eval 'a = 1'
eval 'p a' #=> NameError!

Ruby has multiple different types of variables: global vars, instance vars, class vars, constants, local vars, and pseudo vars such as self. Identifiers without sigils such as @ are considered either local vars, methods, method arguments, block parameters, pseudo variables, or other reserved words. Particularly important aspect I believe is that Ruby intentionally tries to make local vars and methods indistinguishable for human. This talk focuses only on local vars among other variables or methods.

Keywords: parse.y, binding, yarv iseq, continuation, flip-flop operator, regular expression, and gdb Oh by the way, the whole presentation is likely going to be done inside my Vim.

• showtime.vim
• quickrun.vim

def f(x)
  x + 1
end
p f(23)

Tatsuhiro Ujihisa

https://twitter.com/ujm https://github.com/ujihisa

• 7 patches to Ruby
• 4 patches to Vim

asakusarb.png

Past talks

• RubyKaigi 2008 vim

• RubyKaigi 2009 vim

• RubyConf 2009 parse.y

• (RubyKaigi 2017 LT) vim

• RailsDM 2018 vim (C)

• VimConf 2018 vim (C)

• RailsDM 2019 rails (practical!)

• RubyKaigi 2019 local vars (new!)

Work

Quipper LTD, https://www.quipper.com

• Rails apps
• Elixir microservices

Introduced Darklaunch (feature toggles), (somewhat) circuit breaker, and canary release Separate release from deploy

Quipper Limited

• Education
  • https://learn.quipper.com
  • https://learn.studysapuri.jp
• Global
• We are hiring!

Where

• Vancouver, BC, Canada: 2009~2016
• Tokyo, Japan: 2016~2019
• Vancouver, BC, Canada: 2019~

My favourite Ruby features

• continuation
• default parameter
• range: flip-flop operator
• local var

Play with local vars

Talk structure

Motif:

x = 123
p x

1. local var
2. local var
3. local var
4. local var ...

What's Ruby's local variable

x = 123
p x

What's Ruby's local variable

def f(x)
  p x
end
f(123)

Special ones

• numbered parameters @1, @2, @3
• speial local variables self, $1, ...

Scope

x = 1
def f()
  x = 2
  1.times do
    y = 3
    # x == 2, y == 3
  end
  # x == 2
end
f()
# x == 1

(parse.y local_push / dyna_push)

Background: these work

x = 123
p x
######################
eval('x = 123
      p x')
######################
x = 123
eval 'p x'

Q. Why doesn't this code work?

eval 'x = 123'
eval 'p x'

HINT: eval (1/2)

eval(str) == eval(str, binding()) == binding().eval(str)

Binding: execution context at particular place in the code (=~ Hashmap of local vars)

HINT: eval (2/2)

eval(string)
binding().eval(string)

b.eval('x') =~ b.local_variable_get(:x)
b.eval('x = 123') =~ b.local_variable_set(:x, 123)
b.eval('local_variables') =~ b.local_variables

A. Defined only in the first binding

b1 = binding()
b1.eval('x = 123')
b2 = binding()
b2.eval('p x')

This works!

b = binding()
b.eval('x = 123')
b.eval('p x')

Q. Why doesn't this code work?

eval 'x = 123'
p x

Q. Why doesn't this code work? (equivalent variants)

require 'file_a'
p x

(where file_a.rb has x = 123)

require alternatives:

• load
• eval(File.read('file_a.rb'))

FYI: These work instead

eval '@x = 123'
p @x

eval '@@x = 123'
p @@x

eval 'X = 123'
p X

eval '$x = 123'
p $x

FYI: This works too!

eval 'def x; 123; end'
p x

Short answer

Because local vars are static.

eval 'x = 123' # dynamic
p x # x is a method, statically

• Ruby is dynamic
  • def creates a method when it runs
  • method_missing, const_missing
• Lexical scope vs object-oriented scope
  • vars: local, instance, class, special, constant
• Local vars vs methods
  • f

Which phase?

Where does Ruby detect if an identifier is a local var?

+---+    +------+    +---+    +----+
|.rb| -> |tokens| -> |ast| -> |iseq| ->
+---+    +------+    +---+    +----+
      lex        parse   compile     run

Spoiler alert: parse

Observe the lexer

require 'ripper'
pp Ripper.lex(<<~EOS)
x = 123
p x
EOS

:on_ident

Observe the parser (1/4)

require 'ripper'
pp Ripper.sexp(<<~EOS)
x = 123
p x
EOS

:var_ref

Observe the parser (2/4)

pp RubyVM::AbstractSyntaxTree.parse(<<~EOS)
x = 123
p x
EOS

LVAR

Observe the parser (3/4)

pp RubyVM::AbstractSyntaxTree.of(-> do
  x = 123
  p x
end)

DVAR

Observe the parser (4/4)

#!ruby --dump=parsetree
x = 123
p x

NODE_DVAR

node.c dump_node()

Observe the iseq (1/3)

#!ruby --dump=insns
x = 123
p x

0005 getlocal_WC_0 x@0

Observe the iseq (2/3)

pp RubyVM::InstructionSequence.compile(<<~EOS).disasm
x = 123
p x
EOS

0005 getlocal_WC_0 x@0

Observe the iseq (3/3)

pp RubyVM::InstructionSequence.compile(<<~EOS).to_a
x = 123
p x
EOS

[:getlocal_WC_0, 3]

Summary of the ways to observe

• ast

--dump=parsetree
pp Ripper.sexp(str)
pp RubyVM::AbstractSyntaxTree.parse(str)
pp RubyVM::AbstractSyntaxTree.of(block)

• iseq

--dump=insns
puts RubyVM::InstructionSequence.compile(string).disasm
puts RubyVM::InstructionSequence.of(block).disasm

pp RubyVM::InstructionSequence.compile(string).to_a
pp RubyVM::InstructionSequence.of(block).to_a

Challenge: count local vars

How many local vars are there?

x = 1
def f(x)
  1.then do |x|
    y = 2
    2.then do |x|
      y = 3
      a = 9
    end
    p y
  end
end
x = 4

Use ISeq

def count_locals(node)
  if node.first == "YARVInstructionSequence/SimpleDataFormat"
    locals = node[10]
    locals.size + count_locals(node[13])
  else
    node.
      select {|x| x.respond_to?(:each) }.
      map {|x| count_locals(x) }.
      sum
  end
end

p count_locals(RubyVM::InstructionSequence.compile(<<EOS).to_a)
x = 1
def f(x)
  1.then do |x|
    y = 2
    2.then do |x|
      y = 3
      a = 9
    end
    p y
  end
end
x = 4
EOS

Use ISeq (pattern matching)

def count_locals(node)
  case node
    in ['YARVInstructionSequence/SimpleDataFormat', _, _, _, _, _, _, _, _, _, locals, _, _, child]
    locals.size + count_locals(child)
    else
      node.
        select {|x| x.respond_to?(:each) }.
        map {|x| count_locals(x) }.
        sum
  end
end

p count_locals(RubyVM::InstructionSequence.compile(<<EOS).to_a)
x = 1
def f(x)
  1.then do |x|
    y = 2
    2.then do |x|
      y = 3
      a = 9
    end
    p y
  end
end
x = 4
EOS

Anyways

eval 'x = 123' # dynamic
p x # x is a method, statically

How can we make x to be local var?

Declaration and set

• Declare: add an identifier to local var table of the current scope
• Set: change an already-declared var value

x = 123 # declaration and set
x = 234 # set

Ways to set (modify)

• x = 123 and etc (next pages)
• eval 'x = 123'
• binding.local_variable_set(:x, 123)
  • TracePoint
  • tmp = 123; tmp #=> nil

Ways to declare: assign

• Assign
  • x = 123
• Massign too
  • a, b, c = [1, 2, 3]
• Opassign too
  • a += 1 is a = a + 1
• Right assign
  • 123 👉 x

↑Basically same

• a.b = 1 is not

Ways to declare: parameter

def f(x)
  ...
end

5.times do |x|
  ...
end

-> (x) {
  ...
}

This also creates a scope

CHALLENGE

Declare a var in the current scope, without the assign "="

... # insert code here!
p x # 123

Note: x really has to be a local var Note: Do not modify outside "..."

4 Solutions

Declaration is static

if false
  x = 234
end
p x

Declaration happens during parse

x = 234 if false
p x

Iseq optimization

== disasm: #<ISeq:<main>@/tmp/vq8o7Io/73:1 (1,0)-(6,3)> (catch: FALSE)
local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
[ 1] x@0
0000 putself                                                          (   6)[Li]
0001 getlocal_WC_0                x@0
0003 opt_send_without_block       <callinfo!mid:p, argc:1, FCALL|ARGS_SIMPLE>, <callcache>
0006 leave

Solution

eval 'x = 123'
p x

↓

x = 9 if false
eval 'x = 123'
p x

Summary so far

"Local variables are static"

Play with local vars

"Let's add local variables dynamically"

Dynamic local vars use case

Split long monolith script into scripts (NOTE: Not recommended)

• before

    x1 = 3
    x2 = 1
    x3 = 4
    ...

    f(x1, x2, x3)

• after

    # vars.rb
    x1 = 3
    x2 = 1
    x3 = 4

    # main.rb
    require 'vars'
    f(x1, x2, x3) # NameError!

Dirty hack with method_missing

Use method as proxy to local var

x = 123
define_method(:method_missing) do |name, *args|
  binding.local_variable_get(name)
end

👍? 👎?

Modify iseq at require

• See youchan's rubykaigi 2018 talk

  • "How to get the dark power from ISeq"

• See siman-man's railsdm 2019 talk

  • "Dynamic and static aspect of Ruby code analysis"

• See Yuichiro Kaneko's rubykaigi 2018 talk

  • "RNode with code positions"

• joker1007

• moris

• Kevin Deis

How?

class RubyVM::InstructionSequence
  def self.load_iseq(fpath)
    iseq = compile_file(fpath)
    ...
    iseq
  end
end

• iseq obj doesn't expose ways to modify
• to_binary / load_from_binary
• (ast node obj doesn't neither)

Modify AST at require

+---+    +------+    +---+    +----+
|.rb| -> |tokens| -> |ast| -> |iseq| ->
+---+    +------+    +---+    +----+
      lex        parse   compile     run

a.k.a. Macro

AST Transformation

Modify AST (1/)

# before
eval 'x = 123'
p x #! NameError!

# after
eval 'x = 123'
p x #=> nil

How?

Modify AST (2/)

p(x) # (FCALL@1:0-1:3 :p
     #   (ARRAY@1:2-1:3 (VCALL@1:2-1:3 :x) nil))
# trivial conversion
p((false ? x = 9 : x))

iseq:

0000 putself                                                          (   1)[Li]
0001 putself
0002 opt_send_without_block       <callinfo!mid:x, argc:0, FCALL|VCALL|ARGS_SIMPLE>, <callcache>
0005 opt_send_without_block       <callinfo!mid:p, argc:1, FCALL|ARGS_SIMPLE>, <callcache>

vs

0000 putself                                                          (   1)[Li]
0001 getlocal_WC_0                x@0
0003 opt_send_without_block       <callinfo!mid:p, argc:1, FCALL|ARGS_SIMPLE>, <callcache>

Modify iseq at require

+---+    +------+    +---+    +----+
|.rb| -> |tokens| -> |ast| -> |iseq| ->
+---+    +------+    +---+    +----+
      lex        parse   compile     run

• RubyVM::InstructionSequence.load_iseq
• RubyVM::AbstractSyntaxTree.parse_file(fpath)
• RubyVM::AbstractSyntaxTree::Node# first_lineno/first_column/last_*
• iseq -> file -> string -> ast -> string -> iseq

class RubyVM::InstructionSequence
  def self.load_iseq(fpath)
    code = File.readlines(fpath)

    ast = RubyVM::AbstractSyntaxTree.parse_file(fpath)
    traverse(ast) do |node|
      if node.type == :VCALL
      var_name = node.children.first
        if node.first_lineno == node.last_lineno
          code[node.first_lineno - 1][node.first_column..node.last_column] = "(false ? #{var_name} = 9 : #{var_name})"
        else
          raise NotImplementedError
        end
      end
    end
    RubyVM::InstructionSequence.compile(code.join)
  end

  private_class_method def self.traverse(node, &block)
    if RubyVM::AbstractSyntaxTree::Node === node
      yield node
      if node.respond_to?(:children)
        node.children.each {|n| traverse(n, &block) }
      end
    end
  end
end
require '/tmp/a.rb'

It worked!

############### a.rb
eval 'x = 123'
p x
############### main.rb
class RubyVM::InstructionSequence
  ...
end
require 'a'

Modify current environment

!?

a patch to ruby

• Change x always to be local var ref

• Add x into local_table

• HINT FCALL: f(), f(x), f { ... } VCALL: f

Modify AST

How does Ruby detect if an identifier is a local var?

static NODE*
gettable(struct parser_params *p, ID id, const YYLTYPE *loc)

Modify AST

Change return NEW_VCALL(id, loc); in gettable

-> Fails to compile

LiveCoding

end

• "Play with local vars"
• Thanks for people inspiring and helping me
  • asakusa.rb
  • okinawa.rb
  • ruby-kansai