ruby_subprocesses_part_3.rb

# :PROCESS: ruby, "ruby %f 2>&1"
# :PROCESS: markdown, "rdiscount"
# :PUBLISHER: blog, atompub
# :PUBLISHER: source, gist
# :BRACKET_CODE: "[ruby]", "[/ruby]"

# :TEXT:
# In parts 1 and 2 of this series, we look at some of Ruby's built-in ways to
# start subprocesses. In this article we'll branch out a bit, and examine some
# of the tools available to us in Ruby's Standard Library.  In the process,
# we'll introduce some lesser-known libraries you may not have known were
# available.
#
# ### Helpers
# 
# First, though, let's recap some of our boilerplate code.  Here's the preamble
# code which is common to all of the demonstrations in this article:
#
# :INSERT: @helpers
#
# `#hello` is the method which we will be calling in a Ruby subprocess. It reads
# some text from STDIN and writes to both STDOUT and STDERR.
# 
# `THIS_FILE` and `RUBY` contain full paths for the demo source file and the the
# Ruby interpreter, respectively.
#
# ### Method #6: Open3
#
# The Open3 library defines a single method, `Open3#popen3()`.  `popen3()` behaves
# similarly to the `Kernel#popen()` method we encountered in part 2. If you
# remember from that article, one drawback to the `#popen()` method was that it
# did not give us a way to capture the child process' STDERR
# stream. `Open3#popen3()` addresses this deficiency.
#
# `Open3#popen3()` is used very similarly to `Kernel#popen()` (or
# `Kernel#open()` with a '|' argument). The difference is that in addition to
# STDIN and STDOUT handles, `popen3()` yields a STDERR handle as well.
#
# :INSERT: @open3
#
# When we execute this code, the result shows that we have captured the
# subprocess' STDOUT output:
#
# :INSERT: $SOURCE|ruby:/6\./../---/, { brackets: ["[plain]", "[/plain]"] }
#
# ### Method #7: PTY
# 
# All of the methods we have considered up to this point have shared a common
# limitation: they are not very well-suited to interfacing with highly
# interactive subprocesses, e.g. subprocesses which prompt for responses. They
# work well for "filter"-style commands, which read some input, produce some
# output, and then exit.  But when used with interactive subprocesses which wait
# for input, produce some output, and then wait for more input (etc.), they
# often result in deadlocks.  In a typical case, the expected output is
# never produced because input is still stuck in the input buffer, and the
# program hangs forever as a result. This is why, in previous examples, we have
# been careful to call `#close_write` on subprocess input handles before reading
# any output.
#
# Ruby ships with a little-known and poorly-documented standard library called
# "pty". The "pty" library is an interface to the BSD pty devices.
#
# What is a pty device? In BSD-influenced UNIXen, such as Linux or OS X, a pty
# is a "pseudoterminal".  In other words, it's a terminal device that isn't
# attached to a physical terminal.  If you've used Linux or OS X, you've
# probably used a pty without realizing it. GUI Terminal emulators, such as
# xterm, GNOME Terminal, and Terminal.app often use a pty device behind the
# scenes to communicate with the OS.
#
# What does this mean for us? It means if we're running Ruby on UNIX, we have
# the ability to start our subprocesses inside a virtual terminal.  We can then
# read from and write to that terminal as if our program were a user sitting in
# front of a terminal, typing in commands and reading responses.
#
# Here's how it's used:
#
# :INSERT: @pty
#
# And here is the output:
#
# :INSERT: $SOURCE|ruby:/7\./../---/, { brackets: ["[plain]", "[/plain]"] }
#
# There are a few of points to note about this code.  First, we don't need to
# call `#close_write` or `#flush` on the process input handle. However, the
# newline at the end of "Hello from parent" is essential.  By default, UNIX
# terminal devices buffer input until they see a newline.  If we left off the
# newline, the subprocess would never finish waiting for intput.
#
# Second, because the subprocess is running asynchronously and independantly
# from the parent process, we have no way of knowing exactly when it has
# finished reading input and producing output of its own.  We deal with this by
# buffering output until we see a marker ("DONE").
#
# Third, you may notice that "Hello from parent" appears twice in the
# output. That's because another default behavior for UNIX terminals is to echo
# any input they receive back to the user.  This is what enables you to see what
# you've just typed when working at the command line.
#
# You can alter these default terminal device behaviors using the Ruby "termios"
# gem.
#
# Note that both STDOUT and STDERR were captured in the subprocess output. From
# the perspective of the pty user, standard output and standard error streams
# are indistinguishable - it's all just output.  That means using "pty" is the
# only way to run a subprocess and capture standard error and standard output
# interleaved in the same way we would see if we ran the process manually from a
# terminal window.  Depending on the application, this may be a feature or a
# drawback.
#
# You can execute `PTY.spawn()` without a block, in which case it returns an
# array of output, input, and PID.  If you choose to experiment with this style
# of calling `PTY.spawn()`, be aware that you may need to rescue the
# `PTY::ChildExited` exception, which is thrown whenever the child process
# finally exits.
#
# If you're interested in reading more code which uses the "pty" library, the
# Standard Library also includes a library called "expect.rb". expect.rb is a
# basic Ruby reimplementation of the "expect" utility written using "pty".
#
# ### Method #8: Shell
#
# Perhaps more obscure even than the "pty" library is Ruby's Shell
# library. Shell is, to my knowledge totally undocumented, and rarely used.
# Which is a shame, because it implements some interesting ideas.
#
# Shell is an attempt to emulate a basic UNIX-style shell environment as a DSL
# within Ruby. Shell commands become Ruby methods, command-line flags become
# method parameters, and IO redirection is accomplished via Ruby operators.
#
# Here's an invocation of our standard example subprocess using Shell:
#
# :INSERT: @shell
#
# And here is the output:
#
# :INSERT: $SOURCE|ruby:/8\./../---/, { brackets: ["[plain]", "[/plain]"] }
#
# We start by defining the Ruby executable as a shell command by calling
# `Shell.def_system_command`.  Then we instantiate a new Shell object. We
# construct the subprocess within a `Shell#transact` block. To have the process
# read a string from the parent process, we set up a pipeline from the `#echo`
# built-in to the Ruby invocation. Finally, we ensure the process is finished
# and collect its output by calling `#to_s` on the transaction.
#
# There is a lot going on here, and yet it's a very simple example of Shell's
# capabilities.  The Shell library contains many Ruby-friendly reimplementations
# of common UNIX userspace commands, and a lot of machinery for coordinating
# pipelines of concurrent processes.  If your interest is piqued I recommend
# reading over the Shell source code and experimenting within IRB.  A word of
# caution, however: the Shell library isn't maintained as far as I know, and I
# ran into a couple of outright bugs in the process of constructing the above
# example.  It may not be suitable for use in production code.
#
# Note that the child process' STDERR stream is shared with the parent, not
# captured as part of the process output.
#
# ### Conclusion
#
# In this article we've looked at three Ruby standard libraries for executing
# subprocesses.  In the next and final(?) article we'll examine some publicaly
# available Rubygems that provide even more powerful tools for starting,
# stopping, and interacting with subprocesses within Ruby.
#
# :SAMPLE: helpers
require 'rbconfig'

$stdout.sync = true

def hello(source, expect_input)
  puts "[child] Hello from #{source}"
  if expect_input
    puts "[child] Standard input contains: \"#{$stdin.readline.chomp}\""
  else
    puts "[child] No stdin, or stdin is same as parent's"
  end
  $stderr.puts "[child] Hello, standard error"
  puts "[child] DONE"
end

THIS_FILE = File.expand_path(__FILE__)

RUBY = File.join(Config::CONFIG['bindir'], Config::CONFIG['ruby_install_name'])
# :END:

if __FILE__ == $PROGRAM_NAME

  # :SAMPLE: open3
  puts "6. Open3"
  require 'open3'
  include Open3
  popen3(RUBY, '-r', THIS_FILE, '-e', 'hello("Open3", true)') do
    |stdin, stdout, stderr|
    stdin.write("hello from parent")
    stdin.close_write
    stdout.read.split("\n").each do |line|
      puts "[parent] stdout: #{line}"
    end
    stderr.read.split("\n").each do |line|
      puts "[parent] stderr: #{line}"
    end
  end
  puts "---"

  # :SAMPLE: pty
  puts "7. PTY"
  require 'pty'
  PTY.spawn(RUBY, '-r', THIS_FILE, '-e', 'hello("PTY", true)') do
    |output, input, pid|
    input.write("hello from parent\n")
    buffer = ""
    output.readpartial(1024, buffer) until buffer =~ /DONE/
    buffer.split("\n").each do |line|
      puts "[parent] output: #{line}"
    end
  end
  puts "---"

  # :SAMPLE: shell
  puts "8. Shell"
  require 'shell'
  Shell.def_system_command :ruby, RUBY
  shell = Shell.new
  input  = 'Hello from parent'
  process = shell.transact do
    echo(input) | ruby('-r', THIS_FILE, '-e', 'hello("shell.rb", true)') 
  end
  output = process.to_s
  output.split("\n").each do |line|
    puts "[parent] output: #{line}"
  end
  puts "---"

end