How Sunspot implements its wonderful search/index DSL

01_README.md

This code is extracted/adapted from Mat Brown's Sunspot gem. One of Sunspot's nicest features is an expressive DSL for defining search indexes and performing queries. It works by instance_eval-ing a block you pass into it in the context of its own search builder object. In this code, the pig thing1 statement is roughly equivalent to zoo = Zoo.new; zoo.pig(thing1).

Sunspot's DSL has to resort to trickery: the instance_eval_with_context method uses eval to get the block to give up the object it considers to be self, then sets up an elaborate system of delegates and method_missing calls so any methods not handled by the DSL are forwarded to the surrounding object. But as a result, this syntax is minimal and beautiful, and it works the way you expect whether or not you prefer blocks to yield an object.

Without this trick the block would be restricted to either the original, "calling" context (as a closure) or the DSL's "receiving" context (using instance_eval), but not both.

Using instance_eval for cleaner syntax

Carburetor.search do
  # This will raise NoMethodError; this block is executing in the context of 
  # a Carburetor search builder and has no access to data from the controller.
  keywords params[:query]
end

Simply calling the blocks, no eval

Carburetor.search do |search|
  # `instance_eval` breaks the block/closure's normal scoping behavior
  # This works because we're allowing the block to behave normally, as
  # a closure.
  search.keywords params[:query]
end

To be very clear, while I like Sunspot's syntax and prefer code that behaves in one (roughly) consistent way, there is nothing wrong with the latter, more blocky form of this code. I do think it's uglier, in the objective sense that it has a lot of repetitive information that makes this code harder to read, especially as I add statements or nest multiple levels of blocks. I also think homemade muffins are uglier than the ones they have at Starbucks, but in that case I think it's clear that uglier is better.

zoo_dsl.rb

require 'set'

module Util
  def self.instance_eval_or_call(object, &block)
    # Here we assume that if the block takes one or more arguments,
    # you want to execute the block in its own scope, yielding the
    # DSL constructor as an object. In this case the DSL would function
    # like this:
    #
    #   Zoo.setup do |zoo|
    #     zoo.pig likes: "slop"
    #   end
    if block.arity > 0
      block.call(object)

    # But that's boring. The real trick is this: we want to use this DSL
    # without yielding an argument, like this:
    #
    #   Zoo.setup do
    #     pig likes: "slop"
    #   end
    #
    # `instance_eval` is the typical way to accomplish that, eval'ing the
    # block so that `pig` and `cow` are local methods. But doing that, you
    # lose the surrounding context. If this were in a Rails controller,
    # you couldn't use the `params` hash or any of your instance variables
    # because the `Zoo` instance doesn't know about them.
    #
    # Ruby has no real concept of closures, or anything like JavaScript's
    # `Function#apply()` function. So here Mat has created a proxy/delegate
    # class, `ContextBoundDelegate`. Inside your block it looks as if you
    # have access to both the DSL object _and_ its surrounding scope, but this
    # is an illusion. You're actually calling methods on `ContextBoundDelegate`,
    # which in turn forwards them to either the DSL object (if it responds to
    # a particular message) or the surrounding scope (if it's not). I will try
    # to explain how this works below.
    else
      ContextBoundDelegate.instance_eval_with_context(object, &block)
    end
  end

  class ContextBoundDelegate
    class <<self
      # Designated initializer for `ContextBoundDelegate` -- in fact, the only
      # public initializer, as the `#new` method has been made private below.
      # It's possible you could do this same job inside of `#initialize`, but
      # I like that this method is very, _very_ clear about the fact that it's
      # eval'ing something. You're not supposed to know about `ContextBoundDelegate`;
      # it's an implementation detail of your _real_ API.
      def instance_eval_with_context(receiver, &block)
        # Extract the block's context by eval'ing `self`, so we can store it in a variable.
        calling_context = eval('self', block.binding)

        # It's possible that one ContextBoundDelegate may be nested inside another.
        # In fact, Sunspot uses this feature in a few places such as boolean queries.
        # If the calling context has a calling context of its own (i.e., if it's another
        # ContextBoundDelegate) then we need to forward messages to _that_ object instead.
        # Consequently, syntax like this will Just Work:
        #
        #   Zoo.setup do
        #     monkey_house do
        #       chimp eats: params[:chimp_food]
        #     end
        #   end
        #
        # In this example, `Zoo#monkey_house` is a DSL instance method that
        # opens a new ContextBoundDelegate, nested inside the first one. In order
        # to access the `params` object we need to still forward that message
        # to the original surrounding context. So we do.
        if parent_calling_context = calling_context.instance_eval{@__calling_context__}
          calling_context = parent_calling_context
        end

        # One reason why you never initialize this object directly is that it's
        # intended use is as a proxy. Here we construct a new ContextBoundDelegate,
        # then immediately `instance_eval` the block using _the delegate_ as context.
        # It is now aware of both our DSL and the controller around it, and can
        # serve as a kind of internal message bus, ensuring things are called on the
        # correct object.
        new(receiver, calling_context).instance_eval(&block)
      end
      private :new
    end

    BASIC_METHODS = Set[:==, :equal?, :"!", :"!=", :instance_eval,
                        :object_id, :__send__, :__id__]

    # Like all Ruby objects, ContextBoundDelegate inherits from Object,
    # and so carries a lot of baggage in the form of standard instance
    # methods. But our intent is for ContextBoundDelegate to be more or
    # less invisible, to behave as if it _is_ one of our two contexts.
    # So here we're un-defining (i.e. deleting) *all* instance methods
    # except for a few specific ones we need, listed in BASIC_METHODS
    # above. Those are limited to methods responsible for telling whether
    # any two delegate proxies are the same, plus `instance_eval`.
    instance_methods.each do |method|
      unless BASIC_METHODS.include?(method.to_sym)
        undef_method(method)
      end
    end

    def initialize(receiver, calling_context)
      @__receiver__, @__calling_context__ = receiver, calling_context
    end

    # In the case of #id, we want to short-circuit the normal proxying
    # behavior (which favors our DSL receiver over the calling context)
    # and send any #id messages directly to the caller. I didn't understand
    # why at first, but then I did: it's for ActiveRecord and other ORMs
    # where you might call `self.id` or just `id` and expect it to return
    # a record/object identifier. Consequently, one constraint on your DSL
    # syntax is that it can't use the `#id` method for anything.
    def id
      @__calling_context__.__send__(:id)
    end

    # Special case due to `Kernel#sub`'s existence. Kernel methods (such as
    # `rand()`) are forwarded directly to Kernel, bypassing `method_missing`
    # and skipping this object completely unless it implements this method
    # itself. Here we're just forcing #sub to go through our cascading proxy
    # like any other method, just in case either context cares to implement it.
    def sub(*args, &block)
      __proxy_method__(:sub, *args, &block)
    end

    # Receives any methods that aren't explicitly implemented by this proxy
    # thingy, and forwards them to `__proxy_method__`. Why is `__proxy_method__`
    # separate? So it can be reused for special cases like `#sub` above.
    def method_missing(method, *args, &block)
      __proxy_method__(method, *args, &block)
    end

    # Where the magic happens. This method is actually really simple: _every_
    # method or variable you call inside your block is run through here. The
    # proxy attempts to call it on the receiver (i.e. the DSL). If it's not
    # implemented there, it tries the calling context. If _that_ doesn't work,
    # NoMethodError is raised. It actually sends the message to the objects
    # (rather than check for them using `respond_to?()`) because you may want
    # to implement your DSL syntax using `method_missing`, or through some other
    # metaprogramming trick that might short-circuit Ruby's ways of detecting
    # the presence of a method. Hence the cascading rescue statements. With the
    # exception of #id (handled as a special case above), the receiver always
    # has precedence over the caller.
    def __proxy_method__(method, *args, &block)
      begin
        @__receiver__.__send__(method.to_sym, *args, &block)
      rescue ::NoMethodError => e
        begin
          @__calling_context__.__send__(method.to_sym, *args, &block)
        rescue ::NoMethodError
          raise(e)
        end
      end
    end
  end
end

# This is a DSL for describing a zoo, with animals and buildings.
class Zoo
  # Constructor for our zoo. It takes a block, which is passed to
  # `instance_eval_or_call`, which does all the magical things discussed
  # above. Returns our constructed Zoo object.
  def self.setup(&block)
    Zoo.new.tap do |zoo|
      Util.instance_eval_or_call(zoo, &block)
    end
  end

  def initialize
    @data = {zoo:{}}
    # The __current_target__ variable is used to track where we are in
    # the zoo. If we enter a building, like the monkey house or a barn,
    # that will need to be represented here until we come out of it.
    @__current_target__ = @data[:zoo]
  end

  # You describe animals using these declarative methods:
  #
  #   pig goes: "oink", eats: "slop"
  #
  def pig(thing)
    add_thing_to_animal(:pig, thing)
  end
  def cow(thing)
    add_thing_to_animal(:cow, thing)
  end
  def chimp(thing)
    add_thing_to_animal(:chimp, thing)
  end
  def elephant(thing)
    add_thing_to_animal(:elephant, thing)
  end

  # To enter a building, you use a handy block syntax:
  #
  #   monkey_house do
  #     orangutan is:"orange"
  #   end
  #
  # Buildings can be nested inside one another:
  #
  #   african_pavilion do
  #     elephant size:"extra-large"
  #     reptile_house do
  #       snake is:"DO NOT TALK TO ME ABOUT SNAKES"
  #     end
  #   end
  #
  def monkey_house(&block)
    add_building(:monkey_house, &block)
  end
  def barn(&block)
    add_building(:barn, &block)
  end
  def african_pavilion(&block)
    add_building(:african_pavilion, &block)
  end
  def reptile_house(&block)
    add_building(:reptile_house, &block)
  end
  
private
  
  def add_thing_to_animal(animal_name, thing)
    (@__current_target__[animal_name] ||= []) << thing
  end
  
  def add_building(building_name, &block)
    enter_building(building_name)
    Util.instance_eval_or_call(self, &block)
    exit_building
  end
  
  def enter_building(building_name)
    @__previous_target__ = @__current_target__
    @__current_target__[building_name] ||= {}
    @__current_target__ = @__current_target__[building_name]
  end
  
  def exit_building
    raise "You're not currently in a building" if @__previous_target__.nil?
    @__current_target__ = @__previous_target__
  end

end

what_the_pig_says = "oink"

the_zoo = Zoo.setup do
  pig says: what_the_pig_says # Variable from outside the block
  cow says:"moo", provides: "milk"
  monkey_house do
    chimp eats: "bananas"
  end
  african_pavilion do
    reptile_house do
      cow "Why is a cow in the reptile house???"
    end
  end
end

require 'pp'
pp the_zoo

Software development is a game of tradeoffs. So, all in all -- would you prefer Tire to include additional 50 LOC and more complexity just to be able to use the "nicer" block DSL syntax?

(I've modeled the interface upon Prawn, which I still think has a very sane approach to balancing complexity, elegance, implementation....)

Putting "nicer" in scare quotes implies that you think I'm foolish and think my desire for beautiful code is more important than your time. I would say it's less about beauty than providing a syntax that behaves in exactly one way, that better represents common use cases.

The simpler form of Tire's DSL is cleaner and demos well, but can't be used in models to store index data from model attributes, or in controllers to query that data based on params or other values. Prawn's syntax doesn't do this crazy context trick, but then again its README doesn't demonstrate both variations of the syntax, and it does bug me that its developer doesn't use the more practical Prawn::Document.new { |pdf| ... } form in his README.

You seem upset that I expected Tire to behave a certain way and phrased my stupid gist example of another library's DSL, which I prefer, as a complaint about that. You didn't tell me why your way is better, you told me I'm wrong. Is that your whole point?

I apologize for calling your DSL ugly.

You seem upset that I expected Tire to behave a certain way and phrased my stupid gist example of another library's DSL, which I prefer, as a complaint about that. You didn't tell me why your way is better, you told me I'm wrong. Is that your whole point?

No, sorry, there's nothing like that between the lines. I appreciate the concern for beatiful code, and personally value that very strongly.

As I said, I think this is about tradeoffs on every side. I consider the Sunspot DSL superior to Tire (without any irony or anything!), and I don't rule out we may end up with this "nicer" blocks. I included the demo of both syntaxes in the README precisely for the reason their behaviour differs, and was burned by this previously when using Prawn. Notice how the "blocks with arguments" syntax goes down to every block inside the Tire.search main block. It is something which hurts my eyes and fingers as well, but I don't have a) enough mental energy to focus on that when many other important features are missing, b) a really compelling argument it would make that much difference in the end.

I also didn't tell you you're wrong .) I was asking if you, personally, would prefer a more complex codebase of the library in exchange for a more beautiful code. Because I myself would not -- the codbase is complex as it is, and I even think about removing LOCs and complexity in future development, not adding them...

So my whole point was more about pointing out the tradeoffs involved: you make the core code more complex in exchange for having a more beautiful and simpler interface. In this specific case, I am not convinced this tradeoff is worth it...

Cool. Thanks for replying and clarifying, and sorry for being rude. It's early here and I'm in a bad mood for unrelated reasons. Your comments were brief, so I read more into them than was there.

I'm generally okay with complexity if I can account for it. I like that Mat was able to encapsulate this instance_eval_with_context trick into a reusable component. Now that I know how these two classes work I'd be fairly comfortable using them in a project (with ample credit to Mat, plus the standing offer of a beer next time we're in the same city).

I might even go so far as to say this approach is simpler. Your code repeats the whole if arity < 1; instance_eval; else; call algorithm in a few places. The repeated code isn't all that complex so it's not too bad, but encapsulating that conditional in a macro like instance_eval_with_context might be a good thing long term.

One interesting thing I've realized writing my own ElasticSearch library (we started with Tire for a rapid prototype, but need HTTP keepalive support, which rest-client and net/http don't offer I don't think) for the project I'm working on is how hard it can be to map Ruby syntax to Elastic's DSL. On the one hand, the JSON API is so simple, I barely need another layer between my code and the search engine. On the other hand, the query and mapping DSLs can be very complicated and I haven't hit on an abstraction yet that would make them less complicated. For the most part all the ElasticSearch libraries seem to just pass Hashes back and forth between my app and the API, which feels brittle. But it's a hard problem.

Thanks for replying and clarifying, and sorry for being rude. It's early here and I'm in a bad mood for unrelated reasons.

Absolutely nothing to apologize for! I understand, been there myself many times and done much more damage :)

Your code repeats the whole (...)

Yes, and that is seriously crazy, I know that. I intend to refactor that to some method which abstracts that. But that's simple and from the standpoint of our discussion more of a "lipstick on a pig".

What's more important is this: I think the "blocks with args" approach is simpler because it's just Ruby. Ruby behaves this way: when you know about it (the difference between instance_eval and block.call and all that), it's very transparent and not “magical”. I've been so badly burned by “magical” so many times, that I tend to err on the “pessimist” side...

Of course, then come the usual pragmatic reasons: it'd be silly to “lipstick” such minutiae when we still don't have support for search_type=scan or handling binary attachments (for instance).

(...) we started with Tire for a rapid prototype, but need HTTP keepalive support (...)

¡ATTENTION! :) There's Curb client bundlen in, which does keepalive. The HTTP client for Tire is pluggable, so you may just want to write your custom one, if you'd like the rest of the library. See https://gist.github.com/1204159 for some benchmarks.

For the most part all the ElasticSearch libraries seem to just pass Hashes back and forth between my app and the API, which feels brittle.

Obviously, but also absolutely inefficient and ugly looking from where I stand :) Have a look at http://karmi.github.com/tire/#section-83 to see how you can declare many isolated queries and then join them in one big query. I shudder to think I'd have to write that, test that, debug that...

The best thing about that is that you can easily declare queries as:

    def main_query
      lambda do |b|
        b.must { |query| query.string "published_at:#{params[:published_at]}" } if params[:published_at]
        b.must { |query| query.string "hidden:#{ params[:hidden] ? 'true' : 'false' }" }
        b.must { |query| query.string "author:\"#{params[:author]}\"" } unless params[:author].blank?
        b.must { |query| query.string params[:q], :fields => ['title','content','author'] } unless params[:q].blank?
      end
    end

See https://gist.github.com/83c93a8ba6a720979543 for the full example.

Catch me at the #elasticsearch IRC if you'd like to talk more about that...