Alternative expectation interface for MiniTest and RSpec

expectations.md

Expectations

I took the ideas presented here and built a gem called Mustard. Check it out!

There are several expectation/assertion interfaces available for writing tests/specs. Here are some issues I have with them.

Test::Unit/MiniTest

• The order of assert_equals feels backwards
• Oh wait, that should be assert_equal (that too)
• I like to write the subject first

MiniTest::Spec

• Adds 30+ methods to Object
• Any matchers you want to add are another method on Object
• Still too few matchers, so it's ugly: foo.must_be :<, 10

Bacon

• too.many.method.calls.to.simulate.english

RSpec Should

• The space makes for some awkward syntaxes that cause warnings (when used with ==, >, etc.)
• eq() and such aren't bad, but may need parenthesis
• Pollutes the spec context with methods for each matcher

I haven't used the RSpec expect() interface enough to know cons.

Proposed Solution

# should returns an object which can have matchers called on it
5.should.eq 5
5.should.equal 5

# should_not can be used to do a reverse match
5.should_not.equal 7

# some comparison matchers
5.should.be_less_than 8
5.should.be_greater_than 3
5.should.be_lt 8
5.should.be_gt 3
5.should.be_gte 5
true.should.be_true
false.should.be_false

# call the method and see if it is true
[].should.be :empty?
5.should.be :between?, 3, 7

# add matchers
Should.matcher :be_empty do # any args will be passed into block
  empty? # runs in the context of the object and returns true/false
end
[].should.be_empty
[].should_not.be_empty # message: Expected [] to not be empty

# add matcher through class
Should.matcher :be_empty, BeEmptyMatcher
class BeEmptyMatcher
  # methods in here for defining behavior and messages.
end

I like this because it does not pollute the Object space much (just shouldand should_not). Also every method called after it is simply a matcher. No crazy method chains.

What do you think? If there's enough interest I'll write this up as a plugin for both RSpec-Core and MiniTest.

This looks great!!

+1

RSpec has always felt a little heavy to me for some reason. I like minitest but I like the spec style and I feel like I spend more time looking up the methods than I do writing tests, something about it doesn't feel natural and I often feel like I'm doing it wrong.

That being said, this looks pretty clean and solves a lot of your issues and looks like it would solve some of mine, but ruby is so bloated with testing frameworks, plugins for those frameworks, runners, etc, and this would just add another dev dependency to your gemspec. For new users coming in, the options are confusing and overwhelming. I'd rather see something like this get merged into minitest proper – but I guess it'd have to be a plugin first.

👍

👍

Not calling methods on a class in a test seems crazy to me. [].should.be :empty? is against the spirit of TDD where you use the class/code you are developing first. No one calls methods like that - [].empty?.should.be true is close, but still, assert [].empty? is pretty darn clear and it's very close to how the code under test would actually be used.

Even though it's hard to remember the order for assert, it's incredibly clear as to what's being tested and it's very simple to implement and use:

assert_equal value_i_got, expected_value
assert_equal expected_value, value_i_got

no matter which is right, it's clear what you are asserting.

I'd be interested in seeing a cut of this. I like how the expectation is tied directly to should and the simplicity in adding additional matchers.

👍

minitest/unit:

1. It is assert_equal. Go look again.
2. It is almost always expected value on the left, actual value on the right. That's in part because it makes it a lot more readable when you have:

assert_equal 42, a.b.c

vs.

assert_equal a.b.c, 42

You may not see it in this example, but make the expected value about 60 chars long and you will.

minitest/spec:

1. export MT_NO_EXPECTATIONS=1 and have at it.
2. Personally, I don't like spec style. But I hate the idea of wrapping up every value with an extra method call just for the sake of "cleanliness". val(o).must_equal 42? No thank you.
3. What matchers do you propose? (glances at empty issue tracker)

your proposal:

1. Why add should? Because it is only two (w/ negation) methods? In my opinion, every level of indirection just makes error reporting more vague and debugging that much harder. I purposefully dropped all proxies/monads and went for as-direct an approach as possible with minitest/spec.
2. People seem to appreciate minitest/spec's design overall, but I can understand that some might not. That's why we added the environment variable to turn off the top level methods. They wanted to use describe/it but with assertions.

I call bullshit on the bacon comment to be honest, as here is a working implementation in bacon, from your example:

require 'bacon'

class Fixnum
  alias less_than? <
  alias lt? <
  alias greater_than? >
  alias gt? >
  alias gte? >=
end

should "pass the assertions" do
  # should returns an object which can have matchers called on it
  5.should.eql 5
  5.should.equal 5

  # should_not can be used to do a reverse match
  5.should.not.equal 7

  # some comparison matchers
  5.should.be.less_than 8
  5.should.be.greater_than 3
  5.should.be.lt 8
  5.should.be.gt 3
  5.should.be.gte 5
  true.should.be.true
  false.should.be.false

  # call the method and see if it is true
  [].should.be :empty?
  5.should.be :between?, 3, 7

  # add matchers
  class Should
    def be_empty(*args) # any args will be passed into block
      @object.empty? # runs in the context of the object and returns true/false
    end
  end
  [].should.be_empty
  [].should.not.be_empty # message: Expected [] to not be empty
end


# If you really want to remove the dots, patching Should#method_missing for be_
# is trivial, and so is adding should_not.

Bacon is ready for prime time today

It is only 354 lines.

If we deprecated TAP output and a few other things no one uses, then it's under 300 lines.

There is a minor amount of pollution that could be removed, but really it's no more than things you load in most commercial ruby apps.

@raggi I don't like Bacon because I do not understand what some methods are doing. Methods like be, what behavior is this adding? Also, matchers don't feel cleanly separated.

In my solution there are only two method calls: should and a matcher. That feels much simpler to me.

Also isn't Bacon a test framework in itself, not something designed to be a part of RSpec/MiniTest?

@johngrimes I don't think it's worth writing if it's just for myself - the RSpec syntax isn't that bad (still need to try except more). I was also wondering if there's already a project out there with near identical interface that I didn't know about.

@dsilver829 what I mean by "Pollutes the spec context with methods for each matcher" is that if you run self.methods in an it block you will get a ton of methods, each one being a matcher. I don't think matchers should be at this level. I've also run into some conflicts with matchers when I define let blocks in RSpec.

@zenspider I had considered proposing additional matchers to minitest/spec but feel it is adding too much to Object already. The motivation for my solution is moving minitest/spec style matchers off of Object. Surely 2 methods on Object is better than 30+?

Also, this solution has a consistent should_not variation so each matcher automatically has an inverse. No need to make two matchers each time.

@ryanb,

eq() and such aren't bad, but may need parenthesis

RSpec's eq does not need parentheses. My team regularly useseq` without parens.

You might be referring to the statement from Myron and Ilya that == causes warnings in verbose mode. I've actually never been able to reproduce that warning so I'm not sure what they're talking about.

# rspectest.rb 
describe 'thing' do
  it 'should thingy' do
    3.should == 3
  end
end

$ ruby -w -W2 `which rspec` rspectest.rb 
.

Finished in 0.00029 seconds
1 example, 0 failures

No warning in the output above. Tested in 1.9.3, 1.8.7, and REE.

@BMorearty,

Try this:

describe 'thing' do
  it 'should thingy' do
    3.should == 3
    3.should == 3
  end
end

@ryanb,

I've never had any problems with MiniTest's pollutions. Methods on the test class all start with assert_ or refute_; methods on Object all start with must_ or wont_. Never had any collisions.

One of the best thing with Test::Unit-style assertions is that it's so simple to add custom assertions: You just write Ruby. def assert_response is all you need.

I also appreciate how assert_ allows you to write custom failure messages.

I completely agree @ryanb, object pollution is bad. I have tried out expect().to syntax, and it felt right (sorry for xUnit advocates, I hate .that syntax, but it's OK, I don't feel I know better). Maybe both styles could be merged like expect(true).to.be_false. This way you could add a sinfle method to 'it' namespace, two methods to expect's returning namespace, and you could put matchers inside another one. Full separation.

I kind of never really liked RSpec. One ugly column of assertions in Test::Unit would result in one ugly long line of RSpec. Would be great to have something in the middle with a beautiful codebase and stil being crazy fast.

@ryanb be does essentially the same as it does in other frameworks, it provides syntax sugar. In Bacon, be has a few extra useful side effects:

  5.should.be -> (n) { n == 5 }
  5.should.be  { |n|   n == 5 }
  5.should.not.be.nil?
  5.should.not.nil?

  5.should.be("the product of some complex math") do |n|
    # do some complex math with n
    false
  end
  # Bacon::Error: the product of some complex math

The advantage of the code being so short, is that it's very readable.

If you want rspec style matchers, just include methods into the Should class and you're away. It's no more complex than that.

I find it somewhat confusing that you claim you cannot understand what some of bacons methods are doing, when the rspec equivalents are doing much much more in order to achieve their logic. The matcher proxies are MUCH more complicated, I really cannot believe anyone would describe them as being simpler than the bacon implementation.

In bacon almost all assertions you write, in normal usage (e.g. your original examples) execute in this class:

https://github.com/chneukirchen/bacon/blob/master/lib/bacon.rb#L290-L351

That's all of it. ALL OF IT. There is no more.

By contrast, in rspec, the same stuff goes through:

https://github.com/rspec/rspec-core/blob/master/lib/rspec/core/subject.rb#L52-L79
https://github.com/rspec/rspec-expectations/blob/master/lib/rspec/expectations/handler.rb#L15-L61
https://github.com/rspec/rspec-expectations/blob/master/lib/rspec/matchers.rb#L227-L231
https://github.com/rspec/rspec-expectations/blob/master/lib/rspec/matchers/built_in/be.rb#L48-L92

And that's NOT all of it. That's just the parts that you really care about.
There's just no way your comments are based on reading the source here. RSpecs be implementation is very very similar, just much more complicated and indirect.

Now I use (regularly) RSpec, Bacon, MiniTest and Test::Unit. None of them are perfect (there are tradeoffs, I would not "blame" authors for any shortcomings in general use a testing system), but I really can't let the claims stand that minitest or bacon are in any way "harder to understand" than the other two. If you've ever had to debug the libraries themselves, or you've spent some time reading their internals (not just implementing a few matchers here and there), then there is no way that you can describe RSpec or Test::Unit as being "simpler" or "easier to understand", it's just not real.

Onto matchers a little more, if you study what is available, and the way that bacon actually works, you'll realize that you can also implement local scope matchers, that won't even cross-pollute your tests:

module AwesomeMatcher
  def awesome
    -> (o) { o.awesome? }
  end
end

extend AwesomeMatcher

should "be awesome" do
  5.should.be awesome
end

Alternatively, you can just implement a regular method on Should:

class Should
  def awesome?
    @object.awesome?
  end
end

should "be awesome" do
  5.should.awesome
  5.should.be.awesome
end

If you want be_ matchers, you can do that too:

module AwesomeMatcher
  def be_awesome
    -> (o) { o.awesome? }
  end
end

extend AwesomeMatcher

should "be awesome" do
  5.should be_awesome
end

See how easy that is, to be different? I cannot stress enough how this is a product of simplicity. And that simplicity of implementation actually goes further...

Proof? Ok, lets implement generic be_ (that's space be_) matchers then:

module BeDownUnder
  def method_missing(name, *args, *block)
    return super unless name =~ /^be_(.*)$/
    target = $1
    -> (o) { o.send(target) }
  end
end

extend BeDownUnder

should "be awesome" do
  5.should be_awesome
end

Does that feel closer to what you want?

Ah yes, should_not, okay:

class Object; def should_not(*args, &block); Should.new(self).not.be(*args, &block); end; end

Lets review:

• be_ matchers, that you can use space separation: check.
• should_not: check.
• no crazy method chains: check.
• under 500 LoC still: check.
• no warnings: check.
• 1.8.6 -> 2.0.0: check.

So I've implemented what you discussed, and your desires.

I dare you 😜 to implement it in a gist comment, in 10 minutes, for RSpec.

Have fun!

While we are on this topic I want to express my confusion with third-person verb usage in MiniTest. I don't understand it:

assert_equal
assert_includes
assert_match
assert_raises
assert_respond_to
assert_throws

@zenspider I had considered proposing additional matchers to minitest/spec but feel it is adding too much to Object already.

If you feel that the language of minitest is lacking, you should say so. They get implemented as assertions anyhow and just glued into Object.

The motivation for my solution is moving minitest/spec style matchers off of Object. Surely 2 methods on Object is better than 30+?

Well no, actually. If your argument is against global pollution, then anything > 0 is bad. You can't have it both ways. As pointed out elsewhere, I always scope my methods on object to must_* and wont_* so it really isn't any worse than your should/should_not.

What IS worse is the extra method calls and levels of indirection.

Also, as I've pointed out, you can disable the expectation methods in minispec. Clearly 0 methods infecting Object is better than your 2. Right? I mean... I do get to use your own logic against you, right?

Also, this solution has a consistent should_not variation so each matcher automatically has an inverse. No need to make two matchers each time.

I only make two expectations when it make sense to have both. This is the whole:

def assert_nothing_raised
  yield
end

thing. must_not_be_empty is just as useless so why provide it? Your need for consistency actually encourages bad testing and a false sense of security.

Finally (sorry)... I walked through bacon's method flow in my "Size isn't Everything" talk at cascadia ruby conf last year with a nice illustration and everything. You might want to check it out. It is a very beautiful little framework.

Compare your proposed:

Should.matcher :be_empty do
  empty?
end

With raggi's (and what would be very similar in minitest) bacon:

def be_empty(*args)
  @object.empty?
end

(comments stripped on both)

Do you see the main difference there?

In bacon/minitest you only need to know ruby. You only need to know one object system. You only need to know one way to create behavior. Pretty much anyone can look at the code and simply understand what is going on.

In rspec and your example, you need to know 2: ruby and the testing system's notion of objects, classes, and methods. I have no idea what that empty? sitting in the middle of nowhere is supposed to do to anything. I'd bet that 90% of the ppl out there give hand-wavey explanations on how rspec and your proposal actually work.

I will never wrap my head around why there is an entirely new class hierarchy, inheritance, and module system in rspec and your proposal. I will never wrap my head around why you want Should.matcher name do to be some obtuse alias for def name. Levels of indirection just make things more confusing, harder to debug and slower. But most importantly, it is much harder to teach to the people who need it most.

I mentioned this on Twitter this morning but thought I would be worth putting my thoughts here too (and over more than 140 chars).

---

I played around with similar earlier this year in https://github.com/gshutler/mustard. I never published it but it seems I had similar thoughts.

Things that motivated me:

Lack of pollution

Mustard extends Object with must and must_not, everything else hangs off the Mustard proxy.

Intuitive

RSpec and similar matchers have often tripped me up as I couldn't remember which way around spaces and statements needed to be, when parantheses became mandatory, how the be_ statement was written, etc.

Mustard works with regular boolean expressions which you would use with Ruby so there's no new syntax to learn. If your regular boolean statement is foo.vaguely_sensible? :bar then foo.must.vaguely_sensible? :bar is a working assertion.

Readability != "Is English"

We're developers, we read code for a living. I know that <= means less than or equal to so why no just use that rather than added another set of vocabulary to learn/forget?

More assertive verb

Referencing RFC 2119 SHOULD has weak connotations whereas MUST is assertive. Given that tests fail when your expectations aren't met must seems a better fit.

---

All that being said I was doing it more as a thought experiment rather than a serious effort to create an alternative and I've continued to use RSpec matchers day-to-day. However, if there would be wider interest in refining this (I stopped at the point where there were diminishing returns between effort and interest) I'd be happy to pick it up again.

I went ahead and coded my ideas out into a gem. You can find it at:

https://github.com/ryanb/musts

For me it strikes a nice balance of simplicity and utility. Feel free to use it if you like it too. If you don't, you don't have to use it. :)

Renamed the project to Mustard. It can be found here:

https://github.com/ryanb/mustard

I would prefer:

[].must.be_empty

I think is better than be :empty? What if something was supposed to be a symbol and you are explicitly looking for the symbol :empty? ??

If you're still considering interfaces have you looked at https://github.com/sconover/wrong ?

I have similar objections to the test frameworks you call out. Unfortunately I don't care for expectations either. Here's my attempt to answer the problem. MicroTest

+1 on checking Wrong out (https://github.com/sconover/wrong). I think it addresses a lot of the issues you listed.