POC "inline" contract tests with MiniTest

minitest_contracts.rb

# POC "Inline" contract tests with MiniTest
#
# The idea is to run a contract test immediately after a mock object is created.
#
# Example:
#
#   Say a `Registration` uses `Payment` (which uses `PaymentGateway`)
#
#   def test_successfull_payment
#     payment = double
#     payment.stub(:create).with(100) { true }
#     verify_contract(PaymentTest, "test_returns_true_when_successfull", payment)
#     stub_const('Payment', payment)
#   
#     Registration.create('joe', :small).must_equal true
#   end
#
# (sorry for a weird mix of minitest/unit, minitest/spec & rspec/mocks :-)

require 'minitest/autorun'

# minitest + rspec/mocks
require 'rspec/mocks'

class MiniTest::Unit::TestCase
  def setup
    RSpec::Mocks.setup(self)
  end

  def teardown
    RSpec::Mocks.verify
  ensure
    RSpec::Mocks.teardown
  end
end

# minitest/contracts
class MiniTest::Unit::TestCase
  def verify_contract(test, method, *actors)
    test.new("Contract test: #{test.name}").run_test(method, *actors)
  end
end

# code

class PaymentGateway
  def self.create(merchant_id, amount)
  end
end

class Payment
  MERCHANT_ID = 42

  def self.create(amount, options = {})
    PaymentGateway.create(MERCHANT_ID, amount)
  end
end

class PaymentTest < MiniTest::Unit::TestCase
  def test_uses_payment_gateway_correctly
    gateway = double
    gateway.should_receive(:create).with(Payment::MERCHANT_ID, 100) { true }
    stub_const('PaymentGateway', gateway)

    Payment.create(100)
  end

  def test_returns_true_when_successfull(payment = Payment)
    PaymentGateway.stub(:create, true) do
      payment.create(100).must_equal true
    end
  end

  def test_returns_false_when_failed(payment = Payment)
    PaymentGateway.stub(:create, false) do
      payment.create(100).must_equal false
    end
  end
end

class Registration
  def self.price_for(plan)
    100
  end

  def self.create(username, plan)
    Payment.create(price_for(plan))
  end
end

class RegistrationTest < MiniTest::Unit::TestCase
  def test_small_plan_costs_100
    Registration.price_for(:small).must_equal 100
  end

  def test_successfull_payment
    payment = double
    payment.stub(:create).with(100) { true }
    verify_contract(PaymentTest, "test_returns_true_when_successfull", payment)
    stub_const('Payment', payment)

    Registration.create('joe', :small).must_equal true
  end

  def test_failed_payment
    payment = double
    payment.stub(:create).with(100) { false }
    verify_contract(PaymentTest, "test_returns_false_when_failed", payment)
    stub_const('Payment', payment)

    Registration.create('joe', :small).must_equal false
  end
end

First, I applaud you for trying to do this. I'm glad that someone is trying to make this work. Even so, I don't like the idea of running a contract test on a test double instance. This feels backwards, but it might work out. I have to think about it more.

I don't like the idea of specifying in verify_contract() a test name, or even a set of test names, that the test double has to pass. When stubbing create() to return true, you need to write at least one contract test showing when create() returns true. That would mean that your test double stubbing create() to return true should pass at least one contract test. That sounds right, but feels risky. It might work. Key question: what would have to be true for a test double T to pass at least one contract test C_n in C, but break the contract of its interface? (In other words: what would cause a false positive?)

Your example here has one feature that worries me much more: your contract test for Payment includes the implementation detail of using PaymentGateway. This tells me that Payment clients want to work with the PaymentGateway directly. When I see the two PaymentTests test_returns_false_when_failed and test_returns_true_when_successfull, I see more evidence that Payment is doing nothing useful: these tests check that the client returns whatever its collaborator returns, and nothing more. Why are Payment and PaymentGateway different things? It seems like PaymentGateway implements Payment. (Worse: test_uses_payment_gateway_correctly shows an implicit dependency between Payment and PaymentGateway that destroys much of the value of contract testing.)

Keep at it. This could go somewhere interesting, and if it does, it could really help Ruby programmers who don't want omakase. :)

@kurko @jbrains thanks guys for feedback. I guess you're right, it's hard to have the double enforce the contract without complicating the double or making the contract less strict:

1. standard test:

   def test_new_customers
     @repository.add_customer 'alice'
     @repository.add_customer 'bob'
     @repository.add_customer 'carol'
   
     @repository.new_customers.map(&:name).must_equal ['carol', 'bob']
   end

2. something a double (with stubbed out add_customer) can implement:

   def test_new_customers
     @repository.add_customer 'alice'
     @repository.add_customer 'bob'
     @repository.add_customer 'carol'
   
     @repository.new_customers.size.must_equal 2
   end

Back to square one... @jbrains do you know if somebody followed-up on your idea from 2009 on static analysis for violation or lack of contracts when using mocks?

@wojtekmach I commonly use something along the lines of (with RSpec):

describe ClassA do
  it_should_behave_like "object A contract"

  # ...normal ClassA tests
end

describe ClassB do
  it_should_behave_like "object A contract"

  # ...normal ClassB tests
end

shared_examples "object A contract" do
  describe ClassA do
    subject { ClassA.new }

    it "should respond to token" do
      expect { subject.interface }.to_not raise_error NoMethodError
    end
  end
end

In this case, ClassB's instance uses ClassA's instance.