st-examples.scala

scala> import org.scalatest._
scala> import Matchers._
scala> import Inspectors._

scala> trap { List('a', 'b') should have length 1 }
res0: Throwable = org.scalatest.exceptions.TestFailedException: List(a, b) had length 2 instead of expected length 1

scala> trap { List(1, 2, 3) should contain (4) }
res1: Throwable = org.scalatest.exceptions.TestFailedException: List(1, 2, 3) did not contain element 4

scala> trap { forEvery (List(1, 2, 3)) { x => x should be < 2 } }
res2: Throwable =
org.scalatest.exceptions.TestFailedException: forEvery failed, because:
  at index 1, 2 was not less than 2 (<console>:21),
  at index 2, 3 was not less than 2 (<console>:21)
in List(1, 2, 3)

scala> import org.scalautils.StringNormalizations._
scala> trap { (Array("Doe", "Me") should contain oneOf ("X", "RAY", "BEAM")) (after being lowerCased) }
res3: Throwable = org.scalatest.exceptions.TestFailedException: Array(Doe, Me) did not contain one of ("X", "RAY", "BEAM")

secure { List('a', 'b').length == 1 }
secure { List(1,2,3).contains(4) }
secure { List(1,2,3).forall(_ < 2) }
secure { 
  Array("Doe","Me").map(_.toLowerCase).toSet
  .intersect(Set("X", "RAY", "BEAM"))
  .nonEmpty
}

Okay, obviously, no labeling there, and I appreciate that is a concern for you. But it's shorter, and I can just use the API I'm testing directly. ScalaTest has to come up with new pseudo-English names for all the operations of the API, so we have things like after being lowerCased instead of just .map(_.toLowerCase). And why is it after being lowerCased rather than following being lowerCased or after applying theLowerCaseFunction? With a pseudo-English API, these things are rather arbitrary, and with it being done via random implicits, it's not very discoverable either... that's why all else being equal, I prefer to just use the API I'm testing directly and add labeling using ordinary functions.

If there are certain common cases where we might want to attach labels, then we can add combinators for that:

def checkEmpty[A](t: Traversable[A]): Prop = 
  s"expected empty: $t" |: t.nonEmpty

def checkNonEmpty[A](t: Traversable[A]): Prop = 
  s"expected nonempty: $t" |: t.nonEmpty
// ...
// you can have as many ad hoc functions like this as you want!

// handy generic combinator
def examine[A](a: A)(f: A => Prop): Prop =
  f(a) || secure { a.toString |: false } // lazily add the label

secure { examine(List('a', 'b')) { _.length == 1 } }
secure { examine(List(1,2,3)) { _.contains(4) } }
...

I can make the labeling even more fine grained if I want, just by adding more (ad hoc) combinators. But there's no need for implicit magic, and no need for pseudo-English. Adding more combinators of this sort should be weighed against the cost of having the testing API define its own set of labeling functions, mirroring the existing API under test. (Which isn't scalable and gets rather brittle if taken to its logical endpoint...)

The reason I don't care all that much about labeling is that my tests are passing 99% of the time. If a test fails, I will almost certainly have to do some debugging anyway. It's a happy accident if I'm able to diagnose the problem just from the labels. Eagerly adding fine-grained diagnostics just doesn't (IMO) pay for itself - I can always add that info later if I need to, and it's very fast to do so. That is, I claim it's more economical to 'lazily evaluate' adding more fine-grained diagnostic information, and this diagnostic info can be added temporarily and can even take the form of random print statements which get removed as soon as the test is passing. Just like debugging print statements, you don't necessarily need to keep this info around permanently if it can be re-added very quickly in the event that your test starts failing. (Which hopefully is not very often!)

Of course, if you want to keep more fine grained diagnostic info around permanently, you can do so, but that is a judgement call I prefer to make on a case-by-case basis. I usually start with almost no diagnostics -- then the test is super easy to read, as it's just regular code, and I only add labels as needed.

Hi Paul,

I think you're talking about the tradeoffs of using Boolean expressions versus Matcher expressions for assertions. Boolean expressions is the default assertion in ScalaTest, because it is simpler. To use Matchers you have to either import them or mix them in.

Boolean expressions are indeed more concise than matcher expressions, but the tradeoff, as Mike and you are demonstrating here, is that getting a descriptive error message out of Boolean expressions is (actually, has been) more work. I am trying to improve that, but instead of labels, we've been taking the approach of using macros to extract good error messages from Boolean expressions. So far our macros are very simple. This is ScalaTest 2.0:

scala> import org.scalatest._
import org.scalatest._

scala> import Assertions._
import Assertions._

scala> val a = 1
a: Int = 1

scala> val b, c = 2
b: Int = 2
c: Int = 2

scala> trap { assert(a == b) }
res1: Throwable = org.scalatest.exceptions.TestFailedException: 1 did not equal 2

scala> trap { assert(b != c) }
res2: Throwable = org.scalatest.exceptions.TestFailedException: 2 equaled 2

See, can now get a decent error message out of a Boolean equality expression with no hassle. The plan is, in the next release actually, to give good error messages for Boolean expressions that equate to the built-in matchers. Here are some examples:

assert(a > 7) // 1 was not greater than 7
assert(List(a, b).isEmpty) // List(1, 2) was not empty
assert(a < 7 && b > 7) // 1 was less than 7, but 2 was not less than 7

I figure that this will go a very long way towards getting decent error messages for Boolean expressions with absolutely no effort on the user's part, maybe 90 or 95% of the assertions people write. For the others, I was just going to let them do the process you described, which is to go in after the fact and insert a clue string. Something like:

assert(myMethod(a)) // This failed, so I rewrite it to:

val result = myMethod(a)
assert(result, s"a was: $a")

The reason I don't like labels is I think they really clutter up the Boolean expression such that it is hard to see what the heck the Boolean expression was in the first place.

My main beef with the quasi-english COBOL-like EDSL is that if I'm looking at "shouldEqual foo after being lowerCased", I have no clue what the type of after, being, and lowerCased are. I basically have no chance of being able to discover these combinators or know what arguments they can take.

And speaking of good error messages. Compiler errors are rather obtuse when the implicit magic goes wrong. I don't exactly find the following kind of thing to be very helpful:

error: overloaded method value should with alternatives:
(beWord: MySpec.this.BeWord)MySpec.this.ResultOfBeWordForAnyRef[scala.collection.GenSeq[fgl.Node]] <and>
(notWord: MySpec.this.NotWord)MySpec.this.ResultOfNotWordForAnyRef[scala.collection.GenSeq[fgl.Node]] <and>
(haveWord: MySpec.this.HaveWord)MySpec.this.ResultOfHaveWordForSeq[fgl.Node] <and> ...

All of the implicit magic makes the test suite disproportionally difficult to understand and to modify.

Hi Runar,

I'm curious, is it the operator notation that's throwing you off? How is it that you think figuring out the types for (after being lowerCased) would be harder than for (after.being(lowerCased))? One is a method call on a variable in operator notation, the other in dots and parens notation. In either case you'll need to look up the type of after, which turns out is TheAfterWord, then look at the signature of its being method, which is:

def being[N](normalization: Normalization[N])(implicit equivalence: Equivalence[N]): NormalizingEquivalence[N]

There is indeed one type class required by being, Equivalence, but I'm not sure why you and Paul would call that "implicit magic." Equivalence is very much like Scalaz's Equal typeclass, and oddly enough actually I decided to add this in addition to Equality after my conversation with you about the problems with And and universal equality at the Mexican restaurant! Here's Equivalence:

trait Equivalence[T] {
def areEquivalent(a: T, b: T): Boolean
}

So I'm not sure I see the magic.

Anyway, I actually had the same reaction as Paul when I first saw Hamcrest matchers back in my JUnit days. I was perfectly happy with assertions and didn't see the need for the extra overhead of the Hamcrest matcher DSL. I did what he says he does: I just used bare Boolean expressions in assertions, and only if a test failed would I go back and add a clue string. That worked just fine for me.

But later, I did quite like the matchers in Ruby's RSpec when I saw them, and felt they allowed the writer of the code to communicate intent at a higher level, the level of what I'd say to you in conversation if you asked me about the code: "At this point in the test, the result should be less than 7". That's what I tried as an experiment with Matchers in ScalaTest. I saw the tradeoff being a matcher expression makes it more obvious what the intent of the writer was (so long as the reader understands English), but less obvious how it works. What I've observed in many years of users using these things is that by and large people don't care how a matcher expression works. They figure out how to write them by looking at examples, and when they read them, they just think at the high level of the statement, not at the level of the intermediate types produced by the DSL.

...Except when they get a compiler error message like the one you pasted. That is where all the "magic" bleeds out, but it isn't so much implicit magic as it is just DSL implementation magic. That is indeed a pain. It would be nice if we could teach the compiler to give better error messages in such cases.

Bill

Hi Runar,

Perhaps proving your point, I awoke this morning realizing I had grabbed the wrong overloaded being method in my previous comment. Normally the way to figure out which method is called is to follow the types. The type of lowerCased is Uniformity, which is a subtype of Normalization that can also do "universal normalization", i.e., normalize an Any. This was needed because I felt the need to support universal equality via Equality for compatibility reasons. I want ScalaTest to be easily used to test anything that can be thrown at it on the Java platform, including any kind of object written in Java or any other JVM language.

When it came time to add tunable type checks, I at first resisted having two different traits, Equality and Equivalence, and just had Equality. But that meant users would always need to handle Any both for alternate equality implementations and normalizations. That's often a pain, and after our conversation at the Mexican restaurant, and more anguished thinking about it, I decided to just have two traits each. Equality/Uniformity handle Any on the right hand side. Equivalence/Normalization can be used when both left and right are the same type, which === enables. And symmetrically, Equality extends Equivalence; Uniformity extends Normalization. The universal trait is in both cases a subtype of the more restrictive trait.

So the actual being method being called in the (after being lowerCased) case is the other overloaded being method in trait Explicitly, which looks like:

def being[N](uniformity: Uniformity[N])(implicit equality: Equality[N]): NormalizingEquality[N]

Essentially what the two overloaded being methods do is transform a normalization typeclass (either Uniformity or Normalization) into an equality typeclass (either NormalizingEquality or a NormalizingEquivalence) that first normalizes both the left and right sides, and then delegates to the implicitly grabbed equality typeclass (either Equality or Equivalence) to compare the normalized forms.

Bill

we've been taking the approach of using macros ... The plan is, in the next release actually, to give good error messages for Boolean expressions that equate to the built-in matchers

That sounds terrific! Looking forward to it.

(I currently never go near the matchers stuff, because what Paul and Runar said.)

Hi Paul,

By the way, in 2.1.0-RC3 I added toEquality and toEquivalance methods to Uniformity and Normalization, so that you need not use the English-like syntax to go, for example, from a Uniformity to an Equality. Instead of:

(result === "hello")(after being lowerCased)

You can just write:

(result === "hello")(lowerCased.toEquality)

I got the memo from multiple sources that even some of the folks who like seeing English-like DSLs in test code don't want to see it in the production code.

Bill