Why make things so complicated?

Several readers reacted to my small article series on applicative assertions, pointing out that error-collecting assertions are already supported in more than one unit-testing framework.

"In the Java world this seems similar to the result gained by Soft Assertions in AssertJ. https://assertj.github.io/doc/#assertj-c... if you’re after a target for functionality (without the adventures through monad land)"

Josh McK

While I'm not familiar with the details of Java unit-testing frameworks, the situation is similar in .NET, it turns out.

"Did you know there is Assert.Multiple in NUnit and now also in xUnit .Net? It seems to have quite an overlap with what you're doing here.

"For a quick overview, I found this blogpost helpful: https://www.thomasbogholm.net/2021/11/25/xunit-2-4-2-pre-multiple-asserts-in-one-test/"

DoCh_Dev

I'm not surprised to learn that something like this exists, but let's take a quick look.

NUnit Assert.Multiple #

Let's begin with NUnit, as this seems to be the first .NET unit-testing framework to support error-collecting assertions. As a beginning, the documentation example works as it's supposed to:

[Test]
public void ComplexNumberTest()
{
    ComplexNumber result = SomeCalculation();
 
    Assert.Multiple(() =>
    {
        Assert.AreEqual(5.2, result.RealPart, "Real part");
        Assert.AreEqual(3.9, result.ImaginaryPart, "Imaginary part");
    });
}

When you run the test, it fails (as expected) with this error message:

Message:
  Multiple failures or warnings in test:
    1)   Real part
    Expected: 5.2000000000000002d
    But was:  5.0999999999999996d

    2)   Imaginary part
    Expected: 3.8999999999999999d
    But was:  4.0d

That seems to work well enough, but how does it actually work? I'm not interested in reading the NUnit source code - after all, the concept of encapsulation is that one should be able to make use of the capabilities of an object without knowing all implementation details. Instead, I'll guess: Perhaps Assert.Multiple executes the code block in a try/catch block and collects the various exceptions thrown by the nested assertions.

Does it catch all exception types, or only a subset?

Let's try with the kind of composed assertion that I previously investigated:

[Test]
public void HttpExample()
{
    var deleteResp = new HttpResponseMessage(HttpStatusCode.BadRequest);
    var getResp = new HttpResponseMessage(HttpStatusCode.OK);
 
    Assert.Multiple(() =>
    {
        deleteResp.EnsureSuccessStatusCode();
        Assert.That(getResp.StatusCode, Is.EqualTo(HttpStatusCode.NotFound));
    });
}

This test fails (again, as expected). What's the error message?

Message:
  System.Net.Http.HttpRequestException :↩
    Response status code does not indicate success: 400 (Bad Request).

(I've wrapped the result over multiple lines for readability. The ↩ symbol indicates where I've wrapped the text. I'll do that again later in this article.)

Notice that I'm using EnsureSuccessStatusCode as an assertion. This seems to spoil the behaviour of Assert.Multiple. It only reports the first status code error, but not the second one.

I admit that I don't fully understand what's going on here. In fact, I have taken a cursory glance at the relevant NUnit source code without being enlightened.

One hypothesis might be that NUnit assertions throw special Exception sub-types that Assert.Multiple catch. In order to test that, I wrote a few more tests in F# with Unquote, assuming that, since Unquote hardly throws NUnit exceptions, the behaviour might be similar to above.

[<Test>]
let Test4 () =
    let x = 1
    let y = 2
    let z = 3
    Assert.Multiple (fun () ->
        x =! y
        y =! z)

The =! operator is an Unquote operator that I usually read as must equal. How does that error message look?

Message:
  Multiple failures or warnings in test:
    1) 

  1 = 2
  false

    2)

  2 = 3
  false

Somehow, Assert.Multiple understands Unquote error messages, but not HttpRequestException. As I wrote, I don't fully understand why it behaves this way. To a degree, I'm intellectually curious enough that I'd like to know. On the other hand, from a maintainability perspective, as a user of NUnit, I shouldn't have to understand such details.

xUnit.net Assert.Multiple #

How fares the xUnit.net port of Assert.Multiple?

[Fact]
public void HttpExample()
{
    var deleteResp = new HttpResponseMessage(HttpStatusCode.BadRequest);
    var getResp = new HttpResponseMessage(HttpStatusCode.OK);
 
    Assert.Multiple(
        () => deleteResp.EnsureSuccessStatusCode(),
        () => Assert.Equal(HttpStatusCode.NotFound, getResp.StatusCode));
}

The API is, you'll notice, not quite identical. Where the NUnit Assert.Multiple method takes a single delegate as input, the xUnit.net method takes an array of actions. The difference is not only at the level of API; the behaviour is different, too:

Message:
  Multiple failures were encountered:
  ---- System.Net.Http.HttpRequestException :↩
  Response status code does not indicate success: 400 (Bad Request).
  ---- Assert.Equal() Failure
  Expected: NotFound
  Actual:   OK

This error message reports both problems, as we'd like it to do.

I also tried writing equivalent tests in F#, with and without Unquote, and they behave consistently with this result.

If I had to use something like Assert.Multiple, I'd trust the xUnit.net variant more than NUnit's implementation.

Assertion scopes #

Apparently, Fluent Assertions offers yet another alternative.

"Hey @ploeh, been reading your applicative assertion series. I recently discovered Assertion Scopes, so I'm wondering what is your take on them since it seems to me they are solving this problem in C# already. https://fluentassertions.com/introduction#assertion-scopes"

Jernej Gorički

The linked documentation contains this example:

[Fact]
public void DocExample()
{
    using (new AssertionScope())
    {
        5.Should().Be(10);
        "Actual".Should().Be("Expected");
    }
}

It fails in the expected manner:

Message:
  Expected value to be 10, but found 5 (difference of -5).
  Expected string to be "Expected" with a length of 8, but "Actual" has a length of 6,↩
    differs near "Act" (index 0).

How does it fare when subjected to the EnsureSuccessStatusCode test?

[Fact]
public void HttpExample()
{
    var deleteResp = new HttpResponseMessage(HttpStatusCode.BadRequest);
    var getResp = new HttpResponseMessage(HttpStatusCode.OK);
 
    using (new AssertionScope())
    {
        deleteResp.EnsureSuccessStatusCode();
        getResp.StatusCode.Should().Be(HttpStatusCode.NotFound);
    }
}

That test produces this error output:

Message:
  System.Net.Http.HttpRequestException :↩
    Response status code does not indicate success: 400 (Bad Request).

Again, EnsureSuccessStatusCode prevents further assertions from being evaluated. I can't say that I'm that surprised.

Implicit or explicit #

You might protest that using EnsureSuccessStatusCode and treating the resulting HttpRequestException as an assertion is unfair and unrealistic. Possibly. As usual, such considerations are subject to a multitude of considerations, and there's no one-size-fits-all answer.

My intent with this article isn't to attack or belittle the APIs I've examined. Rather, I wanted to explore their boundaries by stress-testing them. That's one way to gain a better understanding. Being aware of an API's limitations and quirks can prevent subtle bugs.

Even if you'd never use EnsureSuccessStatusCode as an assertion, perhaps you or a colleague might inadvertently do something to the same effect.

I'm not surprised that both NUnit's Assert.Multiple and Fluent Assertions' AssertionScope behaves in a less consistent manner than xUnit.net's Assert.Multiple. The clue is in the API.

The xUnit.net API looks like this:

public static void Multiple(params Action[] checks)

Notice that each assertion is explicitly a separate action. This enables the implementation to isolate it and treat it independently of other actions.

Neither the NUnit nor the Fluent Assertions API is that explicit. Instead, you can write arbitrary code inside the 'scope' of multiple assertions. For AssertionScope, the notion of a 'scope' is plain to see. For the NUnit API it's more implicit, but the scope is effectively the extent of the method:

public static void Multiple(TestDelegate testDelegate)

That testDelegate can have as many (nested, even) assertions as you'd like, so the Multiple implementation needs to somehow demarcate when it begins and when it ends.

The testDelegate can be implemented in a different file, or even in a different library, and it has no way to communicate or coordinate with its surrounding scope. This reminds me of an Ambient Context, an idiom that Steven van Deursen convinced me was an anti-pattern. The surrounding context changes the behaviour of the code block it surrounds, and it's quite implicit.

Explicit is better than implicit.

Tim Peters, The Zen of Python

The xUnit.net API, at least, looks a bit saner. Still, this kind of API is quirky enough that it reminds me of Greenspun's tenth rule; that these APIs are ad-hoc, informally-specified, bug-ridden, slow implementations of half of applicative functors.

Conclusion #

Not surprisingly, popular unit-testing and assertion libraries come with facilities to compose assertions. Also, not surprisingly, these APIs are crude and require you to learn their implementation details.

Would I use them if I had to? I probably would. As Rich Hickey put it, they're already at hand. That makes them easy, but not necessarily simple. APIs that compel you to learn their internal implementation details aren't simple.

Universal abstractions, on the other hand, you only have to learn one time. Once you understand what an applicative functor is, you know what to expect from it, and which capabilities it has.

In languages with good support for applicative functors, I would favour an assertion API based on that abstraction, if given a choice. At the moment, though, that's not much of an option. Even HUnit assertions are based on side effects.