Make googletest-like test framework from scratch

Back to 2016, I learned googletest how improving my C++ project. From that, I always want to know how it.

Let's take a look at some basic example of it.

TEST(Arithmetic, Integer) {
  ASSERT_EQ(1 + 2, 3);
}

I think that's pretty easy to understanding for anyone ever wrote a unit test.

In the googletest context, you would get a lot of TEST there like:

TEST(Suit1, test1) {
  // ...
}
TEST(Suit1, test2) {
  // ...
}

TEST(Suit2, test1) {
  // ...
}
TEST(Suit2, test2) {
  // ...
}

Back to our article, think about it, why the compiler does not complain TEST be redefined so many times, and never specify the type of parameters(and the return type)?

Answer: Because that is not a function.

Then the question became: what it that?

The answer is clear, it's a C macro! Only the macro can behave like a function call format TEST(Suit, Test).

As everyone knows, C macro just expands codes inside of it. So the question changed again and now it's: What kind of codes can be valid after expanded with a code block at the global level?

The answer to this also it easy: a function(or a method implementation because we're using C++).

After so many words, let's start coding! Create a file unittest.hpp, code:

#ifndef UNITTEST
#define UNITTEST

#define TEST(suit, test)

#endif

So we can have a main.cpp with code:

#include "./unittest.hpp"

TEST(Arithmetic, Integer)

Then compile: g++ main.cpp, this should fine now.

Then we add a code block:

TEST(Arithmetic, Integer) {}

Oops!

$ g++ main.cpp
main.cpp:3:27: error: expected unqualified-id
TEST(Arithmetic, Integer) {}
                          ^
1 error generated.

Let's fix it. In your unittest.hpp, typed:

#define TEST(suit, test) void foo()

This time, compiler very happy without any complains. So we have to go to the next step: How to executing these tests automatically?

To do that, we must have a place that stores(or reference to) our tests. So we create a global variable.

#include <functional> // for std::function
#include <vector> // for std::vector

std::vector<std::function<void()>> all_test;

And add an insertion call in macro TEST:

#define TEST(suit, test)                                                       \
  void foo();                                                                  \
  all_test.push_back(foo);                                                     \
  void foo()

$ g++ -std=c++11 main.cpp
main.cpp:3:1: error: unknown type name 'all_test'
TEST(Arithmetic, Integer) {}
^
././unittest.hpp:11:3: note: expanded from macro 'TEST'
  all_test.push_back(foo);                                                     \
  ^
main.cpp:3:1: error: cannot use dot operator on a type
././unittest.hpp:11:11: note: expanded from macro 'TEST'
  all_test.push_back(foo);                                                     \
          ^
2 errors generated.

But it won't work, let's see what happened here. The error message is about compiler expects there is a type all_test but didn't, then it complains a type name can't contain ..

To bypass the error and get expected insertion call we need some interesting trick. It's all about C++ constructor promised to be called while the structure is created.

struct unittest_insert {
  unittest_insert(std::function<void()> &&f);
};

unittest_insert::unittest_insert(std::function<void()> &&f) {
  all_test.push_back(f);
}

#define TEST(suit, test)                                                       \
  void foo();                                                                  \
  unittest_insert ut{foo};                                                     \
  void foo()

Now, let's add some print statement into our test and implements run all tests to prove what have we done is workable. The content of main.cpp:

#include <iostream>

#include "./unittest.hpp"

TEST(Arithmetic, Integer) {
  std::cout << "test "
            << "test" << std::endl;
}

int main() {
  run_all_tests();
  return 0;
}

Implementation of run_all_tests:

void run_all_tests() {
  for (auto test : all_test) {
    test();
  }
}

Now we knew how to run tests. We need to know how to determine a fail. That's why we need assertion macros. Here is an example of testing:

TEST(Arithmetic, Integer) { ASSERT_EQ(1, 1); }

Then see how ASSERT_EQ be made.

#define ASSERT_EQ(le, re)                                                      \
  if (le != re) {                                                              \
    throw "assert equal failed";                                               \
  }

g++ -std=c++11 main.cpp and run, where interesting thing is what if you write ASSERT_EQ(1, 2) that you would get a runtime error says:

libc++abi.dylib: terminating with uncaught exception of type char const*
[1]    35777 abort      ./a.out

But before going to improving our error reporting, we should think about a problem: Can we create the second one test? The answer is NO.

You can have a try then get a list of redefinition errors from the compiler. To solve the problem we need to get some help from the macro.

#define TEST(suit, test)                                                       \
  void utf_suit##test();                                                       \
  unittest_insert ut_suit##test{utf_suit##test};                               \
  void utf_suit##test()

##, the magic from the macro, you can get more helps from https://stackoverflow.com/questions/4364971/and-in-macros

Now, we won't get the error from expanding the macro twice. And could get a chance to stop and think about the error reporting design.

At now, we just throw a char const *, and we didn't catch the exception, so we would receive a terminating with uncaught exception error. That caused two problems:

• The test won't keep going caused users won't know how many tests failed actually.
• And users don't know what exception been throw actually.

To solve the problem, what the thing we should do is catch the exception, report and keep going on. Here is the code:

#include <exception>

void run_all_tests() {
  for (auto test : all_test) {
    try {
      test();
      std::cout << "test pass" << std::endl;
    } catch (char const *e) {
      std::cerr << e << std::endl;
    } catch (std::exception &e) {
      std::cerr << e.what() << std::endl;
    } catch (...) {
      std::cerr << "test failed: unknown exception" << std::endl;
    }
  }
}

Now it would report test passed or not, it looks nice as a proof of concept.

Now counting what do we learn from these?

• How to generate a function from the macro
• How to handle the exception

And here have some exercises you can do.

• improve the error reporting, not just say assertion failed, also show the expression why isn't equal(hint: custom exception)
• how to avoid user access the generated function?(hint: try generating class)
• can we report the coverage? (hint: llvm-cov or similar things)
• what if the input expression is not comparable?