Our team supports nine applications out of the same code base (achieved by a combination of configuration, feature toggles and CSS magic). This code base has been evolving continuously over the last five years and we do at least one release every week and often more than that. Given this scenario, you can imagine how vital a role that unit tests play.

We depend a lot on our unit tests (among other things, of course) to ensure that releases go smoothly and that, when we add that shiny new feature that enables the customer to change her seat, it does not break the feature that lets her get the ticket on her mobile! To achieve this, our team adheres strictly to TDD and we have over 10,000 unit test cases that are run every time a commit is pushed to github and this number keeps on growing with every new feature development that we do.

How could we run our unit tests faster?

OK, so we have great unit-test coverage. However, the side effect of this is that it usually took more than 5 minutes to run the unit tests. Now that is not a very big number by itself but it does become an irritant when we run tests on our developer boxes multiple times a day before we push our commits to git. On a given day, a developer could have spent 15–30 minutes waiting for the tests to run and the build to finish. So how could we spped up this process?

It turns out that NUnit-3 test engine has the ability to run tests in parallel. We hoped that it would reduce our test execution time. In addition, we looked at how Rake Multitask could help us reduce our overall build times. Read on to see what happened…

NUnit-3 Test Engine

NUnit-3 test engine supports parallel test execution in two modes:

1. Parallel execution of multiple test assemblies
In this mode we have to supply the list of test assemblies to the runner. For example:

<path_to_nunit3_console_exe> unit — test1 . dll unit — test2 . dll unit — testN . dll [ options]

NUnit test engine then runs each test assembly in a different process. The number of processes can be controlled by specifying other options. More details can be found on NUnit’s github page.

2. Parallel execution of tests within a test assembly
This mode enables parallel test execution within a given test assembly. Developer has the
control to run the test fixtures in parallel (running tests within a given fixture in parallel is not yet supported).

This behaviour is controlled by the [ Parallelizable ] attribute. To enable parallel running of test fixtures one can add the following attribute in the AssemblyInfo.cs

[ assembly : Parallelizable ( ParallelScope . Fixtures )]

The number of threads running the tests can also be controlled by either using the
[ LevelOfParallelism ] attribute or by using the — worker =N option when running the executable.

• There are, however, a couple of things to be considered before complete power of parallel test execution can be leveraged.
  We have to make sure that the tests we run do not share any common resources (for e.g. write to the same file). If they do chances are the tests may fail when run in parallel.
• Second and in our experience the more important point is to verify that any external
  library that you may use within the tests should also be thread safe.We were not able to run the test fixtures in parallel because the mocking library that we use is not thread safe and the tests failed when we tried to run test fixtures in parallel.
  We were able to bring down our test running time to around 2 minutes (from more than 5 minutes) by using the parallel assembly execution mode and have started work to migrate to a new thread safe mocking library so that fixtures can be run in parallel.

Parallel execution of Rake Tasks

Once we achieved some success in reducing test run times, we also looked around to see if we could reduce the overall build time. Apart from compiling and running tests, our build process also does a bunch of other tasks like style compilation, configuration file generation and so on.

We figured out that a lot of tasks can be run independently of each other and that is where we found the rake Multitask feature to be particularly useful. One example usage would be something like:

multitask :build => [:style_compile, :build_web_components, :gen_config_for_all_tocs]

Using Multitask makes Rake execute the prerequisites of the given task in parallel. Thus in this example the tasks style_compile, build_web_components and gen_configs_for_all_tocs would run in parallel.

It may be obvious, but it is worth mentioning that Multitask should not be used if the required tasks need to run sequentially.

Result

By making use of these two strategies, we have been able to reduce our overall build time by about 33% (as observed on TeamCity).

One observation that we have noted is that the reduction in test time has been greater on development boxes than when run in TeamCity. That suggests additional work on our part to fine-tune the options, but for now we are quite happy with the gains we have made.

About the Author

Subhrajit is a developer at Trainline with experience working in Java, C# and Ruby. He is very interested in distributed systems working at scale, as well as continuous integration and continuous delivery, hence the aversion to slow-running builds!

References: