Organizing test code in PHP

You are working on a PHP application or a package. You have decided that you want to use:

• infection/infection for running mutation tests
• phpbench/phpbench for running performance tests
• phpunit/phpunit for running unit, integration, functional, and end-to-end tests

Perhaps you are considering other design and testing frameworks as well, for example:

What are your options for organizing test code?

Systems under test

For this article, let's assume you have relatively simple production code organized in a directory structure as follows (listing generated with the tree command):

.
└── src
    ├── Example.php
    ├── ValueCanNotBeBlank.php
    └── ValueCanNotBeEmpty.php

Let's also assume that you are using a PSR-4 autoloader, so our directory structure and file names determine the naming of namespaces and classes.

💡 You can inspect the source code in the main branch of localheinz/organizing-test-code-in-php.

Options for organizing test code

As far as I can tell, you have the following options for organizing test code:

No tests

By far, this is the simplest solution for organizing test code.

You have fewer lines of code, dependencies, and configuration files. You probably also have more bugs, downtimes, and sleepless nights.

If you are comfortable with that, you can stop reading now.

Declaring test code in the same file

You can declare test code in the same file as production code, but unfortunately, I have not been able to configure phpbench/phpbench or phpunit/phpunit to find benchmarks or tests in the production files.

Unless you have found a way to make this work, declaring test code in the same file is not an option. If you have found a way, I would like to hear about it.

Declaring test code in the same directory

You can declare test code in the same directory as production code.

.
├── src
│   ├── Example.php
│   ├── ExampleBench.php
│   ├── ExampleTest.php
│   ├── Helper.php
│   ├── ValueCanNotBeBlank.php
│   ├── ValueCanNotBeBlankTest.php
│   ├── ValueCanNotBeEmpty.php
│   └── ValueCanNotBeEmptyTest.php
├── phpbench.json
└── phpunit.xml

💡 You can inspect the source code and a pull request that declares test code in the src/ directory in localheinz/organizing-test-code-in-php.

In my opinion, this approach has the following disadvantages:

1. It is hard to tell whether the code in the src/ directory is production or test code. How can you know whether a piece of code is only intended for use in test environments, for example, a test helper or a value object? How can you prevent developers from using test code in production?
2. If you want to run unit, integration, functional, and end-to-end tests, how can you distinguish between these different kinds of tests?
3. If you want to run unit, integration, functional, and end-to-end tests, how can you ensure appropriate test bootstrapping for each test? It is not necessary to bootstrap a database and run migrations for running unit tests, but it might be for integration, functional, and end-to-end tests.
4. Where are you going to place test fixtures or utilities if you need any?
5. You must excessively configure exclusions in composer.json to exclude test code from being autoloadable in production systems.
6. You need to configure exclusions for test code in infection.json to prevent infection/infection from mutating test code and reporting false negatives from mutated test code.
7. You need to configure exclusions for test code in phpunit.xml to prevent phpunit/phpunit from reporting test code as used in tests and not being covered by tests.
8. If you are working on a package, you need to excessively configure exclusions for test code in .gitattributes to prevent the unnecessary distribution of test code to users of your package.
9. If you are working on a package, how are you dealing with users who expect a test/ or tests/ directory in the root of your project, and skip the inspection and consideration of your package entirely because they suspect you have not written any tests?

If you are interested in what others say about declaring test code in the same directory, check out the replies to this tweet by Brent Roose:

An out of the box idea: why would/wouldn't you store your tests alongside your source code, instead of keeping them completely separated? I can come up with a few pros and cons, but would like to hear your opinion.

— Brent (@brendt_gd) September 15, 2021

Declaring test code in a subdirectory

You can declare test code in a subdirectory of your production code.

.
├── src
│   ├── Test
│   │   ├── ExampleBench.php
│   │   ├── ExampleTest.php
│   │   ├── Helper.php
│   │   ├── ValueCanNotBeBlankTest.php
│   │   ├── ValueCanNotBeEmpty.php
│   │   └── ValueCanNotBeEmptyTest.php
│   ├── Example.php
│   ├── ValueCanNotBeBlank.php
│   └── ValueCanNotBeEmpty.php
├── phpbench.json
└── phpunit.xml

💡 You can inspect the source code and a pull request that declares test code in a subdirectory of the src/ directory in localheinz/organizing-test-code-in-php.

You will often see this organization of test code in repositories split from mono-repositories. For an example, inspect symfony/var-dumper which is a split from symfony/symfony.

In my opinion, this approach has the following advantages over declaring test code in the same directory

1. It is easier to tell whether code is production or test code.
2. The configuration of exclusions in composer.json to exclude test code from being autoloadable in production systems has become simpler.
3. If you are working on a package, the configuration of exclusions for test code in .gitattributes to prevent the unnecessary distribution of test code to users of your package has become simpler.

In my opinion, this approach still has the following disadvantages:

1. You still are not distinguishing between unit, integration, functional, and end-to-end tests.
2. You are still not ensuring appropriate test bootstrapping for unit, integration, functional, and end-to-end tests.
3. You still need to configure exclusions for test code in infection.json to prevent infection/infection from mutating test code and reporting false negatives from mutated test code.
4. You still need to configure exclusions for test code in phpunit.xml to prevent phpunit/phpunit from reporting test code as used in tests and from reporting test code as not being covered by tests.
5. You are still placing test fixtures and utilities along with test code.
6. If you are working on a package, potential users may still skip the inspection and consideration of your package entirely because they can not see a test/ or tests/ directory in the root of project and suspect you have not written any tests.

Declaring test code in subdirectories

You can declare test code in subdirectories of your production code.

.
├── src
│   ├── Test
│   │   ├── Performance
│   │   │   └── ExampleBench.php
│   │   ├── Unit
│   │   │   ├── ExampleTest.php
│   │   │   ├── ValueCanNotBeBlankTest.php
│   │   │   ├── ValueCanNotBeEmpty.php
│   │   │   └── ValueCanNotBeEmptyTest.php
│   │   └── Util
│   │       └── Helper.php
│   ├── Example.php
│   ├── ValueCanNotBeBlank.php
│   └── ValueCanNotBeEmpty.php
├── phpbench.json
└── phpunit.xml

💡 You can inspect the source code and a pull request that declares test code in subdirectories of the src/ directory in localheinz/organizing-test-code-in-php.

In my opinion, this approach has the following advantages over declaring test code in a subdirectory:

1. You distinguish between unit, integration, functional, and end-to-end tests by grouping them in subdirectories and corresponding namespaces. Developers will understand where to place performance and unit tests more quickly.
2. If you need test fixtures or utilities, they can all go into subdirectories and corresponding namespaces.

In my opinion, this approach still has the following disadvantages:

1. You are still not ensuring appropriate test bootstrapping for unit, integration, functional, and end-to-end tests.
2. You still need to configure exclusions for test code in infection.json to prevent infection/infection from mutating test code and reporting false negatives from mutated test code.
3. You still need to configure exclusions for test code in phpunit.xml to prevent phpunit/phpunit from reporting test code as used in tests and from reporting test code as not being covered by tests.
4. If you are working on a package, potential users may still skip the inspection and consideration of your package entirely because they can not see a test/ or tests/ directory in the root of project and suspect you have not written any tests.

Declaring test code in subdirectories with separate configuration files

You can declare test code in subdirectories of your production code with separate configuration files.

.
└── src
    ├── Test
    │   ├── Performance
    │   │   ├── ExampleBench.php
    │   │   └── phpbench.json
    │   ├── Unit
    │   │   ├── ExampleTest.php
    │   │   ├── phpunit.xml
    │   │   ├── ValueCanNotBeBlankTest.php
    │   │   ├── ValueCanNotBeEmpty.php
    │   │   └── ValueCanNotBeEmptyTest.php
    │   └── Util
    │       └── Helper.php
    ├── Example.php
    ├── ValueCanNotBeBlank.php
    └── ValueCanNotBeEmpty.php

💡 You can inspect the source code and a pull request that declares test code in subdirectories of the src/ directory with separate configuration files in localheinz/organizing-test-code-in-php.

In my opinion, this approach has the following advantages over declaring test code in subdirectories:

1. You ensure appropriate test bootstrapping for unit, integration, functional, and end-to-end tests by extracting dedicated configuration files.
2. If you are working on a package, the configuration of exclusions for test code in .gitattributes to prevent the unnecessary distribution of test code to users of your package has become even simpler.

In my opinion, this approach still has the following disadvantages:

1. You still need to configure exclusions for test code in infection.json to prevent infection/infection from mutating test code and reporting false negatives from mutated test code.
2. You still need to configure exclusions for test code in phpunit.xml to prevent phpunit/phpunit from reporting test code as used in tests and from reporting test code as not being covered by tests, and that configuration has become even worse.
3. If you are working on a package, potential users may still skip the inspection and consideration of your package entirely because they can not see a test/ or tests/ directory in the root of project and suspect you have not written any tests.

Declaring test code in a separate directory

You can declare test code in a directory entirely separate from production code.

.
├── src
│   ├── Example.php
│   ├── ValueCanNotBeBlank.php
│   └── ValueCanNotBeEmpty.php
├── test
│   ├── ExampleBench.php
│   ├── ExampleTest.php
│   ├── Helper.php
│   ├── ValueCanNotBeBlankTest.php
│   ├── ValueCanNotBeEmpty.php
│   └── ValueCanNotBeEmptyTest.php
├── phpbench.json
└── phpunit.xml

💡 You can inspect the source code and a pull request that declares test code in a separate test/ directory in localheinz/organizing-test-code-in-php.

You can find documentation for this approach in the official PHPUnit documentation.

In my opinion, this approach has the following advantages over declaring test code in the same directory:

1. It is trivial to tell whether code is production or test code.
2. You do not need to configure exclusions in composer.json to exclude test code from being autoloadable in production systems. Instead, you configure an autoloader for test code in the autoload-dev section.
3. You do not need to configure exclusions for test code in infection.json for infection/infection.
4. You do not need to configure exclusions for test code in phpunit.xml for phpunit/phpunit.
5. If you are working on a package, you only need to configure exclusions for test configuration files and the test/ directory in .gitattributes to prevent the unnecessary distribution of test code to users of your package.
6. If you are working on a package, potential users will not skip the inspection and consideration of your package entirely as they can see a test/ or tests/ directory in the root of project.

In my opinion, this approach still has the following disadvantages:

1. You still are not distinguishing between unit, integration, functional, and end-to-end tests.
2. You are still not ensuring appropriate test bootstrapping for unit, integration, functional, and end-to-end tests.
3. You are still placing test fixtures and utilities along with test code.

Declaring test code in separate directories

You can declare test code in subdirectories of a directory entirely separate from production code.

.
├── src
│   ├── Example.php
│   ├── ValueCanNotBeBlank.php
│   └── ValueCanNotBeEmpty.php
├── test
│   ├── Performance
│   │   └── ExampleBench.php
│   ├── Unit
│   │   ├── ExampleTest.php
│   │   ├── ValueCanNotBeBlankTest.php
│   │   ├── ValueCanNotBeEmpty.php
│   │   └── ValueCanNotBeEmptyTest.php
│   └── Util
│       └── Helper.php
├── phpbench.json
└── phpunit.xml

💡 You can inspect the source code and a pull request that declares test code in subdirectories of a separate test/ directory in localheinz/organizing-test-code-in-php.

In my opinion, this approach has the following advantages over declaring test code in a subdirectory:

1. You distinguish between unit, integration, functional, and end-to-end tests by grouping them in subdirectories and corresponding namespaces. Developers will understand where to place performance and unit tests more quickly.
2. If you need test fixtures or utilities, they can all go into subdirectories and corresponding namespaces.
3. If you are working on a package, potential users will not skip the inspection and consideration of your package entirely as they can see a test/ or tests/ directory in the root of project.

In my opinion, this approach still has the following disadvantages:

1. You are still not ensuring appropriate test bootstrapping for unit, integration, functional, and end-to-end tests.

Declaring test code in separate directories with separate configuration files

You can declare test code in subdirectories of a directory entirely separate from production code with separate configuration files.

.
├── src
│   ├── Example.php
│   ├── ValueCanNotBeBlank.php
│   └── ValueCanNotBeEmpty.php
└── test
    ├── Performance
    │   ├── ExampleBench.php
    │   └── phpbench.json
    ├── Unit
    │   ├── ExampleTest.php
    │   ├── phpunit.xml
    │   ├── ValueCanNotBeBlankTest.php
    │   ├── ValueCanNotBeEmpty.php
    │   └── ValueCanNotBeEmptyTest.php
    └── Util
        └── Helper.php

💡 You can inspect the source code and a pull request that declares test code in subdirectories of a separate test/ directory with separate configuration files in localheinz/organizing-test-code-in-php.

In my opinion, this approach has the following advantages over declaring test code in separate directories:

1. You ensure appropriate test bootstrapping for unit, integration, functional, and end-to-end tests.

In my opinion, this approach has the following disadvantages:

1. You may need to merge code coverage reports if you use different configuration files for unit, integration, function, and end-to-end tests and want to collect code coverage from multiple test runs.

Recommendation

I prefer to declare test code in separate directories with separate configuration files, but I am open to changing my mind and adapting to circumstances.

Do what works best for you!