End-to-End Tests: The Pinnacle in Test Automation

This article is featured in the new DZone Guide to Automated Testing: Your End-to-end Ecosystem . Get your free copy for more insightful articles, industry statistics, and more! 

End-to-end testing can mean different things to different people. Regardless of the definition, end-to-end tests are more complex, harder to automate, and many times carry more risks than functional and unit tests. Functional and unit tests focus on whether or not a specific feature of a single application works. End-to-end tests ensure a request or transaction can flow across an entire system of technologies and applications. Failure of an order-to-cash process or payroll process not only represents a software failure but also a business failure. As companies move to adopt Agile across all of their applications and quickly roll out changes across all of their systems, automating end-to-end tests becomes critical for reducing risks and accelerating change.

What Is an End-to-End Test?

Depending on who you ask, end-to-end testing can mean different things. In the custom app development space, end-to-end testing is often used to describe an integration test. In this case, a "vertical" test that may start at the presentation/services level is used to test the back-end responses from the System Under Test (SUT). The test runs across the various layers used by the SUT.

Business process testing is used to test enterprise business processes that include multiple steps and, many times, across multiple applications. In this case, the test is usually driven at the UI layer and is used to validate that the end-to-end business process works across systems as a "horizontal" test. Unlike vertical tests that measure performance up and down the layers of the technology stack, horizontal end-to-end tests are used to ensure that mission-critical business processes work. Tests are run at the UI layer and measure whether or not the test proceeds to the next step in the process.

Figure 1: "Vertical" end-to-end integration test.

Figure 2: "Horizontal" end-to-end business process test.

This is a similar debate to test-driven development (TDD) vs. behavior-driven development (BDD). TDD execution is writing a failing test for one very specific bit of functionality, generating a unit or "vertical" test. An end-to-end test or "horizontal" test is similar to BDD when looking at multiple features working together to complete a workflow or epic in Agile terms.

It is also important to note that end-to-end tests are not just several unit and functional tests strung together. The scope of what is being tested and the goal of the tests are considerably different between functional tests and end-to-end tests. Key differences between functional and end-to-end tests include:

• Functional Tests:

  • Testing is limited to a single piece of code or application.

  • Software is tested to ensure it meets acceptance criteria.

  • Test the way a single user engages with the application.

  • Validate the result of each test for inputs and outputs.

• End-to-End Tests:

  • Testing crosses multiple applications and user groups.

  • Testing ensures a process continues to work after a change is made.

  • Test the way multiple users work across applications.

  • Validate that each step in the process is completed.

Why Are End-to-End Tests So Hard to Scale?

When we talk about end-to-end business process testing, we're talking about testing just like a user, which means there are going to be hundreds — if not thousands — of tests that have to be managed across multiple applications and UIs. For this reason, just building the automation for an end-to-end test is a challenge. In fact, according to the 2017-18 World Quality Report, only 16 percent of end-to-end business scenarios are executed with test tools.

Included below are some key reasons end-to-end tests are hard to automate, including a couple of examples of customers who have had these challenges and how they have overcome them.

Maintenance

Most agree that implementing a test automation solution is good for the business. But to be successful with end-to-end test automation, it needs to take less time to create, maintain, and run tests than it does doing things manually. Unfortunately, many organizations that have tried automation solutions for end-to-end testing have had to abandon their efforts because it either takes too long to create the tests or too much time to maintain them.

A recent case study published in SAP Insider regarding Honda R&D Americas is a perfect example of challenges clients have had with automating end-to-end business process tests. They purchased a test automation solution that was script-based and found it took more time to create and maintain the automation than it did to continue to test things manually. They quickly abandoned it, but a move to SAP HANA forced Honda to revisit the idea of test automation. With SAP HANA, they were looking at a three-month patch cycle and there was no way to continue doing things manually. This time, they focused on ease-of-use and ease-of-maintenance in their selection process. With the new solution (Worksoft Certify), they were able to meet their automation needs and scale testing to keep pace with updates.

Access

Unlike custom app virtual devtest environments that can be spun up in minutes, tests for enterprise applications like SAP need to be run many times in pre-production where the systems are shared and testers can't just spin up a personal instance. Another big challenge is automation that acts like a user needs to act like a real user by logging into the desktop and running applications. This means installing local agents and logging into remote machines to simulate real users. To make things more complicated, you also need to figure out a way to keep the machine awake for the life of the test and prevent things like Windows updates from interrupting tests.

Orchestration

You must be able to orchestrate the workflow of an end-to-end test. The tests need to run in the same order as the real system would process the request in production to validate whether or not it will actually run in production. The tests often need to be scheduled in sequence with complex dependencies. It's difficult to manage distributed and diverse testing resources. The ability to just say, "Go run these thousand tests and let me know when they're done," without the right solution will not validate multiple features are working together. The testing window, especially in global situations, is often strictly limited or closely scrutinized.

Scale

When rolling out a change to a mission-critical system, businesses don't just want one workflow tested. They want them all tested each time a change is made. As changes happen more frequently, more and more tests have to be documented, created, and run in the same testing window. Automation becomes a critical part of the process. Test automation needs to be generated faster and tests need to be run in parallel across more resources so that they can be completed within a given testing window.

Cardinal Health recently undertook an automation effort in an effort to become more Agile and to better support changes to SAP. They quickly realized that in order to be successful, automation needed to become everyone's job. They needed to change the way they work and enable business analysts and testers to better collaborate together along with being able to orchestrate and run tests at scale. By implementing an automation solution that enabled them to create and run automation faster, they were able to spend more time planning, detect more defects faster, and better meet the needs of the business.

Conclusion

End-to-end business process tests are complex by nature. They cover a wide variety of systems and applications and can include hundreds to thousands of steps that need to be orchestrated and driven via a UI that requires a real machine. But IT must evolve test automation to include end-to-end testing to meet the needs of their business. Companies must select tools that are easy to use, easy to maintain, and easy to scale in order to ensure their long-term success with test automation.

This article is featured in the new  DZone Guide to Automated Testing: Your End-to-end Ecosystem . Get your free copy for more insightful articles, industry statistics, and more!