What is the primary task of many software engineers in test nowadays? It is to develop or find the right test automation solution for achieving fast, reliable, easy to understand and maintain tests that can be integrated into CI/CD pipelines. I will share some approaches that he regularly uses with clients during consulting to achieve these goals and discuss many common mistakes. One of the many errors is that the engineers are not doing proper research and setting the right requirements upfront, which leads to losing time developing their own solutions and maintaining lots of problematic tests later. You will learn some fundamental assessment criteria for automation testing designs, why they are essential and how to apply them in practice. We will discuss acquiring the proper requirements for the searched solution and how to conduct the research the right way. Afterward, the presented assessment framework can help you to find the right test automation solution.

What Is the Problem?

Many companies, at some point, decide to start automating their QA activities if they had some. Or for some new projects, want to establish test automation from the start. So, they have to pick the right tool/solution for doing that. This article will be about approaching the problem, selecting the tools to evaluate, comparing them, and deciding which is best.

What Is the Usual Approach?

From my professional experience as a consultant, many companies make poor decisions at this particular step, one of the most important ones. Many of them have only manual QAs. The usual approach is to assign this process to the most senior of them, who may have only basic programming experience or almost zero experience with test automation. Or another usual practice is the transfer one junior/regular developer to help with the effort.

The problem is that none of these roles have the actual expertise to decide which is the best solution nor have the experience to predict and estimate the cost of creating your solution or customizing an existing one.
Usually, there is a massive conflict of interests because these roles typically prefer to create their own thing, not considering the actual cost because they want to learn and get better during this time. Usually, it is hard for managers to see this conflict because they don’t have experience with such solutions and testing.

Three months ago, I consulted a company with a team of 10 manual QAs and no QA lead or architect, but they didn't have any automation until now. They wanted me to train them to write proper automated tests and eventually to check whether they have picked the right solution. It turned out that they assigned one mid developer to pick an automated testing solution who didn't have any previous experience writing tests. So he spent 1 day reading StackOverflow + blogs and directly picked one of the suggested tools without further research or evaluating different options. It took him 2 weeks to automate one basic test scenario for their system. So, when I joined, I showed them the system that I will present to you. We used it to evaluate different options and picked another tool. Using it, we automated the same scenario for half a day.
Now let's take a look at the system.

Assessment Framework

Phase 1: Gather Requirements

First, you need to create a list of requirements that you want to achieve. For example, automate the login, create accounts, submit tickets, run tests nightly, etc. You can do that by team brainstorming, asking senior division members, hiring a consultant, or conducting stakeholder interviews.
Remember that stakeholders such as business owners often don’t have the technical expertise or understand the testing process.

Phase 2: Research Existing Solutions

Pick the so-called HARD requirements and ignore for now the nice-to-have stuff. Based on your HARD requirements, the solution for sure should have - filter the frameworks/tools and pick a few - 3 to 5 most serious candidates. These will be the solutions that we will further use for creating a PoC (proof-of-concept) and assess.

Phase 3: Define Assessment Criteria

The next phase is to define the criteria which you will use to compare the solutions. Since all of them are more or less subjective, we provide a rating from 1 to 5 for each of them. In a minute, I will show you an example. But let’s first discuss them.
By the way this is just an example. You can put whatever you believe is important for you. For me usually, it is the solution to be a programmatic one - not to use UI tools.

• Automate Web
• Automate Mobile
• Automate React
• Good Documentation
• Parallel Execution
• Easy Tests Creation
• Tests Stability
• Tests Readability
• Failures Troubleshooting
• Source Control Support
• Easy CI Execution
• Reporting Tools Support
• Paid Support/Consulting
• Customization Time Required
• Framework Extend Support
• Fast Learning

And a few bonus ones.

• Code Conventions
• Test Environment Configurations
• Responsive Testing
• Secrets Management
• Specific Tools Integrations
• Use SUT Technology Stack (Microsoft Technologies/JAVA)
• Use Non-programming Solution

Phase 4: Implement Proof of Concept

Form a working group of technical experts. Implement 2-3 tests for each solution. It is better if all participants do it.

Phase 5: Rate Each Assess Criteria

Each team member will create a table for each solution. For each solution, we will have a separate column. For all criteria, we will have rows. So once you have the PoC for all the solutions, each participant will put a rating for each measure for each solution. Then, at the bottom, we calculate an average for all criteria, the final score for the solution.
You can further extend the rating phase by prioritizing the different criteria and provide a "WEIGHT" index for them.
Put rating between 1 and 5. 5 is the highest score. If there is a WEIGHT index, you will need to multiple by it. For example, 1.5 for good documentation.

Phase 6: Working Group Total Rating

After we have the ratings of all participants, we create a final rating table. Each person from the working group should provide their ratings based on the PoC. Then, you can also put the WEIGHT index again based on the participants' seniority/authority/role. In the example, in the first row, Participant 1 is the QA Lead, so we multiply all scores by 4.

Phase 7: High-level Test Automation Strategy

1. Create Abstract Test Cases Suite
2. Set Priorities and Categories
3. Put Test Cases in 3 Phases
4. Create a Test Automation Requirements
5. Provide Estimates for Phase 1 Test Cases
6. Provide Estimates for Requirements

What you will need as a test environment, test data preparation for working on phase 1 tests?

•   Mock server and web page fixture generator
•   Test Data Web Service with 3 endpoints - creating test users, test purchases, validating completed orders
•   Custom framework components for working with test data tables?
•   Custom logic for validating PDF information
•   Logic for generating session cookies by user name
•   Bypass captchas on test environment

Phase 8: Final Decision

You need to answer a few essential questions if you believe you should use an existing solution.

Are you going to use the existing solution?
How much time will you need to implement it?
How much will it cost to maintain and support it?
Do you have an easy way to learn it? Are there any trainings?

Or these if you want more to develop your own solution.

How much will it cost?

Do you have the expertise to do it?

How much will it cost to support it over time?

Summary

You will be able to use this pragmatic information-based approach to evaluate test automation solutions/frameworks and other kinds of tools/systems.