Is Selenium Worth It?

End-to-end testing has become increasingly popular, and can be achieved by many different tools and approaches, with different degrees of effectiveness and varying levels of effort required for implementation. Tools like Selenium represent one of the most popular methods of end-to-end testing, but Selenium and similar tools can be challenging to implement and maintain.

In this article, you’ll learn the pros and cons of end-to-end testing using Selenium, and tools like it, to help you decide if Selenium is worth the effort, or if you need to consider a less burdensome alternative.

What Is End-to-End Testing?

End-to-end testing is a practice in the software testing life cycle that keeps the usability, functionality, integration, performance, and security requirements of an application in check, from development to production. This testing verifies that the software and its subsystems all work as expected. End-to-end testing simulates user-like interactions from the start of a workflow to the end.

A good starting point for end-to-end testing is to perform it manually, this can later be supplemented by automated scripts and test suites. These manual tests could simply involve manually exploring an application from the perspective of a consumer. This often leads to the discovery of unexpected defects and bugs, including those that may have been missed by automated tests.

What Is Selenium?

Selenium is among the leading tools for end-to-end testing. It is an open source automation framework that’s used to test many common browsers and cloud platforms. Selenium supports writing test scripts in several programming languages such as Java, Ruby, Python, and JavaScript. Selenium doesn’t work as a single tool, but as an integration of tools, including Selenium WebDriver (collection of APIs for testing web apps across browsers), Selenium Grid ( server that facilitates parallel running of scripts in remote machines), and Selenium IDE (a browser plugin for recording and replaying tests).

Selenium enables automation engineers the ability to write browser-level automated regression test scripts that are easy to scale across multiple environments and browsers. Through Selenium's WebDriver API, automation engineers can write scripts that mimic a user's interaction with web browsers, including those accessed from mobile devices.

End-to-end tests using Selenium manipulate the Document Object Model (DOM) to verify what elements are displayed to users whilst simulating user input and interaction, i.e. simulating a user journey in an application. For instance, an end-to-end test scenario for an e-commerce platform might include a script that verifies sign-up, sign-in, a product search, add to cart functionality, product review views, and checkout logic.

A simple login automation script can be as seen below: The script will load the page and sign up as a normal user would do.

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.testng.annotations.Test;
import java.time.Duration;

//Simple login automation
    public class HelloSeleniumTest {
        @Test
        public void firstTest() {

        //Set and Initialize the web driver and browser specification
            System.setProperty("webdriver.gecko.driver","/replace/with/path/ of your driver”
            WebDriver driver=new FirefoxDriver();

            //Define the page address to be automated and access it through the firefox driver
            String url = "https://www.amazon.com/";
            driver.get(url);

            //Set wait time for page
            driver.manage().timeouts().implicitlyWait(Duration.ofSeconds(20));

            //Locate the user name field and send keys to it
            driver.findElement(By.id("user-name")).sendKeys("standard_user");

            //Locate the password field and send keys to it
            driver.findElement(By.id("password")).sendKeys("secret_sauce");

            //Locate the submit button and act on it.
            driver.findElement(By.id("login-button")).click();
        }
    }

However, end-to-end testing can be expensive, given how long it takes to write and maintain the tests. To offset this expense, such tests are typically created in smaller, more focused quantities, as described by the testing pyramid.

The Testing Pyramid

The software testing pyramid is a model of grouping software tests into categories that follow a pyramid model. The higher up the pyramid you go, the more expensive testing is, and thus, fewer tests of that type should be created.

Building test suites following this model typically helps teams find a good balance between the different types of tests that they need for their applications, with lots of cheap, isolated tests, and fewer expensive, deeply integrated tests.

Let's break down the pyramid:

• Unit testing: This forms the bottom of the pyramid. Unit tests validate small pieces of code to ensure that individual units of code work correctly in isolation.
• Integration testing: Next, to test how your code units interact with other code units to build a functional component, integration tests are needed. Integration tests validate the interaction of a code unit with other external components and dependencies. For example, when a connection to an API or web service to retrieve certain data is made, integration testing can be done to check if the endpoints communicate correctly and if the exact data is returned as desired.
• End-to-end testing: This is located at the top layer of the pyramid; it involves running tests that invoke the entire system from start to finish, with minimal mocked components. These tests are the most realistic in terms of accuracy when compared to actual user interactions with the application. Due to the ongoing maintenance and time required to set them up and run them, they’re generally expensive and thus placed higher up on the pyramid.

End-to-end tests are typically built to resemble actual workflows that happen as part of the user’s journey through the application. They should simulate the kind of interactions that the users will put the application through in real life, and verify that there are no unexpected issues or breakages in these workflows.

In practice, creating end-to-end tests often involves using a tool like Selenium to control a web browser to programatically interact with the application under test in the same way a user would. When writing the test, you would use selectors to identify elements and DOM nodes that you want to interact with and subsequently execute commands to manipulate these elements and finally use assertions to verify that things happened as expected. You can see the login automation script above as an example of this.

Alternatives to Selenium

Many other tools can be used in place of Selenium to achieve end-to-end testing. Here are some alternative tools for testing modern web applications.

Cypress

Cypress is a fast, all-in-one testing framework for creating end-to-end tests that will run on any browser-based application. Cypress also presents an assertion library, with spies and stub capabilities.

Test scripts written in Cypress are simple to understand and execute. Automation engineers who use Cypress get a rich set of APIs, as well as other open source tooling around Cypress, which can make testing relatively straightforward.

Cypress is easy to set up and relatively inexpensive to integrate into any QA culture. It also offers some important features like automatic waiting and time travel debugging, along with screenshot recording of the test process and results.

Playwright

Playwright is another popular tool for performing end-to-end tests for modern web apps. This tool allows QAs to execute end-to-end tests in parallel across major browsers.

Through its auto-wait and introspection feature, robust end-to-end tests can be created, since app elements and features are waited for implicitly before test actions are performed on them. This makes test code much cleaner and easier to work with.

Playwright features powerful and built-in tools such as Test Generator (creating tests from recorded user actions), Playwright Inspector (scanning web pages for element locators and click points) and Trace Viewer (tracing point for failed tests with live DOM snapshots, action explorer and more), all of which help testers generate powerful end-to-end tests.

Challenges in End-to-End Testing

Though end-to-end testing is a reliable way to ensure that your application works exactly as it should from development to production, there are some significant challenges in implementing this method.

Hard-to-Maintain Tests

A major drawback of using a tool like Selenium for end-to-end testing is that it can be difficult to maintain test scripts and avoid flaky tests whenever applications are re-engineered. A Selenium test script written in a tightly coupled manner is likely to break whenever there are changes in the implementation of the application.

A change in the web app would require the test scripts to be rewritten, especially scripts pointing to elements' identifiers or paths. These back-and-forth changes in scripting result in slow, expensive, and hard to maintain end-to-end test suites, as testers try to keep failures on track, fix the breakages, and rerun them in a cycle.

No Built-In Reporting Features

Working with Selenium WebDriver requires the integration of tools like JUnit or TestNG for the generation of test reports, as Selenium lacks a built-in structured reporting feature. Selenium also has a steep learning curve when building end-to-end test suites for large-scale applications.

These inherent difficulties mean that sometimes it can be easier for an organization to use an alternative approach for end-to-end testing that can provide many of the benefits without the challenges (e.g. replay testing).

Replay Testing with Meticulous

Replay testing is a technique for performing tests on web applications using little or no code (codeless testing). This technique involves the use of a well-established platform with features that record performed actions in the browser and then replaying those actions on the new version of the web application.

Meticulous is a tool to easily create end to end tests without writing code and without requiring a staging environment. Use the CLI to open an instrumented browser which records your actions as you execute a workflow on your web app. This sequence of actions can then be replayed on any URL, like a new version of your web app on localhost. Meticulous captures a screenshot at the end of each replay. Screenshots can then be diffed in order to create a simple test. Meticulous makes it easy to integrate these tests into your CI (we'll even do it for you if you add us as a collaborator).

Combining Replay Testing and End-to-End Testing

While replay testing (with Meticulous) and tools like Selenium have different approaches to end-to-end testing, the two can be used together to benefit an organization needing quality tests.

Meticulous can step in to fill the gaps of Selenium tests, especially when an organization is transitioning from manual testing to automation testing. Also, at the initial stages of end-to-end testing, Meticulous can be used for any lightweight automation, and as user workflows become complex, Selenium can be incorporated to help create more robust test suites.

Moreover, the two tools provide the ability to execute tests in parallel, thus accelerating test coverage for your applications while comparing results achieved in each case.

Benefits of Replay Testing Compared to End-to-End Testing

Some of the benefits available to QAs Meticulous replay testing include:

• No maintenance burden: Writing and maintaining large codebases of tests without proper project structure and quality standards can be time-consuming and require skills. Meticulous makes it easy to reset the new 'reference' baseline image or replay with a single click.
• Easy to record: Initiating replay testing simply starts with a click of a button to start the recording. Once the recording is stopped, a test engineer can replay those flows on new frontend code and make alterations and additions to their scripts if desired. This means that your tests require less effort to create.

Drawbacks of Replay Testing Compared to End-to-End Testing

Some drawbacks of replay testing include:

• Does not use the real backend: Many record-and-playback platforms are largely procedural (following exact steps previously recorded), which means that any slight change in the application being tested will require all previous tests to be re-recorded. The procedural pattern of testing is therefore only suitable for frontend regression scenarios, making this type of testing ineffective for complex applications that are data-driven (restricting test scope). Though Meticulous doesn’t use the real backend by default, you should note that you can configure it to replay against a backend if needed.
• Cost factor: Feature-rich replay platforms aren’t open-source, making them relatively expensive. Most require subscriptions and frequent billing management. There are some free options, but they ordinarily have extremely constrained features.

Conclusion

Different organizations will use varied approaches and tools when performing end-to-end testing. This poses a concern in having a stable framework in place to guide the QA team in establishing simple to understand, fast, and scalable test scripts for any application being tested.

Having seen all that’s required for end-to-end testing using Selenium or record and replay, it’s clear that Selenium automation requires some level of technical know-how, as well as maintenance of test scripts and the test environment.

It would therefore be ideal for an organization to consider a less burdensome alternative platform like Meticulous for recording user sessions and replaying them in CI environments to catch regressions in any of its applications.

In this article, you learnt about end-to-end testing with Selenium as the tool of focus. You also saw some of the pros and cons of end-to-end testing using Selenium and other tools like it.

If you’re working towards making safe, performant, and reliable applications, you should give Meticulous a try in your next project.

Authored by Ely Obare