Software testing: Automating installations and functional tests

Automated software testing plays an important role in ensuring quality at every stage of software development. This article discusses how to get started using Python’s Robot Framework.

Every code change bears the risk of causing undesired side effects that lead to error states in the software. In order to find such errors at an early stage (and not only in production), automated tests must validate both previous functionality and new functionality of the software.

First, the project goals (vision) and a rough roadmap should be defined. The necessary budget must not be neglected here. As expected, the effort involved in developing fully automated tests is much higher at the beginning of the project than later during operation. However, a certain “background noise” always remains, as test results have to be evaluated, test cases adapted, and new test functionalities implemented.

[ Also on InfoWorld: Complexity is killing software developers ]

After the procedures and processes of the software to be tested have been clearly defined, the tools and frameworks are selected. Subsequently, the first prototypes should be presented in a kick-off in order to create acceptance for the tools used.

Automated software testing—an introduction

After successfully building the (new) software, the test automation is automatically started with the software version to be tested. In the case of installable software, this must be done without manual intervention in so-called “silent mode.” By the way, most software installers offer a “response file.” A response file is a text file that contains the corresponding dialogs and inputs that the user specified when installing for the first time.

The software is then automatically set up and parameterized, and the correct functioning of applications, services, browser, GUI, etc. is tested. If the software product to be tested supports different platforms (Unix, Linux, Windows), all of these operating systems must be tested, at best in parallel and in a nightly run. In this way, long-running tests can be carried out efficiently, the utilization of hardware resources is better distributed, and the test results are available in the morning for further analysis.

If a large number of tests run in parallel on several different test systems, then the use of a virtual computer system, or virtual machine (VM), is recommended. If you do not want to use your own hardware, cloud computing is an interesting alternative. Here, the IT resources that are needed are also provided.

Testing with the Robot Framework

For the development of fully automated software tests, the Robot Framework is my clear favorite. Decisive factors are the extensive modular system for expanding the range of functions, as well as the simple handling with good integration possibilities. Another plus point is the simple, table-like structure of the test cases, which requires only a short training period.

The Robot Framework focuses on user tests (acceptance tests and end-to-end tests) but is also used for system tests and smoke tests. The Robot Framework, developed in Python, is freely available as open source software (Apache license). A new major release, Robot Framework 5.0, was published toward the end of March 2022.

The installation of the framework is uncomplicated, as only Python 3 and its package manager “pip” are required.

Two established systems are available for development and test execution, but they can also be used together to enjoy the benefits of each environment. RIDE is Robot’s own development environment, which makes it easy for newcomers to learn how to use the framework. RED is an Eclipse-based development environment (IDE) that will delight accomplished software developers.

Testing automation—scenarios

As described above, all functions of the software should be tested automatically. The Robot Framework helps us here with a wide range of libraries and additional packages that can be easily integrated. One example is the very stable SSH library, which makes it possible to run all tests remotely and reusable on Windows, Linux, and Unix.

In the area of end-to-end user interface (UI) testing for web browsers, the new and modern browser library from Playwright (since the beginning of 2021) or the long-established Selenium framework are available. These libraries can be conveniently integrated into the Robot Framework. This makes it easy to test graphically based functions such as mouse clicks or user inputs and outputs on websites.

Of course, self-written test scripts can also be integrated into the Robot Framework without any problems. It does not matter whether Java, Python, or shell scripts are preferred, only the return values of 0 or not 0 need be observed for useful reporting.

Software testing—evaluation

Extensive test reports are automatically generated as HTML pages by the Robot Framework. In addition, the test results can be imported into a test management system such as TestLink. To be able to analyze errors more quickly, a precise error description with output of the log files is very helpful. If there are a lot of tests, it also makes sense to define responsibilities for the tests. This makes it possible to address the corresponding colleagues with the error report via an automated e-mail.

The icing on the cake for the graphical presentation of the test results is the integration with e.g. a Splunk dashboard. This makes it easy to see the current situation comfortably, especially with a large number of software tests.

[ Keep up with the latest developments in software development. Subscribe to the InfoWorld First Look newsletter ]

If the dashboard is not enough, a traffic light system is the best solution. With the previously defined threshold values, the different colors of a USB traffic light can be used. For example, if you come into the office in the morning and see the traffic light shining red, it is immediately clear: errors have occurred. Otherwise, if you see a green traffic light, your day can start less urgently because all tests have been run and the software has passed without errors.

Fully automated tests are an indispensable factor, especially in agile software development, to ensure continuously error-free software. Valuable resources can be saved through test automations that run nightly. The Robot Framework is an excellent tool that can be used to automate all software tests.