Let’s get one thing out of the way real quick: should product managers be concerned with test automation? We already have so much on our plates, so why add QA and test automation to an already impossible list of tasks?

The answer is — you shouldn’t, not if you don’t have to. It’s definitely best to leave code testing and automation to dev. If you’re a part of a growing or mature company, that’s probably the case anyway, so go read something more useful!

But in many smaller companies, it’s not always clear where QA sits. The reason is that the QA needs to have a broad view of the entire system, including any new feature, and in a way, they “represent” the PM in making sure things were developed as planned. Also, very often, until a full-time QA is hired, the PM designing the features is also the one doing the testing. So when that QA person is finally hired, it seems to make sense to put them under Product.

The issue with QA

When you’re just starting off building a product, you can add features very easily. But then, one day, the worst of all happens — you start having users. And if something breaks, they tend to get irritated. So you start running manual regression tests, at least for critical parts of your product (regression tests = trying to make sure nothing old got broken while building the new stuff).

Want to read this story later? Save it in Journal.

While we aspire to have smaller, more frequent releases, each regression test might take a few hours. If issues are found and fixes are deployed, the test needs to be run again. And of course, they need to be run both on the staging environment before the release, and in production after it.

For obvious reasons, this usually leads to one of two outcomes: clustering features in larger releases every few weeks; or quicker, less detailed regression testing, where bugs start slipping through the nets. The latter usually leads to blockers in production, which require hotfixes, which in turn are not tested properly and lead to more bugs. You know the drill, right?

Automation to the rescue

There are also different types of automation — unit testing, integration, and end-to-end (e2e) tests, as I tried to broadly explain in a previous post. Since it’s mostly highly technical and requires code, you might think it’s completely out of reach, and in the hands of the developers in any case.

As I said earlier, it’s awesome if devs do it, and you can worry about adding business value. But if you’re the one running endless regressions, and assuming you don’t have the dev capacity to handle it, it’s kind of your problem to solve. Luckily enough, it’s completely solvable.

First, let’s quickly describe what a test is:

1. you perform a set of actions (clicking, typing, etc.)
2. you verify that a desired result has happened (a popup appeared, a value changed)

There are some great point-and-click tools where you can record yourself doing the sets of actions, and set the desired results. It’s really as easy as that. The recording is done by using CSS selectors, which is how the bot knows which buttons to click and which inputs to fill. Because of that, some knowledge of CSS can come in handy, although it’s not a must.

Let’s go over some tools:

Selenium

This is probably the most popular tool for automation. Selenium is an open-source tool with a large community behind it. It holds some benefits:

• it’s free
• it’s easy to use
• you can run as many tests as you’d like, whenever you’d like to

As-is, selenium will offer the basics of what you need. You can easily record tests using Selenium IDE, and play them locally on your device. Because it’s open-source, many of the paid tools are actually built on top of Selenium, so it makes sense to start with it.

Selenium’s limitations, however, are substantial — used locally, you can’t share your tests with others. You can use dev to improve this, but automating your automation is takes time, and if you ask me it a little too meta anyway.

If you’re interested, here’s a great article that elaborates on some of the pros and cons of the IDE.

Selenium-based management tools

Since Selenium is a pretty simple tool, it has somewhat of an ecosystem around it. Selenium itself offers Selenium Grid, and there are other independent solutions that offer the missing Selenium functionality, such as reporting, scheduling, multi-browser, and more.

Which some of them might be solid solutions, I’ve failed to find any that offer a real no-code experience. most are highly technical tools designed for developers, which is why I won’t go too much into them in this post.

Testim.io

I’ll be honest — this is one of my favorite tools of all time. At times, I’m damn near smitten with it, and you know why? because it does something complex in a very simple way.

Testim.io works in a similar way to Selenium, in that it has a point-and-click interface through which you can define your tests. On top of that, it also offers some extremely useful features:

• you can share tests and collaborate with your team.
• you can organize your tests in different test suites.
• you can see what got broken very easily.
• they do this cool thing where if your devs change the CSS, Testim kinda guesses which element you meant to click. I won’t get too much into it, but it reduces the need to maintain your tests by a good amount.

The main drawback of using Testim, at least for us, was the limitations on the number of tests you can run. While it allows enough test runs to run your entire regression a couple of times a week, it would have been great to be able to use it on every build, and even routinely every night, just as an extra precaution. I still recommend using it though. the setup was very easy, and it did allow us to share the tests across our team, which was great.

To avoid doubt: although Testim.io is a fellow Israeli company, I’m in no way connected to them. I’m just a die-hard fan with a blog.

How long will it take?

A while ago, I was PM at a growth startup. For 2 years, and even as the team grew, our QA still ran manual tests almost every other day. Looking back, I truly admire his consistency. It couldn’t have been easy.

After that, as director of product at a seed-stage startup, I decided I’d try to make a move on automation. QA was under my responsibility, so I took a week or so to explore 4–5 tools. I finally settled down on Testim.io, and then let my QA (manual and non-coder) run with it. It wasn’t smooth, but within 2 weeks we had something we could start working with.

When I started my new startup TagBox, and having a very small team and an ultra-tight schedule (you’d be surprised how much that crowded PM schedule gets worse when adding marketing, sales, and fundraising, but that’s a story for another day), I started with Testim as soon as we onboarded our first customer, and things breaking started to become a potential issue.

I tasked it to our PM, who, while incredibly talented, has never done any sort of automation before. It took him three days (!) to create the first layer of regressions. And other than a quick maintenance fix every now and then, that was the last I had to worry about regression bugs.

Other things to consider

A couple of things are worth keeping in mind while implementing automated testing on your team:

1. Even no-code solutions require CSS knowledge:
   The good thing is that unlike Javascript, CSS is something that most QA people already know, or can probably learn in a couple of days. But do keep that in mind if your QA is not very technical.
2. Build modularly:
   It’s worth paying attention to how you set up your tests. You want them to be small enough so that each function is separated, without many repetitions. That way you can add them together in many ways, without duplicating actions.
   For example, if you’ll add a login to each test, and change to the login page will mean you’ll need to fix all of your tests. It’s better to create a test for the login flow once, and then add it to a larger suite of tests.
   As a rule of thumb, I’d say (based only on my personal experience), you should aim for having 10–20 tests that cover the common and critical actions in your app. Of course, that depends on the complexity and stage of what you’re building, but I find it mostly convenient.
3. Maintenance:
   Even if you do build your tests properly, some maintenance will be required, so keep that in mind. As your product grows, developers will naturally change some things around, and you’ll need to adapt.

Summary

There’s a TON to be written about QA and test automation. It’s increasingly popular to say that QA is a crutch, and that developers need to take responsibility for the quality of their code.

But as it’s still a fact of life in many cases, and in particular in small and medium startups — you should do what you can to make life easier for you and for your team.

Value for effort — setting up test automation has been one of the most effective things we’ve done at TagBox, and I couldn’t imagine going back to repetitive manual tests.