Running your end to end tests on Kubernetes

Jean Baudin

In a previous article, I explained how to use the Google Cloud Functions for building a system of branded website. Today, I'm going to explain about how we used Kubernetes to run our end to end tests.

Integrating the build and the deployment of your website in a CI/CD software brings quite often some challenges.

In our case, at Travix, we work with GoCD for managing our CI/CD. For managing our infrastructure (CI/CD/front-end production), we rely on the Google Cloud Platform and more specifically on GKE.

Travix is an OTA (Online Travel Agency), managing up to 40 websites across the world through the brands Cheaptickets, Vayama, Budgetair, Vliegwinkel and Flugladen.

GoCD works with a system of pipelines. The pipeline can have one or many stages, within this stages there will be one or many jobs.

In order to execute a pipeline, its stages are ran on an agent. An agent is a worker with a given set of features. Each stage is deployed in its own workspace. It executes its jobs until the end then it exits. Many stages could run on one agent.

When the number of pipelines increases, the complexity of your tests and the time it takes to run them, increases as well. You may also want to deploy some more infrastructure for your tests on your backend. For example, for your end to end tests, you could deploy your own Selenium Grid. However, it makes your architecture a bit more complicated to maintain.

1. The context

Here, we will talk about the front-end e2e tests. Let’s say that we have 50 branded websites sharing a common source code, having their own bunch of settings/features which make them a bit uniq.

Such as what I described earlier, if you want to add some tests in your release flow, it can become quite a huge infrastructure, quite hard and quite painful to use and to maintain.

Our e2e tests are written in NodeJS using the framework webdriver.io . We use Babel for some sugar syntax (JavaScript hipsters). The project is stored in a Git repository. The main users of the e2e tests are our front-end developers. They develop the UI then create page objects for writing the scenarios for the tests.

Each time we are building one of the 50 websites, we are running the e2e tests for this website against Chrome and Firefox. Until they have passed, the site won’t be deployed to production. It means that each time that I want to release all websites, I have to clone 2 times my repository, perform 2 npm install for each site. I am so running 2 * 50 times my e2e tests.

Running a 100 times your stage could lead to some issues with your agents as well (memory consumptions due to the usage of the Babel runtime, ~1Gb of RAM, availability of your agents). Talking about the time, 1 stage takes approximatively 4 to 5 minutes to complete. So we have to wait for almost 100 * 4.5 minutes before being able to release all sites.

How do we reduce the time it takes to run our e2e tests? How do we make them more “memory” friendly? How to reduce the complexity of our infrastructure?

Now that we introduced the situation and have written down the questions raised by it, let’s answer them.

We could reduce the runtime not by increasing the amount of agents or their available memory, but by using the agent for simply spawning a Job in our Kubernetes Cluster.

A Kubernetes job is a pod (one or more container) executing a precise task limited in time (in opposition with a micro-service or a server always up and running). A job is running until it completes either by failing or by succeeding.

Here, we want one job to take care of running the e2e tests, bringing a browser when it’s required and storing the reports from our framework in some persistent storage.

2. Let’s containerize the things

So what we are going to do, at first, is to put our tests in a container. For doing so, let’s first describe what are our steps for running the end to end test. We have a linter for testing the source-code, then we have a compilation and finally we can run our scripts with NodeJS.

Here are the Docker files using the multi-stage builds for creating the container and the JavaScript file testing “our site” with its package.json.

To build the image, make sure that you are using Docker 17.09+ (I am using Docker 17.12.0-ce-mac45, channel Edge). Clone the Gist in your workspace, cd to the directory, start docker-machine, source your environment then build the image:

$> git clone https://gist.github.com/1e4518f09a4d28b32d78bcce1411c6e3.git gist
$> cd gist
$> docker-machine start
$> eval $(docker-machine env)
$> docker build -t my-repository-name/e2etest    

Here it is, your container is built. In order to test our scripts with Chrome, let’s mount an image containing the Chrome browser. For doing so, we will use the official Chrome standalone image from SeleniumHQ.

Run this command in your terminal:

$> docker run -d -p 4444:4444 -v /dev/shm:/dev/shm selenium/standalone-chrome    

We are running an image executing Chrome as a daemon. You can access the browser on the port 4444 using the ip address of your Docker machine (do docker-machine ip in your terminal to see the ip address).

Now, let’s run our tests on our machine:

$> docker run -e BROWSER=chrome -e SELENIUM_HOST=$(docker-machine ip) -e SELENIUM_PORT=4444 -t my-repository-name/e2etest
Title is: WebdriverIO at DuckDuckGo    

As you can see, when we run the container in non-interactive mode, it runs our end to end tests.

Let’s take a look at how to manage the reports. We add to our e2e scripts a call to the saveScreenshot method of Webdriver.io in order to create a screenshot from the browser of the page we are testing. We also output some verbose in the report directory.

Your script should look like this now.

Now that we output the logs and create a screenshot in the report directory, let’s mount a volume to our Docker image for storing the files.

We will use a tiny script using fs.watch. Each time a file got modified, the script pushes it to a Google Cloud Storage Bucket. This way, we have a persistent storage for our reports.

Here is the small script watching the files and uploading them to the bucket with its Docker file and its package.json.

Clone the Gist in your workspace, build the container.

$> git clone https://gist.github.com/3efff749b08f0b95ca0fa907a7c56597.git gist-catcher
$> cd gist-catcher
$> docker build -t my-repository-name/catcher    

When it’s done, create a directory report in the same folder.

Now, let’s run the container in a terminal and use another terminal for running the e2e tests.

You should see something equivalent:

Terminal 1

$> docker run -e BROWSER=chrome -e SELENIUM_HOST=$(docker-machine ip) -e SELENIUM_PORT=4444 -v /path/to/workspace/report-catcher/report:/var/app/report -t my-repository-name/e2etest
Title is: WebdriverIO at DuckDuckGo    

Terminal 2

$> docker run -v $PWD/report:/var/app/report -t my-repository-name/catcher
chrome.2018-01-15T23-34-04.1.log change
chrome.2018-01-15T23-34-04.1.log change
chrome.2018-01-15T23-34-04.1.log change
chrome.2018-01-15T23-34-04.1.log change
chrome.2018-01-15T23-34-04.1.log change
chrome.2018-01-15T23-34-04.1.log change
chrome.2018-01-15T23-34-04.1.log change
chrome.2018-01-15T23-34-04.1.log change
chrome.2018-01-15T23-34-04.1.log change
chrome.2018-01-15T23-34-04.1.log change
chrome.2018-01-15T23-34-04.1.log change
chrome.2018-01-15T23-34-04.1.log change
chrome.2018-01-15T23-34-04.1.log change
screenshot.jpg change
screenshot.jpg change
chrome.2018-01-15T23-34-04.1.log change
chrome.2018-01-15T23-34-04.1.log change
chrome.2018-01-15T23-34-04.1.log change    

The catcher and the e2etest containers are linked using the volume $PWD/report.

3. Deploying our containers in Kubernetes

Now that we have everything running on my machine, let’s deploy the containers to a Kubernetes cluster.

We will assume that you already have a Kubernetes cluster setup. Also, we will assume, such as we do, that you are working with the Google Cloud Platform and have the sufficient credentials for creating an Application Service Account able to write in a Google Cloud Storage Bucket. What will be required in the next steps is also to have access to a Docker repository for pushing the images.

In order to do so, do the following:

$> docker push my-repository-name/e2etest
$> docker push my-repository-name/catcher    

Let’s start the Kubernetes manifest.

At first, we create a given namespace for running our jobs and declare the main container of our job.

apiVersion: v1
kind: Namespace
metadata:
    name: my-end-to-end-tests
---
apiVersion: batch/v1
kind: Job
metadata:
    name: e2e-chrome-mysite-1
    namespace: my-end-to-end-tests
spec:
    parallelism: 1
    completions: 1
    backoffLimit: 3
    activeDeadlineSeconds: 20
    template:
        metadata:
            labels:
                app: "e2e-chrome-mysite-1"
                run: "1"
                version: "1"
        spec:
            restartPolicy: Never
            containers:
            - name: chrome
              image: my-repository-name/container-e2e
              env:
              - name: "TARGET_URL"
                value: "http://www.mywebsite.com"
              - name: "BROWSER"
                value: "chrome"
              - name: "SELENIUM_HOST"
                value: "127.0.0.1"
              - name: "SELENIUM_PORT"
                value: "4444"

We (try to) restrict our job for running only one time on the cluster by setting parallelism to 1 and completion to 1. I mean try because it may happen that the pod is run twice. We also restrict the amount of backoffLimit and the activeDeadlineSeconds for avoiding an infinite loop while trying to start the pod.

The SELENIUM_HOST is set to 127.0.0.1. The standalone browser will be in the same job, so all the containers are sharing the same IP address.

We have to create the secret required for accessing the GCP APIs. It’s needed for our sidecar container pushing the reports to the Google Cloud Storage Bucket. Such as what we did locally, we share also a volume with the previous container.

After the namespace declaration, add the secret declaration:

---
apiVersion: v1
kind: Secret
metadata:
    name: e2e-chrome-mysite-1-secrets
    namespace: my-end-to-end-tests
    labels:
        app: e2e-chrome-mysite-1
    type: Opaque
    data:
        gcloud.json: "my-base64-encoded-file"
---    

Add the following at the end of the previous container declaration:

volumeMounts:
    - name: reports
    mountPath: /var/app/reports    

Let’s declare our new container:

- name: catcher
      image: my-repository-name/catcher
      env:
      - name: "VOLUME_PATH"
        value: "/var/app/report"
      - name: "GCLOUD_APPLICATION_CREDIENTIALS"
        value: "/home/.config/gcloud/auth.json"
      - name: "GCLOUD_BUCKET_NAME"
        value: "my-bucket"
      - name: "GCLOUD_PROJECT_NAME"
        value: "my-project"
      volumeMounts:
      - name: reports
        mountPath: /var/app/reports
      - name: secrets
        mountPath: /home/.config/gcloud    

And now, we declare our volumes:

volumes:
    # extended memory for the browser
    - name: extended-mem
      hostPath:
        path: /dev/shm
    # shared volume where we write the report from the main container.
    - name: reports
      emptyDir: {}
    # secret volume containing the app credentials for the side container pushing to GCP
    - name: secrets
      secret:
        secretName: e2e-chrome-mysite-1-secrets    

As you already noticed, I added one more volume /dev/shm. It is used for extending the memory of the last sidecar container we need, the browser.

The browser could come from anywhere, as soon as your cluster is able to pull the image.

Such as what I wrote earlier, we will use a standard SeleniumHQ Docker image for a standalone Chrome.

Add these lines after the declaration of the catcher container:

- name: chrome-browser
image: selenium/standalone-chrome:3.8.1-erbium
ports:
- containerPort: 4444
env:
- name: SCREEN_WIDTH
    value: "1680"
- name: SCREEN_HEIGHT
    value: "1050"
volumeMounts:
    - mountPath: /dev/shm
    name: extended-mem    

We are providing the volume /dev/shm to the container in order to reduce the amount of chances for the browser to crash.

You can find the complete manifest here.

We are ready to deploy our job. Let’s run the following commands in a terminal:

$> kubectl apply -f ./kubernetes.yaml    

If you run kubectl get ns , you can see that your namespace has been properly created.

Let’s now see if the pod has started as well:

$> kubectl get po -n my-end-to-end-tests -w    

You should see the status of your job.

In order to see the logs, run the following command:

$> kubectl logs e2e-chrome-mysite-1-rand1 chrome -n my-end-to-end-tests    

You can see the logs coming from each container, just play with the second parameter of kubectl logs, the value can be chrome, chrome-browser or catcher.

When your main container completes, you can get its status and its exit code using this little bash script.

For going further, you could automate all the steps we went through for triggering the job and killing it in your CI.

Here, we did everything manually (triggering the job, killing the job). However, today, you can find some tools for managing and monitoring your Kubernetes jobs such as Azure Brigade or Apache Airflow.

Also, because we are pushing our reports to a Google Cloud Storage Bucket, you could quite easily implement some parsing of them using a Google Cloud Function. From that, you could build a custom alerting system using Slack.

You have a lot of options and it is upto you to choose what you'd like to do.