Duolingo uses Codespaces and Tailscale for secure remote development

(Duolingo)

Since language learning app Duolingo launched back in 2012, all its coding was done locally on engineer laptops or desktops. There were a few problems with this approach.

Firstly, when new developers joined the business, the amount of time taken to get them set up and ready to code; second, the ability to work and collaborate in the cloud. As Art Chaidarun, Senior Staff Software Engineer at Duolingo, explains:

Normally, software engineers write code that runs on their laptop. They write it on their laptop, and then we push it to some repository, like GitHub. The way a remote software engineer works is, the editor is online. It's hosted in a web browser. We're coding in the cloud.

In simple terms, it’s the same as the difference between local Microsoft Word where you’re tied to a particular physical device, and Google Docs, which you can connect from any laptop.

Duolingo started using GitHub Codespaces around a year ago, which allowed its engineers to carry out their work in the cloud. Codespaces made it easier and quicker to set up the development environment and be ready to code within a few minutes. It's also more consistent as the online environment is generated the same way from the same templates for everybody, as opposed to local instances where you rely on developers following instructions correctly.

However, the Codespaces deployment introduced a new requirement, as there's no way for it to get behind Duolingo’s firewall to access the private resources needed for development. Chaidarun says:

A lot of our development environment we don't have to expose to the world, it's behind a firewall, which was our old L2 VPN. But we needed a similar solution for Codespaces.

Collaboration

For this, Duolingo turned to Tailscale, a VPN for DevOps and infrastructure teams wanting to collaborate securely.

For engineering teams needing secure access to dynamic hosts, services and applications, VPNs are the answer. However, traditional VPNs can have latency and security issues due to public endpoints and unencrypted traffic. 

Built on the open-source WireGuard VPN protocol, Tailscale is a way round these problems, offering a zero config VPN for building secure networks. Chaidarun adds:

We use Tailscale for accessing our private resources locally on our laptops, where we rely on our old VPN to secure certain private resources on AWS and the local office server. If we're going to be coding in an account with Codespaces, then we need some way to access those. That's where we use Tailscale.

Chaidarun briefly looked at other options like ZeroTier, but chose Tailscale due to the thorough and clear documentation.

It even came with code samples that I could easily adapt to our use case, which was far ahead of the others. I also remember from reading Hacker News over the past many years, that Tailscale very frequently comes up as an option that people are happy with.

While Duolingo is using Tailscale and Codespaces for remote development, it’s not in the sense of remote working: the majority of the company work out of the firm’s Pittsburgh office. Chaidarun says: 

It's basically coding on your computer versus on a website. When we work in Codespaces, the code is actually not on our computer at all. If someone steals my laptop, it's not really much of a concern as far as that goes.

The difference between the coding process before using Codespaces and Tailscale and now is night and day, according to Chaidarun:

Before Codespaces, it was almost a nightmare to get started with writing code at Duolingo. It was a trial by fire, you’d spend your first couple days or even your first week at Duolingo just setting up your laptop, installing all the tools and the library software packages that you need to write code. You have to follow these long instructions very carefully, they're probably out of date, they're probably wrong in some places. Everyone just had to do this, every new engineer, so it's a big headache.

With Codespaces, everything is more standardized and in the cloud, so engineers don't have to rely on tweaking their laptops locally. This has reduced the time taken to set up the main code base from days or even a week, down to just seven or eight minutes to spin up a new Codespace. Once that's done, developers can write code in Duolingo’s main code base the same way as before.

There are currently just over 100 engineers using Codespaces and Tailscale, which is a little over a third of Duolingo’s engineering organization. If one of the team wants to use it, they can, but it's not a requirement. Chaidarun adds:

I'm the one who led that movement and we didn't want to force feed it to people. We wanted adoption to be more organic, so we've never required that people try remote development or Tailscale. But we have seen a linear growth in usage over the past year, especially when the summer interns came and they had nothing locally to begin with. They really enjoyed Codespaces.

The key benefits have been ease of onboarding new developers, and making it easier to troubleshoot user problems. Chaidarun explains: 

If all the code is living on everybody's laptops, then eventually people install weird things or they break their configuration somehow and need to figure out how to fix them. Whereas if everything is running in the cloud and you can just tell them to throw away the environment, just spin up a new one, then it takes a few minutes.

Tailscale is working without any problems at the company. For the authentication process, users sign in with Google SSL, which everyone is on already, says Chaidarun: 

Over the past year, I don't remember having anyone complain about problems with it. Often nowadays when somebody who's been here for a while complains about problems they’re having with their local environment on their laptop, a common reply is - just use Codespaces. That usually fixes it for them.

Duolingo has a weekly release cycle every Wednesday, and most engineer teams work in two-weeks sprints. Once the goals are agreed, the engineers go off and do their own thing, and then review each other's code afterward. While pair programming or pair coding isn’t a prevalent part of the development culture at Duolingo, if it were to become more common, Codespaces would be a natural fit. Chaidarun concludes:

We're not doing quite as much of that yet, but it's much easier to collaborate in real time that way.