They Might Never Tell You It’s Broken

This blog post is a public service announcement (or maybe a reminder) for anyone working on a programming project that they already have, or intend to release to the public, be it as something open source, a pet project, or a startup.

I’ve worked on a few successful open source projects over the last 10 years, often with multiple contributors. I would consider these projects successful in that they got hundreds of stars on GitHub and each attracted multiple open source contributors. I actually shut down one of these project because reviewing pull requests alone was becoming a second unpaid job on top of my regular job, taking multiple hours out of my evenings after work, which became exhausting, but that’s a story for another post. What I want to tell you about today is  something important, that I believe any developer should know, but I personally didn’t understand until I had been working on open source projects for a few years.

As part of my PhD, I developed Higgs, an experimental JIT compiler for JavaScript written using the D programming language . I developed it on GitHub, completely in the open, and wrote about my progresson this blog. Pretty soon, the project had 300 stars on GitHub, a handful of open source contributors, and I was receiving some nice feedback. It made me happy to have so many people taking notice of my research work. As part of Higgs, I had written my own x86 machine code generator, which enabled it to do machine code pirouettes LLVM couldn’t. I did all my development on Linux, but had done my best to keep the code as portable as possible, and so, that code would work fine on MacOS as well, I had assumed. Unfortunately, I was wrong.

About a year into its development, Higgs had enough of a small community that it made sense to have create a chat room to exchange with other contributors and users. About a dozen people joined over the next two months. One day, someone I had been exchanging with on the chat room for two weeks reached out to me to signal a strange bug. They couldn’t get the tests to pass and were getting a segmentation fault. I was puzzled. They asked me if Higgs had MacOS support. I explained that I’d never tested it on MacOS myself, but I couldn’t see any reason why it wouldn’t work. I told this person that the problem was surely on their end. Higgs had been open source for over a year. It was a pretty niche project, but I knew for a fact that at least 40-60 people must have tried it, and at least 50% of these people must have been running MacOS. I assumed that surely, if Higgs didn’t run on MacOS at all, someone would have opened a GitHub issue by now. Again, I was wrong.

The problem, it turned out, was that MacOS had more strict requirements for keeping the stack pointer aligned. This wasn’t difficult to fix. The more important lesson, that I didn’t understand until that point, is that you can’t count on the people trying your project to quickly and reliably signal bugs to you. Most of the time, if it doesn’t work, they won’t report the problem. There are a few reasons why this might be:

• They assume that someone else has already reported the problem, and there would be no point in saying anything. The bigger your project, the more likely people are to assume that someone else has already reported the issue.
• They think the fault might be on their end, they may be confused and feel too embarrassed to reach out for help. Nobody wants to look stupid or get told to RTFM , and so, they choose silence.
• They are just trying your project out of curiosity, and are in no way invested. They will find an alternative to your project, or go do something else.

It’s a horrifying thought, but it could be that for every one person who opens an issue on GitHub, 100 or more people have already tried your project, run into that same bug, and simply moved on. So, what can you do? You can also encourage people to report bugs. I visibly write in my GitHub README that reporting bugs is encouraged and welcome: “Please tell me if something is wrong, you’re helping me make this project better.” Another obvious thing that you can do is to have robust automated testing. Some continuous integration services can automatically test on both Linux and Mac.

More broadly, if you want your project to be successful, I think it’s important to try and put yourself in the user’s shoes. Every once in a while, try installing your software from scratch, along with all the dependencies. Ideally, you want your installation process to be as simple and frictionless as possible. If your software requires the user to perform 20 steps to get it running, you’ll be losing potential users even before they have a chance to figure out if it works on their system or not. In general, I find that writing software with a minimalist design philosophy , minimizing external dependencies as much as is reasonable to do so, will help you avoid bugs, and streamline your installation process.