🤠 "Where have all the hackers gone?" + a way to discuss programming languages 👨‍🎤

Sunday, May 14, 2023 :: Tagged under: engineering essay. ⏰ 14 minutes.

🎵 The song for this post is CHEMICAL LOVE, by "Kaleb James and Chey" for the game Bust-a-Groove. 🎵

I'm talking with friends and coworkers about programming languages (surprise), and I'm landing on a rough shape that these conversations take. I'll share it here and hope you find it useful, especially if we ever talk about them. Then I'll use that framework to make the case that the way we talk about them in company settings strikes me as fear-based and bogus (🔥s ahead).

When you compare languages, what are you actually comparing?

While visiting a friend, he noticed I had a Golang shirt, he told me he loved Golang, I told him I like the shirt but the language less. He got excited at the idea of "getting into it" later, and after dinner we plopped down on couches and said "alright! Let's get into it!" I wanted to talk about its garbage collector (1, 2), how goroutines/channels are a delightful abstraction but I prefer BEAM's abstractions because it allows for Supervisors, Golang's very loose approach to correctness… and he didn't want to talk about any of that. He, on the other hand, emphatically talked about how much he loved that "the Go developers knew that all you need is a for loop. Someone brought Scala into my company and I hate the mental shift."

This was not a fruitful conversation, I think we both felt like we weren't valuing what the other cared about. When people talk about languages they like or dislike, I group the things people talk about into three broad categories, which I'll call soil, surface, and atmosphere:

Soil is the properties of running code in that language. Most of it is when the code is actually running (basic performance characteristics, "is it a binary or interpreted by a VM," scheduler and/or relationship to multicore/parallelism, garbage collector) but a more broad generalization is "everything that isn't code editing that doesn't directly involve "community,"" so I'll involve things like build times and some properties of its tooling. So the stuff I brought up in the Golang discussion are "soil," but so is:

• That Python code is often hundreds or thousands of times slower than other languages.

• Languages with long compile times (Scala, Kotlin) vs. short ones (Golang).

• If your language has generics, does it implement them with type erasure, monomorphization, or something else?

• Erlang having a preemptive scheduler, and how iolists + how string concatenation works under the hood mean you can render the template of doom.

Surface is what people usually think about when comparing languages: the features! Source code! It's whether it has your favorite looping construct (or doesn't, per the "only need a for loop"). Syntax, FFI, which regexes it supports, the semantics of its specific object system. Other examples are:

• Less/Sass and CoffeeScript were pure "surface" plays: not about the core capabilities, but seemingly expanding them with pure "fixes" to suboptimal surfaces.

• "Ruby is so much better than Java! Look at how you open a file in Java (shows you 30 lines) vs. Ruby! (2 lines)"

• Much of the appeal of MERN or MEAN stacks is that "it's all JavaScript/JSON," meaning you can reduce the amount of "surface" to learn.

• Strong reactions on both sides of CSS-in-JS and Tailwind feel "surface"-ey, though they can have "soil"-ey impacts.

Finally atmosphere: these are things that aren't the language or its code, but the broader community. Hiring, Stack Overflow answers, number of stars on its popular GitHub projects, number of packages in their repositories. I'll go a little further and include downstream effects of that community, e.g. "does the language have a VSCode plugin and language server" seems to be more a function of atmosphere than the others:

• Most "you can hire React developers easily" is completely divorced from any technical discussion of React as a framework. Ditto Java, Python, Ruby, JS…

• Most of a language's culture: Elm and Clojure both have something like "if Evan/Rich didn't think you should have that thing you wanted, have you considered that maybe you're wrong? 🙂". Golang telling you you're dumb and wrong for wanting generics for a decade before shipping them. Scala having all that silly drama.

A light note for exceptions: as Hillel says, "all taxonomies are broken, full stop." Here, "tooling" fits all three: usually, the existence of tooling (like the language server) is a community thing, but the properties of that tooling is very "soil," based on decisions the core developers made and with big downstream effects on the working with the code day-to-day (e.g. compile times) or even after it deploys (e.g. "can you do postmortem debugging?" — kind of a runtime thing too).

The other is "static types"… I feel like this is surface, but you can argue it's soil.

Anyway, with this in mind, here are some observations!

Discussions get less concrete as you go upwards

I feel like when discussing language properties, "soil" is often pretty concrete (e.g. general performance, memory safety of Rust vs. C++ programs), "surface" is more feelings-ey but still grounded somewhere in reality (BEAM languages can have supervisors because of OTP and immutability, Go programs can't; but what this means for the safety of your program and ultimate business impacts are hard to measure). Whereas "atmosphere" questions are extremely based in feelings: "you can't hire FP developers!" (have you actually tried? almost every FP shop I know has never had trouble finding enthusiasts, and per the Python paradox, they're often quite good…) "There isn't library support!" (most repos have an awesome repo collection, what specifically do you need?).

I'm not saying atmosphere questions don't matter, just that I think they're the hardest to qualify the impacts on. You can quantify, for example, that npm has n packages and Cargo has m, but what do those numbers actualy mean for the purposes of a long-lived, business-effective codebase? You can estimate from sources like HackerRank or Leetcode or Greenhouse or GitHub Jobs how many Java developers there are vs. Scala ones, but how does that number translate to finding exceptional people for your specific company? Are you hiring… 30,000 people this year? Unless it's a raw "bodies" problem (and most companies, the harder part is finding the right bodies) how is that super relevant?

This frustrates me because I feel like…

The current thinking is "atmosphere at all costs!"

My computing career came of age in the late aughts. It was a very different time: people played with technology choices a lot more! Twitter was suffering with Ruby on Rails, so they tried a new language with promise: Scala. Heroku invested in Erlang, and so did a little messaging app called WhatsApp. In 2006, ITA Software received $100m in funding even though they were 10 years old and written in Common Lisp (they sold to Google in 2010 for $700m in cash. This was impressive back then; Instagram didn't have its industry-breaking $1bn acquisition until 2012).

Today, we're afraid to do anything that's not JavaScript, Java, Python, or Ruby. I have a feeling 10 years from now it'll just be JS. We all read that excellent McKinley article about "innovation tokens" and we decided to be floor-avoiders instead of ceiling-breakers. Engineering leaders lead from a place of fear and risk-aversion instead of optimism and believing in their team. Where have all the c̶o̶w̶b̶o̶y̶s̶ hackers gone?

Yes, using an exotic technology may spend an innovation token, but like a lot of things in life, innovation tokens are completely made up, it's like talking about the finite number of "love tokens" you can give your spouse in a given year. I'd like to offer a different framing: maybe the biggest way to reduce your innovation tokens is to have a small imagination. Imagine receiving a bag of tokens after you pick your a tech stack. Picking a great, appropriate technology that's not one of the Big Four, you have to reach into a bag to spend a token, but then you see your bag has 6 left; when you pick a Boring one, you don't spend any immediately, but there are only 3 in the bag.

Seen here: happy engineers feasting on innovation tokens they've been gifted by a CTO who trusted them. From the dinner scene in Hook.

Is it hard to learn a language?…

People often talk about great engineering teams (and the teams they'd like to build and work on) in pretty high-minded ways. Raise if your hand if you feel this way:

• I prefer to work with an excellent software engineer, who doesn't tie their identity to a specific language or technology (e.g. would prefer a great hacker than someone who identifies as a "Ruby developer" or "JS developer.")

• My preferred teammates are people who understand the concepts underneath the technologies, and don't equate the technology to the entire stack (e.g. "databases" to this person doesn't just mean "Oracle" or "Mongo", someone who understands BOM and DOM as distinct from "Svelte").

• The kind of person I want on a team has had exposure to multiple technologies, can articulate tradeoffs between them, and is capable of understanding and working with new ones.

Like with equal housing rights, in theory, almost everyone polled agrees on a set of neutrally-stated positive ideals. And also like equal housing rights, somehow, when it comes time to practice or vote on those ideals, the outcomes don't match. It turns out, many engineering teams are perfectly fine hiring "Java developers" who sweat if you showed them some Python, people who can't effectively articulate tradeoffs between various tech stacks, and people who, if asked to spend a week getting up to speed on something new, will instead complain and argue for using something that will be suboptimal for years and years to follow.

Here's an interesting question: as with housing rights, how come people feel one way in their head but forget those ideals when actually voting with their actions? I touch on this more later (for the tech case). But take a moment to reflect on your ideal engineering team, then reflect on one you've built or are building, think of where they may be different, and why. "You go to war with the army you have, not the army you want." Yeah, but why?! You built, and are building this team!

For all my advocacy in this post, it may surprise you to hear that I believe it takes years to be excellent at a language. It's not just syntax, it's soil and atmosphere: common bug flows and how to spot them, footguns, tooling, library ecosystem, culture. I have a bunch of tips for learning a new language, especially weird ones, and you'll even note that even that post starts with please please please be careful before introducing one of these to your companies.

So why am I making all these arguments here for Going For It? Because I think in the presence of experienced mentors, it only takes a few days, maybe a week or two, for a developer to get up to speed on a new language to an adequate level, especially with mandatory code review. This may sound preposterous, but in my observation…

…most people don't know much about Boring Stacks everyone chooses. Also, they don't generalize!

The thing about most Python Developers™ I've worked with is that they don't even know that much about Python. They couldn't articulate when and how to use abc vs. just inheriting from object, or why you might prefer namedtuple over dataclasses. They won't know that import x.y.z will import x/__init__.py and x/y/__init__.py and x/y/z.py (or x/y/z/__init__.py). They couldn't tell you who GIL is or why you shouldn't def send_emails(list_of_recipients, bccs=[]):. This game is especially fun when talking about JavaScript 😉

Most developers have major gaps in their understandings of the tools they use. This is fine. But the thing about "pure atmosphere" arguments like "it's easier to hire people with [X] experience" is, in my observation, that their experience isn't necessarily that deep, and that people who think they're hiring skilled users of these languages are lying to themselves. Many people across the 7 companies I've worked for are best described as "adequate."

Training isn't free, but it's a fixed cost per developer, and especially with skilled mentors, probably smaller than you think. I find it extremely unlikely that the fixed cost of 2-4 days of training is less than the cost of superlinear build times + managing 1000s of Sidekiq or Celery queues when you could have just picked Go.

---

Another thing: these techs don't actually standardize. I've worked at 4 companies using Flask; none had comparable app structure or used the same plugins (yay microframeworks!). "Just use React," but for a while, it was "to state manager or not state manager (Redux)," then it was "to hooks or not to hooks," then it was "to CSS-in-JS or Something Else (and/or BEM, or now, Tailwind)." Every company I've worked for builds their own Design System and/or component library that has its own weird calling conventions. No two React companies I've worked at tested the same way, or had transferable knowledge on their asset pipeline or JS transpilation options.

Why are we at "atmosphere at all costs"?

These can probably be separate blog posts, but this is long enough so I'll just bullet out some flamebait and maybe follow-up later:

A decade of 0% and dumb money made tech stupid; "VC tech company" became a playbook. Startups used to be the Wild West, and VC was expected to go to 0 because the bets were so dumb and big. Wild-eyed creatives were more welcome there. Over time "venture capital" kept the name and tried to keep the cowboy branding of Big Bets, but in practice became more akin to regular investing, and VC-backed tech developed a playbook that expected regular returns. Punks in garage bands who might have "the juice" got replaced with clean-cut studio musicians who got Music degrees. This led to risk aversion and a fetishization of Boring.

Infusion of traditional Business and Capital culture, which traditionally serves smoother-brains, the boring, and the unimaginative. Look at airport business books: they're often quite braindead. Like what happened to The Rude Press, edgy media outlets that were "merely" profitable, and watching blue checks on Twitter licking their lips thinking that we can replace TV writers with AI (or that they're a whole artist by typing "hot cyberpunk girl" in a Midjourney textfield); as tech got more successful, we got more of the class of person in the business game who gets uncomfortable with creatives and their expression, and wants to believe you can create something amazing and innovative with Known, Controlled, Riskless Process.

Engineering Management and Leadership culture becoming A Thing; people became herd animals around "Boring." The last decade we saw a rise in the Tech Managerial class and its thought leaders. Managers from other domains without tech knowledge didn't do well, but neither did engineers promoted to management. What was needed was a hidden, Third Thing. A lot of influencing is speaking authoritatively even if narratives aren't settled; a favorite snowball fight in this community was Manager READMEs (for, against). Despite having lots of Assured Professional Voice, I've observed Engineering Leadership to be a winding road in practice (excellent short Brandur read here). I think "Boring" is a fine strategy some of the time, but in the spirit of being an "adult in the room," I think it caught on as a Universal Good, and the class of managers herd animal-ed their way into it.

The newest, easiest way to Fear the Unknown. The funny thing about "atmosphere costs/benefits are the most important" when they're the hardest to make concrete: I feel like the late aughts that I'm lionizing used to be about "surface," and were similarly, maddeningly fuzzy. Functional programmers insisted without evidence that their programs were More Correct. Dynamic types people insisted without evidence that their programs weren't less correct. I think we were playing the same game of appealing to feelings, but before a decade of increased CS enrollment, and boot camps, and Stack Overflow, and GitHub, and advances in text editing (Language Servers, the monoculture of VSCode), we couldn't talk about atmosphere, so we went as high up as we could.

Meaning: it is, as it always was, a cope for the terror of leading an engineering team. You're at sea with the livelihoods of many in your decisions, and while you're pretty sure you know how to read the stars, it's the second week of cloudy skies in a row; you start to get superstitious. You're more inclined to believe the Authoritative Voice of a Calm Professional. You'll take fewer risks in the things you can control because you're so afraid of the things you can't.

(sidenote: more of y'all need to work in the arts for a bit 😉)

---

I have a lot more thoughts on those points (there are a million ways to fuck up bringing exotic tech and with all this said, most people shouldn't do it; also, "Boring" has saved a ton of companies. But I promised flamebait!). Takeaways:

• Consider, when talking with someone else about about technologies: is it a soil, surface, or atmosphere conversation? Decide together which one you'd like to have, and the limitations of the layer you picked, because it's easy to talk past each other if you don't.

• Re-evaluate what you consider "boundaries" between technologies. Is React really just React everywhere? If your team uses Retool, for example, isn't this also a programmable interface with various bolted-on technologies with access to your production datastores? Why is adding Retool easier than a Java service? And do consider the "training time" of learning Retool onerous?

  Another fun thought example: do you consider it so dangerous to build native phone apps instead of React Native, Flutter, or LiveView Native? Why is that different?

• Tech choice won't be ultimately responsibe for the success of your company. But it will definitely shape what your path looks like, and it's a mistake to say "tech doesn't matter." Bleacher Report went from 150 Ruby servers to 5 ("probably overprovisioned") Elixir servers after a port. Languages with proper concurrency like JVM, BEAM, or Go don't require Sidekiq or Celery queues and additional workers. BEAM's preemptive scheduler means you don't get a noisy neighbor issue like Lyft dealt with in Python, or which will probably bite you in Node's "event loop but on a single core." Training isn't free (let me say that again: it's not free!) but some tech was purposely built to be efficient and others scale horribly.

• Reconsider "sacred cows" in all areas of tech, generally. Microservices aren't inevitable (consider FB's Blue App, Dropbox, YouTube, Instagram: all monoliths). PHP was the laughingstock of the last decade and yet it powered Slack, Lyft for many years, and Hack still runs Facebook. "Cloud is better," except Stack Overflow has much more data and traffic than you do and runs on 9 machines. WhatsApp got acquired for $19b while doing manual, bare-metal deploys.

  And: if the conditions are right and your team has the fire for it, you can run in a tech stack that isn't JS, Ruby, Python, Java.

Additionally:

• Take a moment to ask yourself: what don't I know about my favorite technologies? What would be unlocked if I did?

• What training or exercises could you and/or your team undertake to level up on the tech stacks you do use?

---

If you liked these, you might like:

Edit (5/18/2023): The original version of this article claimed Golang had a cooperative scheduler (and linked to this 2018 article), but thanks to some friends at HN, Go has had a preemptive scheduler since 2020.

🧘 The rest is just as important 🛁 →

Thanks for the read! Disagreed? Violent agreement!? Feel free to join my mailing list, drop me a line at [email protected], or leave a comment below! I'd love to hear from you 😄

Email