Ok, so Phoenix 1.6 is out and it brings a major change in the default assets setup by replacing webpack with esbuild, also dropping Node and NPM as dependencies for new projects. Both Chris McCord and José Valim have explained these changes (all nicely wrapped up in this piece on the fly.io blog). It generally came down to the amount of overhead and headache that the Node + NPM + webpack combo was putting both on the Phoenix team and some Phoenix app developers.

This is 100% understandable and I actually admire the team for not hesitating to drop webpack after it replaced brunch in Phoenix 1.4. Phoenix needs to keep up with the pace of the fast-evolving front-end universe. And if something just doesn’t work and there are alternatives, why stick with the pain-causing option?

But now let’s put the struggles with Node + NPM + webpack aside and instead discuss what these changes really mean for us, developers and our day-to-day efforts to paint and script our Phoenix apps.

Damn, it’s fast!

If you’ve previously developed a Phoenix app with webpack, you’ve probably noticed what happens when you restart the dev server while already having an app open in the browser. Phoenix server starts fast, you happily start clicking around, then suddenly after a delay (~10s and up, depending on the complexity of assets) the page gets reloaded again due to Phoenix live reload finally picking up the new assets. Not cool!

Well not anymore, because esbuild is so crazy fast that it’ll finish its business right away and the static plug will receive these changes even before the socket gets to warm up. What’s great is that this holds even for complex projects that import monsters like three.js or monaco editor.

On production, you’ll also see vastly reduced compilation times and resource usage which results in following advantages:

1. Shorter time to go live.
2. CI cost reduction due to shorter build times & less resources needed.
3. Less strain on the server (e.g. in environments like dokku).
4. Simplified Dockerfiles & pipelines if you drop NPM (discussed below).

Here’s the real-world example. One of my recent Phoenix projects was running on a VPS with 1GB of RAM (which considering Phoenix efficiency is enough for many production deployments). After adding monaco editor, even though webpack was still building reasonably fast on my local machine (somewhere around 20s), docker build that worked before has started to get killed due to the excessive memory usage.

This was a real blocker and I had following choices:

1. Completely revamp my CI/CD pipeline (e.g. to build on separate, beefed-up CI machine & push to paid docker registry).
2. Throw money at it and buy more RAM (even though the live app may never ever need it).
3. Switch to esbuild and not only solve this problem once & for all, but also get all the extra perks listed above.

Well, this wasn’t the hardest choices that I had to face as an engineer. :)

Don’t get me wrong. Options 1 and 2 also have their place (mostly in more mature apps), but choice 3 is just so elegant & pragmatic that it’s a no-brainer + by no means does it rule out the other two.

Of course, there’s always a price. And here the price is a less optimized bundle that doesn’t go through heavy-duty optimizers such as terser. Still, it turns out the difference is often ~10% so it may not be worth it. At least this is what many large players are concluding recently, switching their projects to tools that don’t take hours to do the job.

Life without NPM…?

The issues around NPM may’ve been the trigger, but is ditching NPM a good default choice for an average Phoenix project? And if not, would esbuild even work & make sense with NPM still a part of the equation? What about projects that do rely on NPM packages?

These questions may seem easy to answer for experienced front-end devs & bundling ninjas, but not so much for engineers focused on the backend (e.g. many Phoenix & LiveView devs). Webpack in itself was confusing enough and now there’s another paradigm shift…

Well, I went through all these dilemmas recently and here’s what I came up with so you don’t have to.

My oh-so-controversial opinion is that ditching NPM, as much as it works nice for maintainers of mix phx.new, is almost never a good idea from the perspective of the project itself — even the simplest one. Let’s see what the freshly generated Phoenix project does to get away without Node:

1. It pastes the Milligram CSS library code along with version in the comment into assets/css/phoenix.css.
2. It pastes the topbar JS library code along with version in the comment into assets/vendor/topbar.js.
3. It uses the Elixir esbuild package to run esbuild, with its version and default options specified in config/config.exs.
4. It tells esbuild to consider the deps directory as an alternative to assets/node_modules (to pull Phoenix‘s own JS libs).
5. It overrides some of esbuild options in config/dev.exs (those related to watcher) and some in mix.exs (those related to deployment).

The gain is clear — developers will be able to bundle their projects even if Node/NPM/Yarn ceases to exist (or, for a more realistic example, suffers availability or yanking issues). But here’s the price that we’re paying:

• CSS/JS libs are added via copy-paste
• Their naming is and will always be inconsistent (Milligram’s case)
• They must be upgraded manually with comment-pulled versions
• They are no longer subject for any security checks
• esbuild version is put in a weird place (Elixir config)
• Its options are spread across many Elixir files
• Its package lookup covers all Elixir packages (even those without JS)

Overall, this IMHO is a major step backward for fresh Phoenix 1.6 projects compared to what we were getting before. And while it’s obviously a subjective judgement, I think that the risks are not serious enough to justify what basically is a step back into the chaos of the pre-NPM era.

Of course, these are just the defaults — docs for Elixir’s esbuild clearly state that NPM is still supported and you can always pass --no-assets and do things 100% your way. But it’s easy to underestimate the power of defaults, especially those that cover area outside of target audience’s expertise — which is the case of Phoenix devs and JS bundlers.

We can already see an emergence of packages such as bulma-elixir that build on top of this default to avoid NPM by “any means necessary”. I saw this years ago as a Ruby on Rails developer and such wrappers (e.g. font-awesome-rails that I’ve been using) really are a majorly bad idea. Why?

• These packages are always behind the original source
• Many of them are born out of immediate need & get abandoned
• Like the copy-pasted assets, they won’t be covered by security checks
• We won’t ever have wrappers for even 1% of what NPM holds

To summarize, things may work out at the beginning — looking nice & clean, but it’s a dead end — a tech debt waiting to bite us from day one.

The best of both worlds

Now that I’ve established that esbuild is great but ditching NPM not so much, my suggestion for all fresh Phoenix 1.6 projects is to go with esbuild + NPM combo in which esbuild does its crazy fast bundling and NPM is responsible for managing packages in node_modules directory and for configuring & running esbuild. Here’s a nice quote:

“However, using the command-line interface can become unwieldy if you need to pass many options to esbuild. For more sophisticated uses you will likely want to write a build script in JavaScript using esbuild’s JavaScript API.”
— esbuild docs

This kind of setup means that we’re ditching the out-of-the-box convenience of the Elixir esbuild package along with its care not to leave zombie processes etc. But worry not — it only takes 2 files, each a few lines long, to get the same result. And so here they come!

I’m assuming that you’re starting with the result of mix phx.new from Phoenix 1.6. In case of an upgrade, you may first follow the great Phoenix 1.6 upgrade guide provided by Chris McCord.

Start by creating assets/package.json:

{
  "name": "myapp_assets",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "build": "node esbuild.js",
    "deploy": "node esbuild.js --deploy",
    "watch": "node esbuild.js --watch"
  },
  "dependencies": {
    "phoenix": "file:../deps/phoenix",
    "phoenix_html": "file:../deps/phoenix_html",
    "phoenix_live_view": "file:../deps/phoenix_live_view"
  },
  "devDependencies": {
    "esbuild": "^0.12.24"
  }
}

Note that you’re ending up with exactly the same scripts that were provided in Phoenix 1.5 and before, so your existing Dockerfiles and CI pipelines will work without a single change, as opposed to 1.6’s Elixir-embedded esbuild that drops Node from the pipeline but does require invoking a brand new mix assets.deploy task.

Next, create assets/esbuild.js:

const esbuild = require('esbuild')

// Decide which mode to proceed with
let mode = 'build'
process.argv.slice(2).forEach((arg) => {
  if (arg === '--watch') {
    mode = 'watch'
  } else if (arg === '--deploy') {
    mode = 'deploy'
  }
})

// Define esbuild options + extras for watch and deploy
let opts = {
  entryPoints: ['js/app.js'],
  bundle: true,
  logLevel: 'info',
  target: 'es2016',
  outdir: '../priv/static/assets'
}
if (mode === 'watch') {
  opts = {
    watch: true,
    sourcemap: 'inline',
    ...opts
  }
}
if (mode === 'deploy') {
  opts = {
    minify: true,
    ...opts
  }
}

// Start esbuild with previously defined options
// Stop the watcher when STDIN gets closed (no zombies please!)
esbuild.build(opts).then((result) => {
  if (mode === 'watch') {
    process.stdin.pipe(process.stdout)
    process.stdin.on('end', () => { result.stop() })
  }
}).catch((error) => {
  process.exit(1)
})

Finally, run the following commands:

npm --prefix assets install
npm --prefix assets run build

That’s it for the JS side of things. Now, when it comes to Elixir, you first have to change the watcher in config/dev.exs:

config :myapp, MyappWeb.Endpoint,
  watchers: [
    node: ["esbuild.js", "--watch", cd: Path.expand("../assets", __DIR__)]
  ]

While you’re at it, remove the esbuild config from config/config.exs.

Then, visit mix.exs and apply following changes:

• in deps: remove the esbuild dependency
• in aliases: change the assets.deploy alias to invoke cmd npm --prefix assets run deploy instead of esbuild default --minify
• in aliases: change the setup alias to invoke cmd npm --prefix assets install as a last step after ecto.setup

Finish the cleanup by running mix deps.unlock --unused.

That’s it! Here’s the recap of the end result:

• Still using the same crazy-fast esbuild, but with more configuration options for multiple named entry points, plugins etc
• Bundling config that’s equivalent to the one provided by mix phx.new but encapsulated in a single esbuild.js file
• It’s still invoked by mix phx.server in development, still with nice output messages from watcher and still without zombie processes
• You may start adding JS/CSS libs via npm install --save somelib and importing them in assets without relying on wrappers from Hex
• You may enrich your development experience with ESLint, Prettier etc with basically a single additional command
• Once again, package.json is the one place to audit & manage versions of all JS/CSS libs + esbuild itself

Summary

Phoenix 1.6 is a great upgrade with all the bloat of webpack replaced by a cutting-edge esbuild swiftness, not mentioning other amazing changes such as built-in auth generator and Heex templating (seriously, no other lib releases get me pumped up as much as Phoenix’s do 🤩).

OTOH the choice to stay away from NPM in generated projects is a win from the perspective of the Phoenix team, but I personally see NPM as a necessary piece in every modern web app, even if only for managing 2 packages like the OOTB Milligram and topbar.

We’d be one step closer to really ditching Node if there was a native alternative to NPM, Yarn (my personal favourite that fixes many of NPM’s pitfalls) and PNPM. But it seems that there’s no such option.

And since the Phoenix team indeed had to take the burdens of Node off their shoulders, I hope that this piece will help all developerss to refine the stock setup and get the best of both worlds, avoiding the pitfalls of the previous generation of front-enders who had to organize their assets without a version manager (anyone here remembers bower?).