Differential modern/legacy browser bundling, selective compilation of node_modules, and more

The future is already here — it’s just not very evenly distributed.
– William Gibson (c. 1993)

If you’ve spent any time developing web applications with JavaScript in recent years, you’ve been promised a future that never seems to arrive.

More than 80% of Internet users worldwide have access to an “evergreen” web browser that natively supports almost all the latest ECMAScript features and keeps itself updated automatically, which means new language features become available almost as soon as they are standardized. Take a moment to appreciate this new reality, since it has profound consequences for web development. Now that so many ECMAScript features are natively supported in the overwhelming majority of web browsers (and Node), it is increasingly a shame that you can’t just execute modern JavaScript directly, without first compiling it to JavaScript of yesteryear.

If you’ve ever dreamed of using native async functions in modern browsers, but the thought of abandoning support for older browsers made you (rightly!) uncomfortable, Meteor 1.7 has exactly the solution you’ve been looking for.

Going evergreen without leaving nevergreen users behind

The future we envisioned when we first began compiling code with Babel is finally here, or at least almost finally here, and yet most web frameworks and applications still compile a single client-side JavaScript bundle that must work simultaneously in the oldest and the newest browsers the application developer cares to support, holding the vast majority of users hostage to the needs of ancient Internet Explorer and Safari versions.

Though the market share of legacy browsers is shrinking, it is not yet anywhere near zero, and those users who cannot or will not update have already resisted years of recommendations, scolding, and pleading. Like it or not, these users are equal members of the World Wide Web, and we web developers must accommodate their needs. However, we do not have to let legacy limitations dictate how our websites work in modern browsers.

Delivering both modern and legacy assets poses a daunting challenge: not only must you build multiple JavaScript and CSS asset bundles for different browsers—with different dependency graphs and compilation rules and webpack configurations—but your server must also be able to detect the capabilities of each visiting client, so that it can deliver appropriate assets at runtime. Testing a matrix of different browsers and application versions gets cumbersome quickly, so it’s no surprise that responsible web developers would rather ship a single, well-tested set of static assets, and forget about taking advantage of modern JavaScript features until legacy browsers have disappeared completely.

If you resign yourself to this gloomy way of thinking, and you run the same code in Chrome 68 as you do Internet Explorer 8, not only will the latest technologies remain always just beyond your reach, but you will miss out on some basic quality-of-life improvements that are nearly universally supported, such as const and let declarations that raise warnings (unlike var) when you use a variable before it has been initialized. You’ll also be shipping many more kilobytes of code to modern browsers than they need, slowing down initial page load times, which has well documented negative effects on user behavior.

To our knowledge, Meteor 1.7 is the first full-stack JavaScript application framework to offer a solution for the problem of differential compilation and asset delivery, so you may not have realized any of this was feasible until reading this blog post. If you did have an inkling that such a system was possible, or you think you have a better solution than the one we will describe below, we welcome your feedback! We fully anticipate that other frameworks will take their own approach to solving these problems, and we’re eager to see how they do it.

Meteor 1.7: The best of both bundles

With Meteor 1.7, the awkward balancing act of building a single bundle for both the newest and the oldest web browsers is no longer necessary. Meteor now automatically builds two sets of client-side assets, one tailored to the capabilities of modern browsers, and the other designed to work equally well in all supported browsers, so that legacy browsers can continue working exactly as they did before. This “legacy” bundle is equivalent to what Meteor 1.5 and 1.6 delivered to every browser, so the crucial difference in Meteor 1.7 is simply that modern browsers will begin receiving code that is much closer to what you originally wrote.

Best of all, the entire Meteor community relies on the same system, so any bugs or differences in behavior between the bundles can be identified and fixed quickly, and everyone will benefit from those refinements.

In this system, a “modern” browser can be loosely defined as one with full native support for async functions, which includes more than 80% of the world market, and 85% of the US market (source). This standard may seem hopelessly optimistic, since async/await was only recently finalized in ECMAScript 2017, but the statistics clearly justify it. And just in case you’re wondering what that percentage looks like for less ambitious ECMAScript features, a mere 87% of browsers support native class syntax. The lesson here: you cannot aim low enough to make Internet Explorer happy, so you might as well aim high.

The precise boundary between modern and legacy browsers is designed to be tuned over time, not only by the Meteor framework itself but also by each individual Meteor application. For example, here’s how Meteor currently guarantees that every modern browser supports native ECMAScript class syntax, so the babel-compiler package can avoid compiling class syntax when targeting the modern bundle:

import {
  setMinimumBrowserVersions,
} from "meteor/modern-browsers";

setMinimumBrowserVersions({
  chrome: 49,
  firefox: 45,
  edge: 12,
  ie: Infinity,
  mobile_safari: [9, 2],
  opera: 36,
  safari: 9,
  electron: 1,
}, "minimum versions for ECMAScript 2015 classes");

The minimum modern version for each browser is simply the maximum of all versions passed to setMinimumBrowserVersions for that browser. The Meteor development server decides which assets to deliver to each client based on the User-Agent string of the HTTP request. In production, different bundles are named with unique hashes, which eliminates the possibility of cache collisions, though Meteor also sets the Vary: User-Agent HTTP response header to let well-behaved clients know they should cache modern and legacy resources separately.

Managing modern/legacy code in Meteor packages and applications

Although you reap many of the benefits of the modern/legacy system just by updating to Meteor 1.7, you can also create your own Meteor packages and application modules that take advantage of differential bundling.

If you’ve written a Meteor package before, the following package.js code should be familiar:

Package.onUse(api => {
  api.mainModule("client.js", "client");
  api.mainModule("server.js", "server");
});

This code still works in Meteor 1.7, though it means the client.js module will be loaded by both the modern and the legacy bundle. If you need to load different code in the different client bundles, you can split the client.js module into a modern.js and a legacy.js module, and then add them to the web.browser and legacy targets, respectively:

Package.onUse(api => {
  api.mainModule("modern.js", "web.browser");
  api.mainModule("legacy.js", "legacy");
  api.mainModule("server.js", "server");
});

The same targeting rules also apply to api.addFiles and api.addAssets.

🆕 In Meteor 1.7 applications, it’s now possible to specify custom entry point modules in the "meteor" section of your package.json file:

{
  "name": "your-app",
  "dependencies": {...},
  "meteor": {
    "mainModule": {
      "client": "client/main.js",
      "server": "server/main.js"
    }
  }
}

This meteor.mainModule configuration allows you to override Meteor’s default rules for eagerly loading modules. Whereas previously you would have to put modules in an imports/ directory in order to prevent them from being eagerly evaluated, when you use meteor.mainModule to specify entry points, all other modules besides the entry points will be loaded lazily. In other words, you no longer need an imports/ directory, and Meteor’s module loading behavior will work more like non-Meteor Node projects.

The same technique we used with api.mainModule in package.js to specify different client entry points for modern and legacy bundles also works for meteor.mainModule:

{
  "name": "your-app",
  "dependencies": {...},
  "meteor": {
    "mainModule": {
      "web.browser": "client/modern.js",
      "legacy": "client/legacy.js",
      "server": "server/main.js"
    }
  }
}

If you decide to use a modern JavaScript feature that is not supported natively by all browsers that support async/await, such as Service Workers, please remember to call setMinimumBrowserVersions to enforce your personal assumptions about which browser versions should be considered modern:

import {
  setMinimumBrowserVersions,
} from "meteor/modern-browsers";setMinimumBrowserVersions({
  chrome: 45,
  firefox: 44,
  edge: 17,
  ie: Infinity,
  mobileSafari: [10, 3],
  opera: 32,
  safari: [11, 1],
  electron: [0, 36],
}, "service workers");

Although these new minimum versions may prevent some recent versions of Safari from receiving the modern bundle, they will ensure that every modern browser has a native implementation of Service Workers. Useful resources for determining appropriate minimum versions include caniuse.com, Mozilla’s MDN documentation, and the electron-to-chromium npm package.

If you include a particular module in both the modern and the legacy bundle, but it contains some code that should run in only one of these bundles, you can dynamically detect which bundle is active by referring to the Meteor.isModern boolean flag:

if (Meteor.isModern) {
  // Do something that's only safe in modern browsers
} else {
  // Do something that's safe in all browsers
}

However, you should avoid Meteor.isModern if possible, since modern syntax can't be guarded with conditional statements, and both code paths will appear in both bundles, which defeats the purpose of differential bundling.

Impact on bundle sizes

Because the modern bundle is compiled by the babel-compiler package with far fewer Babel plugins than the legacy bundle, and needs far fewer polyfills from the ecmascript-runtime-client package, the modern bundle should benefit from improved performance of native JavaScript features, and will also be considerably smaller than the legacy bundle.

🆕 Beginning with Meteor 1.7, when you create a new Meteor application, you can pass the --minimal option to meteor create to produce an application that uses as few Meteor packages as possible, so the modern client bundle will be as small as possible:

If you use the bundle-visualizer package (introduced with Meteor 1.5) with this minimal application, you will see both bundles represented in the same sunburst chart. Notice that the modern bundle is approximately 1/3 the size of the legacy bundle, or about 16KB after gzip. Smooshing both bundles into the same sunburst may not be the best way to display this data, but it certainly drives home how much smaller the modern bundle tends to be:

To get a better sense for the breakdown of the bundles, visit https://meteor-night-bundle-visualization.meteorapp.com/ and hover your cursor over the various segments.

Please note that your immediate savings may not be quite this dramatic. For example, when we updated https://engine.apollographql.com to Meteor 1.7, we saw only an 8% reduction in bundle size, since much of the code used by that application comes from node_modules, which Meteor does not (re)compile by default.

Selective compilation of node_modules

Speaking of compiling node_modules, one of the fundamental reasons why differential compilation and bundling is so difficult in the npm ecosystem is that npm packages are precompiled before they are published to npm, so consumers of npm packages typically have to settle for whatever compilation strategy the package author decided to use. Even if the package author publishes multiple builds of their package, the compilation strategy might not be exactly what you want.

More and more npm package authors are publishing their source code along with ready-to-use, precompiled code, and that’s great news for application developers who want to take matters into their own hands. However, a blanket policy of recompiling every package in node_modules is dangerous because it’s not uncommon for a node_modules directory to contain hundreds of megabytes of data. Instead, most bundling tools require the application developer to configure which npm packages should be recompiled. The ideal configuration API for this task is an extremely contentious unsolved problem in the JS community. Meteor 1.7 gamely joins this ongoing conversation with an answer of its own.

In Meteor 1.7, if you want to control how an npm package is compiled, or if you just want Meteor to compile the package for modern/legacy browsers and Node, here’s what you should do:

• Clone the package repository into your application’s imports directory (that is, if you’re still using an imports directory)
• Make any modifications necessary to compile the package to your liking, such as adding/modifying a .babelrc file or altering the "main" field of package.json to point to src/index.js instead of lib/index.js
• Use meteor npm install imports/the-package to link the package into node_modules

For ease of updating the package in the future, we recommend using a Git submodule to manage the cloned package repository:

# Clone the package as a git submodule
git submodule add \
[email protected]:visionmedia/superagent.git \
    imports/superagent# Tweak imports/superagent, and commit any changes locally# Creates a symbolic link at node_modules/superagent
meteor npm install imports/superagent

This works by exposing the package source code as part of your Meteor application, outside node_modules, so that Meteor’s compiler plugins can process the code. Often no modifications will be necessary, since the recompilation performed by Meteor takes care of everything for you. If you really need complete control, though, you’ve got it. 👩‍🔬🔬💪

How to install Meteor 1.7 from scratch

If this is your first time using Meteor, or you would prefer to reinstall Meteor from scratch to reclaim some disk space, the following commands will download Meteor 1.7 and install a system-wide meteor command.

Mac and Linux

curl https://install.meteor.com/ | sh

Windows

First install the Chocolatey package manager, then run this command using an Administrator command prompt:

choco install meteor

How to upgrade to Meteor 1.7

Mac, Linux, and Windows

As usual, running meteor update in an application directory will update the application to Meteor 1.7. Running meteor update outside of an application directory will download Meteor 1.7 for later use, and meteor create new-app will create a new Meteor 1.7 application.

Rolling back

In the unlikely event that the update leaves your application in a bad state, and you don’t feel like debugging it right away, please make sure your application’s .meteor directory is committed to your version control system (e.g. Git, Mercurial, etc.) before the update, so that it’s easy to revert the changes if you encounter problems.

Release notes

As usual, the full release notes for Meteor 1.7 (and now Meteor 1.7.0.1) can be found in our History.md document.

In addition to running meteor update, you will almost certainly want to install the latest versions of the @babel/runtime and meteor-node-stubs npm packages:

meteor npm install @babel/runtime@latest
meteor npm install meteor-node-stubs@latest

If you’re using Mongo, note that Meteor 1.7 now ships with Mongo 3.6.4, which may require migrating your Mongo 3.2 development database. In general, if you’re comfortable just deleting and recreating your development database, you can use meteor reset to avoid migration headaches.

Please feel free to ask questions in the comments below, or consult the Meteor forums, or open an issue on GitHub. We’re here to help!

Further reading

If you prefer a slideshow to a blog post, check out this talk I (Ben Newman) gave at Heavybit Industries on May 30th:

We will update this blog post once the video from this talk becomes available, though that may take another week or so.

In summary

We think this modern/legacy bundling system is one of the most powerful features we've added to the Meteor framework since we first introduced the ecmascript package in Meteor 1.2 in September 2015, and we wish other frameworks and bundling tools the best of luck finding their own ways to deliver different assets to modern and legacy browsers, without burdening their developers with the tremendous technical debt that would come from building a system like this from scratch. In the spirit of community and collaboration, we genuinely hope our implementation provides some essential clues to developers who aren’t using Meteor.

Based on recent conversations with Yehuda Katz, we anticipate Ember will be the next framework to provide this kind of functionality. As a highly opinionated full-stack web framework, Ember shares many of Meteor’s super-powers, and we fully expect them to teach us a lesson or two about the right way to do differential bundling.

In the meantime, if you’ve ever been tempted to give Meteor a try, there’s never been a better time to get started. 💫