453 commits, 44 contributors, five months

Well, it’s the future now, and Meteor 1.8 is the culmination of that plan.

Not only have we eliminated many of the performance problems specific to Meteor 1.7, we also dug deep into the historical internals of Meteor’s compiler plugin system, as far back as version 1.2 (September 2015!) to reap some pretty impressive build time improvements, all without compromising our fundamental design constraints.

Build performance

Delayed legacy builds

Meteor 1.7 introduced a new client bundle called web.browser.legacy in addition to the web.browser (modern) and web.cordova bundles, so that modern evergreen browsers are no longer forced to run code compiled for browsers that stopped receiving software updates years ago.

This development remains a significant competitive advantage for Meteor, since no other framework has successfully copied this functionality, though there is some confusion over what counts as success. In this talk, I predicted that it would be difficult for other frameworks to follow suit, and my claim still stands:

Naturally, this extra bundle increased client build (and rebuild) times. We were comfortable with this performance regression at the time, because it felt like the cost of progress, but also because we figured we could improve build times across the board in a follow-up release, rather than taking drastic measures like disabling the legacy build in development.

However, since developers spend most of their time testing the modern bundle in development, and the legacy bundle mostly provides a safe fallback in production, Meteor 1.8 cleverly postpones building the legacy bundle until just after the development server restarts, so that development can continue as soon as the modern bundle has finished building.

Since the legacy build now happens during a time when the build process would otherwise be completely idle, the impact of the legacy build on development server performance is minimal. Nevertheless, the legacy bundle still gets rebuilt regularly, so any legacy build errors will be surfaced in a timely fashion, and legacy clients can test the new legacy bundle by waiting just a little longer than modern clients. Applications using the autoupdate or hot-code-push packages will reload modern and legacy clients independently, once each new bundle becomes available.

In other words, we have all but erased the technical debt of compiling an additional client bundle, which was one of the biggest reasons for slower build times in Meteor 1.7. And we did it without imposing any additional configuration burden on Meteor developers. 🚫 🏗✨

Lazy compiler plugins

Even if we had not figured out how to delay the legacy build, Meteor 1.8 would still be faster than Meteor 1.7 by a wide margin, because we made a number of other substantial performance improvements, slashing build times across the board for the modern, legacy, Cordova, and server bundles. In fact, erasing the cost of the legacy bundle was really just icing on the cake, compared to what we’re about to describe.

A fantasy we can never indulge

When you’re designing a compiler system, it’s tempting to assume that individual files can be compiled one at a time, because that assumption greatly simplifies incremental recompilation, dependency tracking, and caching. Babel, for example, never needs to look at other JavaScript modules to determine how to compile the current module.

This assumption breaks down for languages like TypeScript, many compile-to-CSS dialects such as LESS and SASS, and indeed any programming language that needs to perform static analysis across multiple files at build time. If one TypeScript module imports an enum from another module, for example, the compiler needs to know not only that the export was an enum, but also whether it was a const enum or not, because that affects whether the enum values can be compiled down to integer constants elsewhere in the program. LESS mixins imported from other files are useless without access to the contents of the file where the mixin was defined.

For these reasons, Meteor’s compiler plugin system was designed from the beginning to provide a complete list of files that might need to be compiled to any compiler plugin that registers an interest in handling a particular file extension such as .js or .ts or .less, rather than invoking the compiler plugin once for each individual file. This was absolutely the right decision, and we have no intention of ever abandoning it.

The cost of sticking to our principles

Though we remain committed to this design, we must also acknowledge that it encourages compiler plugins to compile every file they receive, regardless of whether the file will ever actually be imported, and even if the compiled code appears nowhere in the final build.

This unnecessary compilation accounts for a substantial share of build time in many large Meteor applications, and so we are thrilled to announce that Meteor 1.8 finally introduces a simple mechanism for avoiding compilation of unused files.

If you are not a maintainer of any compiler plugins, you can safely skip the rest of this section, and simply enjoy the benefits of this new feature, automatically!

Here’s how it works. Compiler plugins that call inputFile.addJavaScript or inputFile.addStylesheet may now delay expensive compilation work by passing partial options ({ path, hash }) as the first argument, followed by a callback function as the second argument, which will be called by the build system once it knows the module will be included in the bundle. For example, here's the old implementation of BabelCompiler#processFilesForTarget:

and here’s the new version:

The compiler plugin still processes every possible file, and must specify how each file should be compiled (using the callback function), but any unused files will now be ignored by the build system, and their uncalled callbacks will be discarded after the bundle is built. ☎️ 🚮 👋

If you are a maintainer of a compiler plugin, we strongly recommend migrating to this new API, since the performance benefits for your users are potentially massive. Although this new API is only available in Meteor 1.8, you can use inputFile.supportsLazyCompilation to determine dynamically whether the API is available, so you can support older versions of Meteor without having to maintain multiple versions of your plugin.

Dependency upgrades

Babel 7

When Meteor 1.7 was released, Babel 7 was still in beta (beta 49, specifically). Meteor has kept pace with Babel prereleases very closely, shipping a new version of our babel-compiler package for almost every beta release (all 56 of them) and release candidate.

Though shipping beta software has risks, we were confident we could work around any breaking changes or bugs using our meteor-babel abstraction, thus shielding Meteor developers from those disruptions. Still, it’s a huge relief that Babel 7 has been finalized, and we can go back to normal semantic versioning for future releases.

Because of a specific breaking change in @babel/[email protected] (the removal of the @babel/runtime/helpers/builtin directory), we will not be updating Meteor 1.7.0.x and earlier to the final version of Babel 7. Those versions of Meteor will remain on version 7.0.0-beta.55. In other words, updating to Meteor 1.8 is necessary to get the benefits of non-beta Babel 7, and you will definitely want to install the latest version of @babel/runtime by running the following command:

meteor npm install @babel/runtime@latest

If you have been using custom Babel plugins in a .babelrc file, now might be a good time to make sure they are fully compatible with Babel 7.

Mongo 4

Meteor 1.8 ships with MongoDB 4.0.2, a major upgrade from version 3.6.4 in Meteor 1.7. This means your local development database will now use version 4.0, though the [email protected] driver will happily connect to production databases going all the way back to Mongo 2.6:

The key benefit of MongoDB 4.0 is its support for multi-document transactions, which are now supported by Meteor’s oplog tailing system thanks to this pull request. Of course, if you decide to take advantage of transactions in development, you’ll also want to be using MongoDB 4.0 in production. Your database provider should be able to help with that.

Making the most of this version of MongoDB will be an ongoing process, but we are confident the update should be mostly seamless for apps that were previously using MongoDB 3.x.

Node.js 8.12.0*

* Actually still 8.11.4, for now.

We originally intended to ship Node.js 8.12.0 with Meteor 1.8, but those plans were interrupted by credible reports of high CPU usage in production, apparently due to excessive garbage collection. These reports began while Meteor 1.8 was in the final release candidate phase of testing, so we did not feel comfortable moving forward with the 8.12.0 update. Instead, Meteor 1.8 ships with the previous version, 8.11.4.

If you are eager to start using Node.js 8.12.0 (perhaps because of performance improvements related to Meteor’s use of Fibers), we encourage you to get involved in the Meteor 1.8.1 prerelease process. Just run the command

meteor update --release 1.8.1-beta.n

where the n in beta.n refers to whatever the most recent beta version is in our Release 1.8.1 pull request. The first release that includes Node.js 8.12.0 is also the first beta release, 1.8.1-beta.0. If you deploy an app to Galaxy with this version of Meteor, it will automatically be executed using Node.js 8.12.0. Please report your findings (good, bad, or interesting) in this issue.

We hope to ship Meteor 1.8.1 as soon as we can diagnose and fix these performance problems, and that will happen sooner if you can reproduce the problem, or otherwise help us figure out what’s going on! 🐌 🔜

Features you’ve always wanted

meteor create --react

Meteor has been a great way to use React for almost as long as React has existed. In fact, my own personal interest in Meteor was rekindled while I was working at Facebook on the JavaScript Infrastructure team, which was responsible for maintaining the React open source project.

React is famously unopinionated about where your app’s data comes from, and I thought Meteor could provide a compelling answer, with its advanced real-time data system, so I cooked up an internal demo to show the team. It worked incredibly well, which will come as no surprise to anyone familiar with both Meteor and React. Now, obviously React was not about to start depending on Meteor, or even tailoring itself to Meteor, but the ease of setting up that demo went to show how flexible both technologies are.

Much has changed since then, but Meteor’s commitment to the React developer experience is as strong as ever. That’s why Meteor 1.8 is finally introducing an easy way to create new Meteor apps with React instead of Blaze (though you can certainly use the two side-by-side, if you like):

meteor create --react my-new-react-app

Though relatively simple, this application template reflects the ideas of many contributors, especially @dmihal and @alexsicart, and it will no doubt continue to evolve in future Meteor releases. ⚛️💫

Overruling the constraint solver

If you’ve ever struggled to get Meteor’s constraint solver to untangle complicated package version conflicts, or cursed a no-longer-maintained package for accidentally constraining the version of a package you wanted to upgrade, Meteor 1.8 introduces an incredibly powerful escape valve that you can use to give better guidance to the solver.

Specifically, the .meteor/packages file (where you list the desired versions of all your top-level packages) supports a new syntax for overriding problematic version constraints from packages that you do not control.

If a package version constraint in .meteor/packages ends with a ! character, then any other (non-!) constraints imposed on that package anywhere else in the application will be weakened to allow any version greater than or equal to the constraint, even if the major/minor versions do not match. Note: this syntax only works in .meteor/packages, so there is no risk of abuse by anyone but the application developer.

For example, using both CoffeeScript 2 and practicalmeteor:mocha used to be very difficult because of this api.versionsFrom("1.3") statement, which unfortunately (and unnecessarily!) constrains the coffeescript package to version 1.x.

In Meteor 1.8, if you want to update coffeescript to 2.x, you can simply relax the practicalmeteor:mocha constraint by putting

[email protected]_1! # note the !

in your .meteor/packages file. Again, this means that all other version constraints imposed by other packages on the coffeescript package will lose their power to hold back the coffeescript version, though they will continue to require a minimum version. In other words, the coffeescript version still needs to be at least 1.x, so that practicalmeteor:mocha can count on that minimum. However, practicalmeteor:mocha will no longer constrain the major version of coffeescript to 1.x, so [email protected]_1 will work.

Shout-out to Geoffrey Booth, lead CoffeeScript maintainer, who originally got this conversation started, after much frustration with the exact problem described above.

Intuitively, this makes sense because most accidental version constraints are right about the minimum version they require, but wrong about the maximum version they will accept. And if they’re right about both, well, then there’s no accident.

When you need this syntax—if you need it—we think you’ll find it’s exactly the trick you’ve been missing. 👌🧙‍♀️

What’s next

Node.js 8.12.0, but actually

As discussed above, we will be tracking progress on the Node.js 8.12.0 garbage collection mystery in this issue, and we plan to release Meteor 1.8.1 as soon as those problems are solved.

An officially supported typescript package

Now that Meteor supports lazy compiler plugins, we anticipate two important developments for TypeScript in Meteor:

1. The widely used barbatus:typescript plugin will begin using lazy compilation, which should improve performance dramatically.
2. We will finally be able to share the typescript plugin we’ve been dog-fooding with our Engine app for the past few months.

To stay abreast of these developments, hop on over to this feature request.

More flexible meteor create options, e.g. --apollo

The new meteor create --react command is just the beginning. If you can believe it, we have even more opinions about the best way to set up an Apollo-based Meteor app, an app that uses TypeScript alongside ECMAScript, and many other different templates for new Meteor projects.

One drawback of the current meteor create --argument system is that the templates are tied to the current Meteor release, so it’s tricky to ship updates without releasing a new version of Meteor. We think we can overcome this barrier through a combination of doing more frequent Meteor releases, and fetching the app templates from an evolving Git repository, instead of baking them into the meteor-tool package.

Compiling compiler plugins after installation

Compared to the rest of the JavaScript ecosystem, one of Meteor’s selling points is that package and application code is compiled and bundled when the application is built, using whatever package versions the application developer chooses, instead of relying on past publishing choices of package authors who cannot anticipate the needs of the individual application.

The one exception to this rule is that Meteor compiler plugins are compiled and bundled before they are published to Atmosphere, which means they can end up using outdated versions of certain dependencies, unless the plugin author diligently releases new versions of the plugin. See this issue for an example of such a problem, and how it was fixed.

If you are the maintainer of a Meteor compiler plugin that uses ecmascript, we recommend taking this opportunity to bump the minor version of your plugin, add the latest @babel/runtime to its private npm dependencies, and republish the plugin (and its parent package) using Meteor 1.8. Bumping the minor version is important so that you can continue releasing patch updates for older versions of Meteor.

If you are an application developer who is struggling to use a compiler plugin that has not been updated for Meteor 1.8, you can always take matters into your own hands by cloning the plugin repository into your local packages/ directory, so that it will be rebuilt along with your app.

Rebuilding plugins at application build time is the general direction in which we want to move, though we hope to make that behavior the default, without requiring application developers to make local clones of plugins.

A slimmer version of the autoupdate package for minimal Meteor apps

Thanks to recent work by Christian Klaussner, the autoupdate package will soon cease to depend on the mongo and minimongo packages, which will save 60KB+ of bundle size (minified, before gzip), making it much more tolerable to use autoupdate and/or hot-code-push in an app that was created with the meteor create --minimal command.

Installing Meteor packages from npm?

Since Meteor packages can be cloned into a local packages/ directory instead of being installed from Atmosphere, there’s no fundamental reason why they couldn’t also be installed into a node_modules directory using npm or yarn. To a first approximation, the trick is simply to teach Meteor to scan top-level node_modules packages for package.json files with a meteor section that refers to the relevant package.js configuration file.

Imagine being able to npm install packages that have the ability to run eager initialization code on both the client and the server; can be recompiled differently for modern, legacy, Cordova, and server bundles; and work just like normal npm packages when installed in a non-Meteor application!

We’re still in the early stages of implementing this idea, but we find it appealing because it points to a future in which Meteor applications are built on top of enhanced npm packages that benefit (and benefit from) the wider npm ecosystem, without giving up their Meteoric super-powers.

How to install from scratch

If this is your first time using Meteor, or you would prefer to reinstall Meteor from scratch to reclaim some disk space, the following commands will download Meteor 1.8 and install a system-wide meteor command.

Mac and Linux

curl https://install.meteor.com/ | sh

Windows

First install the Chocolatey package manager, then run this command using an Administrator command prompt:

choco install meteor

How to upgrade

Mac, Linux, and Windows

Though we try very hard to make each Meteor release as backwards-compatible as we reasonably can, even the smallest differences in behavior (intended or unintended) could prove significant for existing applications. To document noteworthy changes, we have begun highlighting Breaking changes and Migration stepsin our History.md release notes.

In Meteor 1.8, the only absolutely essential migration step is to install the latest version of the @babel/runtime npm package, since a number of breaking changes were introduced during the long Babel 7 beta period:

meteor npm install @babel/runtime@latest

As usual, running meteor update in an application directory will update the application to Meteor 1.8. Running meteor update outside of an application directory will download Meteor 1.8 for later use, and meteor create new-app will create a new Meteor 1.8 application.

Rolling back

In the unlikely event that the update leaves your application in a bad state, and you don’t feel like debugging it right away, please make sure your application’s .meteor directory is committed to your version control system (e.g. Git, Mercurial, etc.) before the update, so that it’s easy to revert the changes if you encounter problems.

In summary

This has been a pretty long release announcement already, so we’ll keep this section brief.

With 453 commits over five months of development, Meteor 1.8 preserves, extends, and patches the groundbreaking features of Meteor 1.7, with vastly better build performance. If you’re already using 1.7, you should definitely run meteor update at your earliest convenience. If you’re still running Meteor 1.6 because 1.7 was too slow, or you had trouble updating for any reason, feel free to skip directly to Meteor 1.8. If you’re coming from an even older version of Meteor, see the migration guide.

As always, please have a look at the full release notes for details that may be particularly relevant to your applications.

If you get stuck, please feel free to ask questions in the comments on this post, or file a GitHub issue, or check the forums to see if other Meteor developers are facing the same challenges.

Now go forth and meteor update! 💫