It’s one thing to run node server/index.js, it’s quite another to ensure that it runs 24/7, restarts when the process crashes, or when the server reboots or when armageddon happens.

Running software in production is complicated, so let’s talk about the two methods we use at Trainline to keep our Node apps running.

• systemd: This is a service manager comes bundled with all of the major Linux distributions. A service managed by systemd is defined with a unit file that is placed into a well known folder (The file normally containing ExecStart=node server/index.js as one of the lines).
• PM2: This is a service manager that runs on node.js and takes advantage of the cluster module native to node.js. To start your node.js application as a service, you would run pm2 start server/index.js.

systemd

Systemd is popular for running node.js applications because it’s built-in to the operating system — you can trust it to load your service after it loads your device drivers!

All you need is a unit file that looks something like this

[Service]
ExecStart=/usr/bin/node server.js
Restart=always
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=MyBeeGeesApp
User=nobody
Group=nobody
EnvironmentFile=/etc/default/MyBeeGeesApp.env
WorkingDirectory=/opt/MyBeeGeesApp[Install]
WantedBy=multi-user.target

…saved in lib/systemd/system/

There are npm packages available (strong-service-systemd for example) that will help generate this file for you.

Once in place, it’s just a matter of running the following commands:

systemctl daemon-reload
systemctl enable MyBeeGeesApp
systemctl start MyBeeGeesApp

You now have a single instance of server/index.js running on your machine.

Not quite!

As you may know, node runs as a single threaded event loop, unless you use the native cluster module and fork your application. This means that if you were to run your node service on a multi-core system, your app wouldn’t be able to take advantage of all the cores available. Your service isn’t stuck using just one core, but it would only use the equivalent of a single core in a multi-core environment.

Is this a big problem? Not necessarily. Some of our node.js services run on a couple of small two-core machines like an AWS t2.medium instance. If we need to support greater throughput, we tend to scale-out by increasing the number of instances that serve the application.

However, it’s possible that you have other considerations preventing you from deploying lots of small instances — infrastructure costs like licensing for third party monitoring systems, for example, are calculated per-host rather than per-compute-unit/per-core.

We can solve this is by duplicating the service file above and then starting each app one by one:

systemctl start MyBeeGeesApp
systemctl start MyBeeGeesApp-2
systemctl start MyBeeGeesApp-3

Each invocation of your service would need it’s own port, possibly causing issues for systems that rely on designated ports for services. A service might have 4 designated ports for 4 cores, other may have 8 ports for 8 cores.

This isn’t necessarily a problem, but it’s common to see scenarios which make this type of configuration more complicated to configure — for example in a blue/green deployment, instead of switching to proxy from one port number to another, you would be switching between two sets of ports in your load balancer.

You can overcome this problem by placing an internal load balancer (like nginx) within the instance so that the application appears to listen on to one port — but you introduce additional pieces of complexity for each thing you introduce.

Alternatively, we can switch from our systemd approach and use PM2.

While you can use the native cluster module to fork your application to help scale node to multiple threads, the cluster module itself is a little bare bones. The module just forks your app, leaving it is up to you to manage the master and worker processes (e.g. spawn a worker if a worker dies, kill all workers if the master dies). PM2 on the other hand, is a wrapper around the native cluster module which takes care of these things for you.

Getting started with PM2 is relatively easy.
If you want to run a server with as many node processes as CPU cores, you run:

pm2 start -i 0 server/index.js

You’ll notice that the command exits immediately and your service can be found by typing pm2 list — showing you a list like this:

┌──────────┬────┬─────────┬───────┬────────┬─────┬─────────┐
│ App name │ id │ mode    │ pid   │ status │ cpu │ mem     │
├──────────┼────┼─────────┼───────┼────────┼─────┼─────────┤
│ index    │ 1  │ cluster │ 10008 │ online │ 82% │ 54.0 MB │
│ index    │ 2  │ cluster │ 10009 │ online │ 78% │ 53.6 MB │
│ index    │ 3  │ cluster │ 10020 │ online │ 74% │ 48.7 MB │
│ index    │ 4  │ cluster │ 10039 │ online │ 70% │ 43.8 MB │
│ index    │ 5  │ cluster │ 10056 │ online │ 64% │ 38.6 MB │
│ index    │ 6  │ cluster │ 10073 │ online │ 57% │ 36.3 MB │
│ index    │ 7  │ cluster │ 10088 │ online │ 52% │ 33.1 MB │
│ index    │ 8  │ cluster │ 10103 │ online │ 39% │ 28.2 MB │
└──────────┴────┴─────────┴───────┴────────┴─────┴─────────┘

If you were to run the command on an 8 core machine. The first time you run the command, a ‘God Daemon’ is started in the background, with subsequent pm2commands interact with this process.

Now, were your app to crash pm2 will restart it, but if you rebooted the machine, neither pm2 nor your node application would start. To make sure your app restarts upon reboot, you need to save the pm2 configuration using pm2 save, then as root, run pm2 startup.

PM2 Gotcha #1: Multiple God Daemons

One of the biggest gotchas when moving from systemd to pm2 is dealing with the multiple instances of service managers running. Systemd is almost always the first process that is run by the kernel, i.e. PID 1. This means there’s only one instance of systemd runnning at any point, so while any user with sufficient permissions can run systemctl start MyBeeGeesApp, rest assured there’s only one instance of systemd that is listening for that command and executing the necessary service definition.

This is not quite the same with pm2. As I’ve previously mentioned, the moment you run the first pm2 command whether pm2 start ... or pm2 list, a pm2 God Daemon is spun up in the background.

This process runs per user so if you happened to run pm2 start index.js as root (which you shouldn’t) and later, as someuser, ran pm2 start index.js, you’ll be running your app twice, on two separate God Daemons — one per user.

As a result, you’ll have the option to run an app within pm2 as someuser from the God Daemon that belongs to the root user but for obvious reasons, not the other way around. As a result, one of the first steps in implementing PM2 is to decide on how you want your God Daemon to run.

At trainline, we’ve gone for a someuser God Daemon approach. So if your deployment scripts normally run as root, you would have to do something like this:

sudo -u someuser pm2 start -i 0 index.js

But before you run off and develop your deployment scripts with just the line below, read about the second Gotcha!

PM2 Gotcha #2: PM2_HOME

PM2 puts all of its state into files within a folder. This folder is critical as it is where the PID files, logs as well as the list of apps to start up are kept.

Unfortunately, PM2 tries to be a little bit clever in determining where to find this folder. The exact algorithm is found here. However, in order to guarantee that your God Daemon always knows where to look for it’s PM2_HOME folder, you should pass in the location of the .pm2 folder as the PM2_HOME environment variable.

This makes your pm2 startup a little more explicit and verbose:

pm2 startup ubuntu -u someuser --hp /home/someuser

Notice that the --hp ‘home path’ flag above specifies the parent of the PM2_HOME folder.

This means that if your deployment scripts normally ran as root, you’d be expected to run something like:

sudo -u someuser PM2_HOME=/home/someuser/.pm2 pm2 start -i 0 index.js
sudo -u someuser PM2_HOME=/home/someuser/.pm2 pm2 save
pm2 startup ubuntu -u someuser --hp /home/someuser

Use PM2!

PM2 provides us with a reliable, hardware-utilising approach to scaling out our Node applications — allowing us to make the best use of our AWS instances with the least hassle.

We use it for our Node apps, and think you should too.