When the Compute Module 4 was released (see my CM4 review here), I asked the Pi Foundation engineers when we might be able to boot off NVMe storage, since it was trivially easy to use with the exposed PCIe x1 lane on the CM4 IO Board.

The initial response in October 2020 was "we'll see". Luckily, after more people started asking about it, beta support was added for direct NVMe boot just a couple weeks ago.

And in a stroke of perfect timing, I also received a hand-made MirkoPC board from Mirek in Poland last week, which among other features includes a full-size M.2 M-key slot, meaning I could test out NVMe boot on it with a Compute Module 4 and a WD_black SN750 500GB SSD I use for testing.

Video version of this post

I also posted a video about booting from NVMe on my YouTube channel; it's embedded below:

Setting up NVMe Boot

The setup process is a little involved right now, since you have to set the correct boot order, update the Pi's bootloader, update the Pi's firmware, but it's not too crazy.

Because I had some trouble with the MirkoPC's built-in microSD card slot, I chose to use an eMMC-powered Compute Module 4, so I flashed Pi OS to it, then set the Pi to USB mass storage mode.

I followed the beta instructions to change the BOOT_ORDER, but I had to modify the sed command on my Mac, since the arguments are different for the non-GNU version of sed shipped with macOS:

sed -i '' 's/\(BOOT_ORDER=.*\)6\(.*\)/\1\26/' boot.conf

(Note the addition of the ''.)

For the firmware update, I used the built-in SD Card Copier utility to copy all data from the eMMC storage over to the NVMe SSD, and made sure to check the "New Partition UUIDs" option when doing so.

After that, I rebooted the Pi, and used lsblk to verify the NVMe was mounted as / and /boot, and then got on to benchmarking.

Benchmarking NVMe vs microSD, eMMC, and USB adapters

The two main things I wanted to test were boot times and app launch/use times, to see what kind of impact the NVMe storage would have.

I already know from my World's tiniest NVMe RAID video that NVMe storage is, on average, 12x faster on the Pi for normal file operations (copies, random read/write), but how much does it affect everyday performance?

Well, boot times are hardly affected by disk IO, it seems; after realizing I had to disable Bluetooth in /boot/config.txt and disable the hciuart service, since my CM4 didn't have a wireless module, I got pretty much the exact same boot time across NVMe, eMMC, and microSD bootups:

But the next test reveals more of the typical speedup you'd encounter on a Pi, day-to-day. I wrote a Node.js benchmark script for Chromium to test how long it takes to launch Chromium, load a webpage, and quit last year, for some other testing.

I ran that same benchmark on each storage device (three times, restarting the Pi between each test, with < 2% standard deviation in the results), and got the following times:

In this test, NVMe storage is 44% faster than microSD, and around 20% faster than eMMC. The bigger surprise was how much better eMMC fares than microSD; the 8-bit configuration on the CM4 (vs. 4-bit on the CM3+ and earlier) makes a big difference, especially for random IO.

But 44% is a pretty significant difference. You will definitely notice the speedup from NVMe boot on the Pi, and that's not even considering the fact that you can get 4 TB (or larger) NVMe SSDs, and they have much better write longevity compared to all but the most expensive microSD cards.

And I know people will ask about it, so I also compared the exact same NVMe drive in a USB 3.0 to NVMe adapter, on the exact same Pi, using the exact same VL805 USB 3.0 chipset, to see what overhead there is when running NVMe storage through USB 3.0 on a Pi 4 model B:

So yes, there is a measurable difference—as I've found across many tests these past few months. Using native NVMe (vs. through USB adapters) is about 10% faster.

Anyways, I thought the MirkoPC would be a great little board to test NVMe boot on, though it works on any CM4 board (look at the list of Compute Module 4 project boards I am maintaining here), as long as you can adapt an M.2 NVMe SSD to the board's PCI Express connection (many boards have M.2 slots built-in!).

Check out the YouTube video for more details about my tests, and stay tuned (maybe subscribe to the blog's RSS!), as I'll continue exploring more storage options for the Pi, and I'll also be posting more about the MirkoPC soon!