FreeBSD Developer Workstation Setup | Klara Inc.

FreeBSD is an Open Source Operating System and, as such, the project is always eager to bring in more contributors and developers. From the outside it can be hard to figure out what sort of contributions might be accepted, and it is a common refrain that many people don’t know where and how to start.

Once you have picked your favorite bug (or least favored) or missing feature, it can still be difficult to figure out how to set up a development environment. FreeBSD as an entire Operating System is not very opinionated in how you go about development and it isn’t common to find developers talking about their set ups and process.

This article will provide some pointers into how some FreeBSD committers work on the Operating System and ports collection. We will discuss the hardware configurations that are used day to day to implement, build and test the operating system.

While it is now possible to build FreeBSD on other Operating Systems, using FreeBSD as your development host is still the easiest and best supported way to do so. The development process for everyone breaks down into a couple of stages no matter which part of the OS they are working on:

1. Preparing changes (working on the code, ports or docs)
2. Building and testing
3. Submitting patches

The first and third stage can happen on any platform where you can get a copy of the FreeBSD source tree and the tools required to work with the project. Building can be done on FreeBSD and other platforms, but testing requires running a FreeBSD system to see how things are working in the real environment.

Preparing Changes

You are free to use your favorite text editor to do FreeBSD development; it is common that developers use text editors such as vim and emacs. It is entirely possible to use a modern full featured IDE such as visual studio code to work on the OS. The actual mechanics of modifying the OS and adding features is agnostic to which text editor you use.

The FreeBSD source tree is a large and daunting place when you first explore it. The layout of the tree is described in the developers handbook. There are some common tools developers use to help navigate their way around: cscope and ctagsoffer source code search and integrate well with text editors. cscope, vim and tmux make for a solid development environment when connected to a remote machine. tmux or screen enable multiple development windows and provide the option to split up a terminal into multiple pieces and run concurrent sessions.

FreeBSD builds, test suites, and stress tests can take a long time to run. When you have to wait on processes to complete, it is typical for your mind to wander and you find yourself refreshing twitter or reading the news. A tool like ntfy coupled with a message bus notification service like pushover integrates really well into a command line based development process. Pushover can deliver notifications to your phone so you are alerted when long running tasks or processes have completed. It is a great way to kick something off and leave it in the background without having to worry that you will forget to come back to it as soon as it is done.

Building and Testing

The FreeBSD build process is documented in build(7). In summary, to build the FreeBSD Operating System you need to go through two main stages. First you need to build the world – this is all the software and components you run that make up the OS. Second, the kernel is built – the kernel handles multitasking and separates software from the fine details of the hardware it has to run on. The build world process compiles all the utilities in the system (think sh, ls, cat, etc…), but it first builds all the tools required to build the operating system itself.

The buildworld stage makes sure that you have an up to date build tool chain. If you do not, then it will create the tools it requires to run the build process. Not being up to date can lead to weird and difficult to understand errors later when you build the kernel. If you are doing development for a different processor architecture than your host computer – maybe you are cross compiling from a very fast aarch64 system and targeting a slow i386 system – then buildworld will handle creating the cross compilation environment for you if you pass the right flags.

buildworld will check your system to see if everything is recent enough and if it is it will skip building the LLVM compiler. This is a great optimization, as building LLVM takes a large portion of the FreeBSD build time, but it being skipped can sometimes make you think that builds are magically running faster.

The second stage in building the operating system is to compile the kernel via the buildkernel command. If you are doing kernel development this will be a frequent command to run.

From the FreeBSD source tree, the following command will perform both build stages and leave you with a built, but not installed operating system.

$ make buildworld buildkernel 

Using make options to speed up builds

Running the buildworld buildkernel process can take a lot of time depending on the hardware it is run on. The process might take as little as an hour or as much as several days on something as tiny and slow SBC like a raspberry pi.

Even an hour is impractical for a development environment so there are options to the build process to speed it up by skipping the most expensive to run steps. Normally you need to have run the full build process once to have the correct compiler tool chain created.

The buildworld and the buildkernel processes offer ‘fast’ environments. These steps use make to its fullest and skip the normal ‘clean’ takes of removing existing object files before starting the build.

$ make buildkernel -DKERNFAST
$ make buildworld -DWORLDFAST

make defaults to using a single job for the build process. The FreeBSD build process lends itself really well to being parallelized across many cores. A good rule of thumb when specifying the number of jobs to make is to use the number of cores or cpus that the system thinks it has. The number of logical processors is available in the hw.ncpu sysctl node. We can specify the number of jobs for make to use via this sysctl like so:

$ make -j `sysctl -n hw.ncpu` buildworld buildkernel

Putting more cores to work reduces your time to build significantly. On the same machine using 48 cores vs 24 cores we see a big improvement, with the buildworld buildkernel processed being reduced from 42 minutes to 29 minutes.

FreeBSD has an incremental build mode called Metamode. Metamode uses bmake’s meta mode data and the filemon(4) file system device to track changes and substantially reduces build times by only compiling objects that have changed between builds.

To use Metamode you need to follow these steps to set up the Metamode environment:

1.	Add WITH_META_MODE=YES to /etc/src-env.conf
2.	load the filemon kernel model: # kldload filemon
3.	Run # make cleanworld once. If objects don’t have .meta files then META_MODE won’t do the right thing on an incremental build

Then you can follow the normal makeworld process.

Submitting Changes

Once you have written and tested a change it needs to be submitted to the project. Here, FreeBSD committers can commit to the relevant git branch and push, but many changes need to go through review and this process is the same for committers and non-committers.

FreeBSD uses phabricator (phab) for code review and changes can be submitted to phab directly from the development source tree by using the arcanist command line tool.

$ export EFIDIR=$HOME/efistage
$ git add awesomefile.c
$ git commit -m "A very descriptive and helpful commit message"
$ arc diff HEAD^

Creating a review is the best way to make sure that your patches don’t get lost on a mailing list.

Workstation Hardware

The build and test stages of preparing a FreeBSD change are where we get to have fun talking about machines and their performance. There really is no set way to do FreeBSD development and you are free to use the computers you have.

When doing development, the stability of the machine that you type on is very important. When you are getting started, it is easy to decide that you will build a FreeBSD machine and use that for everything FreeBSD related. Kernel changes require a lot of reboots and all development leads to crashing bugs. It is much easier to have a one stable host to write code on, and let testing happen on a second machine. This can be either a dedicated box or a virtual machine running on or next to the development box.

When working on the FreeBSD Operating System there are three roles for machines:

1. Development (writing code and submitting upstream)
2. Building
3. Testing

While these roles can be fulfilled by a single machine, you will have a better experience if testing happens on a dedicated host (either physical or virtual). If the build machine can be far away that enables it to be much more powerful (think hotter and louder).

A common workstation set up is a laptop with FreeBSD guest virtual machines for development and testing. This set up has the benefit that you can run the platform that your employer requires you to or the one with the best support for your hardware and still have a FreeBSD host available for testing and development on the go.

Using a virtual machine on a laptop has pretty severe performance downsides when it comes to building the operating system and ports. If most of your development cycles is building new kernels, base utilities, documentation or small ports then building on the laptop is probably okay. When you come to updating the system and running buildworld you might find yourself hurriedly looking for a socket to plug your laptop into as your battery runs down. When building on your local machine, whether in a virtual machine or on the host directly, you will be using a lot of computation for FreeBSD and it will limit your ability to do other things at the same time.

A much better solution is to keep the really hot noisy build process away from the main computer where typing happens and host it somewhere else. If it is available to you then a racked machine in a data center or closet far away is great. This can be a machine you own or a virtual or dedicated host in the cloud. Dedicated machines from cloud providers can be very affordable for reasonably fast hardware even if it is a couple of years old and you are limited by the processor topologies.

Finally, there is the option of using virtual machines in the cloud and spinning up a fast VM for the build process and letting it turn off when you are done. This form of development is more work to set up, but offers a chance to use very fast hosts for short periods of time. If you can keep your tooling working to do this then you may be able to greatly speed up builds and save money if you are only doing large builds occasionally.

Optimizing Machines for Building

The biggest improvement in build time comes from increasing the number of cores available on the machine. To show this we can compare build times for 3 systems with different processors using the following process:

# chown user:user /usr/src /usr/obj
$ cd /usr
$ git clone https://git.freebsd.org/src.git
$ cd /usr/src
$ ntfy done make -j `sysctl -n hw.ncpu` -DWITHOUT_SYSTEM_COMPILER buildworld buildkernel

With these 3 systems we run a build command that should ignore the system compiler and build LLVM as many times as needed in the worst case. The table below shows the processor, number of cores, amount of memory in hosts and shows the total time to run the buildworld buildkernel process. To show the effect of different values of jobs the value to make -jon the biggest machine has been reduced.

Hostcpu yearcpucoresjobsmemorybuildworld+buildkernel time freeBSD version (host)i3 NucQ4 2019Intel Core i3-10110U @ 2.10GHz4464GB228 minutes14-CURRENTAMD DesktopQ1 2015Opteron 63803232128GB58 minutes14-CURRENTXeon dev box in data centerQ2 2016Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz481128GB619 minutes12.2-RELEASEXeon dev box in data centerQ2 2016Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz4824128GB42 minutes12.2-RELEASEXeon dev box in data centerQ2 2016Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz4848128GB29 minutes12.2-RELEASE

From this table you should get a good idea of how generations of processor and number of cores affect the build process. Increasing the number of cores does not offer a linear increase in build time improvement.

Conclusion

FreeBSD is great in that it lets you do development the way that you want to, even though, when you are starting out, this level of choice can make it difficult to figure out what you should do. The FreeBSD build system offers a lot of power, but it is a complex machine and it takes study to figure out how to use all of its power. If you want to start FreeBSD development hopefully this article has offered some pointers to software and suggested a hardware configuration (throw all the cores you can at it) that will help you get the most out of your hacking time.

Like this article? Share it!