While QEMU is still the leading choice for an open source VMM (Virtual Machine Manager), a rust alternative could be great for security and we wanted to give it a shot.

We are now able to virtualize GUI apps and OS in linux by modifying crosvm, a fantastic rust based open-source VMM, and wanted to share how we did it.

Disclaimer: This is a work in progress and only meant to a proof of concept. So please bear in mind that it is buggy in more than a few places and endless optimizations away from being performant and lightweight.

It is very unlikely to break your system, as the demo is entirely containerized, but it is possible.

If you want to run the demo, ensure you have docker and git installed.

1. Clone the repository:

$ git clone https://github.com/Openw3b/demo-openvmm-1 && cd demo-openvmm-1

2. If you want audio in the demo, modify pulseaudio server config on the host to accept TCP connections from the VM and restart it using `systemctl restart pulseaudio`

load-module module-native-protocol-tcp auth-ip-acl=172.17.0.1/24

/etc/pulse/default.pa (Authenticate IP of your docker0 interface over TCP)

3. Run Firefox

$ ./scripts/quick_start firefox

It might take a while to build, but you should see a firefox window once done

• Resizing the window on host resizes the Firefox window running inside the guest

4. RunVLC

$ ./scripts/quick_start vlc ~/Downloads/bunny.mp4

• Opening any media file on the host opens it in VLC inside a VM
• Audio playback from VM works

5. Run Ubuntu Desktop

$ ./scripts/quick_start ubuntu

• Clipboard sharing works too
• Resizing is yet to be added for OS

How it works?

Before going into the details of how everything works in this PoC, lets look at an example of how to start a VM in crosvm

crosvm run -c 8 -m 4096 \				# vCPU, Memory
  --disable-sandbox \				# Sandboxing ToDo 
  --gpu backend=2d,height=1080,width=1920 \ 	# Display
  --tap-name tap_appvm \				# Networking
  --rwroot fs.qcow2 \				# Filesystem
  --display-window-keyboard \			# Keyboard input
  --socket vm.sock \				# Control socket
  --vhost-net \					# Enable Vhost networking
  --shared-dir shared:shared:type=fs \		# Shared folder (virtiofs)
  bzImage					# Kernel

Example for running crosvm

Its looks fairly straightforward, but getting it to work under linux wasn't. This is so because crosvm was built initially for use in chromium os. But thanks spectrum os mailing list, crosvm's documentation and past experience from making this PoC using QEMU, we were able to get it to work.

We start crosvm programmatically by importing it as module in our rust wrapper (open-vmm). We also built a guest agent in rust that communicates over vsock to support window resizing and clipboard support.

Now let's look at how we got the individual components working under linux.

1. Display

We used virtio-gpu device on the host with Virtual display to get a display output from the VM. Running crosvm with "--gpu" flag opens an XWindow which displays the GPU buffer shared with the VM.

Apps run in an ubuntu VM running i3wm under Xorg with configuration tweaked to run borderless for now. OS VMs can run any Wayland/Xorg DE just like in QEMU.

We went with this approach as it meant the least attack surface and will work with mediated GPU pass-through using something like libVF.IO for our application and OS VMs. This will in future allow for near native GPU performance for apps (and games) using hardware assisted virtualization.

The guest agent resizes the application in the VM on resizing the host window. This doesn't work with OS virtualization yet.

Other approaches we considered for display output:

a. Using virtio-wl

GUI virtualization was already possible using virtio-wl. But it needs the guest to use chromium kernel or virtio-gpu with wayland context which hasn't been merged into the kernel yet.

Plus there is a subjective security concern towards exposing wayland server to the untrusted VM. Wayland is designed with security in mind, but using virtio-wl/virtio-gpu with wl context still extends the attack surface to the wayland server and GPU DRM drivers.

b. Using firecracker/cloud hypervisor with nested X server

These VMMs are derived/forked from crosvm and are built to run on linux without any tweaks and were very tempting to use, but they lack virtio-gpu support and the only way to run GUI apps was to run a nested X server like x11docker.

This approach provided a lot of inspiration for the end outcome, but didn't fit into the isolation requirements we had.

2. Clipboard

Clipboard is sent to the guest only when needed. When the user presses ctrl+v and the app window is focused, the host sends the clipboard content from the host to the guest agent and the agent simulates the same key combination on the guest.

Similarly for ctrl+c the host requests the clipboard from the guest and the content is copied to the hosts clipboard.

This can be extended to support images, files, etc.

3. Audio

Sound is sent from the guest over TCP socket to pulseaudio on the host. The simplest way to do this is to set the PULSE_SERVER environment variable inside the VM to connect to the host pulse auido server.

This isn't the ideal way to do this at all, but this is just a workaround for the demo till emulated audio (ac97) works in linux host (Issue).

4. Keyboard and Mouse input

Keyboard and mouse input is supported by crosvm natively using --display-window-keyboard and --display-window-mouse. But we couldn't get the mouse to work due to some bug. So we did a workaround for now by replaying mouse events from the host Xwindow to the guest via the agent.

In future, once the bug is fixed, we wouldn't have to use the agent for this.

5. File sharing

Files are shared with the guest using virtio-fs shared directory. Any files needed to be shared are bind mounted into this folder.

This also allows for runtime sharing of files and folders and enables features like drag & drop and multimedia and file clipboard operations.

6. AppVM Disk

The App VM images are essentially built from a Dockerfile for now to allow for reproducible builds. We experimented with alpine and ubuntu base images.

Alpine base images resulted in tiny VM images (~300MB for a Firefox VM image with Xorg and dependencies). But it needed a lot of tweaking to get it to work, so we opted for a Ubuntu instead which resulted it large image sizes, but was easy to prototype with.

The below Dockerfile is what we used to create the base image and can be passed in args to build any apps during build.

FROM ubuntu

RUN apt-get update && \
    apt-get install -y dhcpcd5 util-linux systemd systemd-sysv
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y xorg
RUN apt-get install -y i3-wm xterm sudo xss-lock netplan.io nano net-tools inetutils-ping iproute2 xdotool

# Disable gdm so that we can start i3 by default
RUN systemctl disable gdm

# Install Firefox and VLC
ARG PACKAGES='firefox vlc'
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y $PACKAGES

# Copy custom init script
COPY files/init /init
RUN chmod +x /init

# Networking auto config
COPY files/netplan.yaml /etc/netplan/00-default.yaml
RUN systemctl disable dhcpcd

# Creating a new user for login
RUN useradd -s /bin/bash -d /home/user/ -m -G sudo user
RUN chown -R user:user /home/user

# Autologin tty0 with user 
RUN mkdir -p /etc/systemd/system/[email protected]
COPY files/override.conf /etc/systemd/system/[email protected]/override.conf

# For autostart of i3 with config
COPY files/.xinitrc /home/user/.xinitrc
COPY files/config /etc/i3/config
COPY files/.bash_profile /home/user/.bash_profile

# App startup script
COPY files/app.sh /opt/app.sh
RUN chmod +x /opt/app.sh
RUN echo 'exec --no-startup-id /opt/app.sh' >> /etc/i3/config

The docker build output rootfs is converted into a qcow2 image and used along with the kernel to boot into the OS.  

There is a much better way to do this with NixOS and Nix store, which will result in single small base image with Nix store (or a subset of apps) mounted into the VM, which we will be using here on.

7. Kernel

We built our appvm linux kernel with Virtio gpu, virtio fs, input driver support built-in. Other than that, it's just a standard linux kernel.

We plan to strip unnecessary drivers, components from the kernel and harden it to further reduce the attack surface.

Whats next for OpenVMM?

A usable release of OpenVMM will be released soon, if you'd like to contribute towards that, please stay tuned for the initial release or subscribe to get notified.

This will enable application VM sandboxing for all linux users and a minimal rust alternative to qemu.

Bheem OS

Bheem OS is a next generation secure operating system. It takes the security by isolation principle from Qubes OS further by virtualizing every application and most OS functions into rust MicroVMs.

The goal is to build an attractive, simple and fully featured OS that everyone would like to use, while providing ultimate security, performance and stability.

This PoC is a step towards making that happen. We will be following up with another post in the coming weeks with more information on Bheem OS and another demo.

About Openw3b

Openw3b's vision is to create an ecosystem of free and open-source software and hardware solutions that provides a compelling alternative that values peoples privacy and freedom.

Bheem Linux is a key component in this ecosystem with more to come.

Credits

Additional references