ZFS Compatibility

The best free filesystem on Earth – ZFS – also often named OpenZFS recently – has also become very portable in recent years of its development. The OpenZFS Distributions page lists 6 (six) operating systems already.

They are:

• FreeBSD
• Illumos
• Linux
• MacOS
• NetBSD
• Windows

… but if you would like to create a ZFS pool compatible with all of them … which options and ZFS features should you choose? There is OpenZFS Feature Flags page dedicated exactly to that topic.

These are the ones that have yes value in all operating systems.

• async_destroy
• bookmarks
• empty_bpobj
• enabled_txg
• filesystem_limits
• lz4_compress
• hole_birth
• multi_vdev_crash_dump
• spacemap_histogram

I would also include these as only older NetBSD 4.0.5 version does not support them – but they are supported in newer NetBSD 5.3 version.

• embedded_data
• large_blocks
• sha512
• skein

There is also a dedicated zpool-features(7) man page for that information on the OpenZFS page and also zpool-features(5) man page on the FreeBSD page.

On the FreeBSD system the /usr/share/zfs/compatibility.d directory has files with supported ZFS Feature Flags for many major operating systems and ZFS versions.

% ls /usr/share/zfs/compatibility.d
2018
2019
2020
2021
compat-2018
compat-2019
compat-2020
compat-2021
freebsd-11.0
freebsd-11.1
freebsd-11.2
freebsd-11.3
freebsd-11.4
freebsd-12.0
freebsd-12.1
freebsd-12.2
freenas-11.0
freenas-11.1
freenas-11.2
freenas-11.3
freenas-9.10.2
grub2
openzfs-2.0-freebsd
openzfs-2.0-linux
openzfs-2.1-freebsd
openzfs-2.1-linux
openzfsonosx-1.7.0
openzfsonosx-1.8.1
openzfsonosx-1.9.3
openzfsonosx-1.9.4
truenas-12.0
ubuntu-18.04
ubuntu-20.04
zol-0.6.1
zol-0.6.4
zol-0.6.5
zol-0.7
zol-0.8

Unfortunately it misses NetBSD and Illumos systems for example … but having information from the OpenZFS Feature Flags page we can find Feature Flags set that will be supported everywhere.

Here are the stats for supported ZFS Feature Flags. The higher the number the more operating systems and ZFS version it covers.

% grep -h '^[^#]' /usr/share/zfs/compatibility.d/* \
    | sort -n \
    | uniq -c \
    | sort -n
   2 draid
   5 bookmark_written
   5 device_rebuild
   5 livelist
   5 log_spacemap
   5 redacted_datasets
   5 redaction_bookmarks
   5 zstd_compress
   9 allocation_classes
   9 bookmark_v2
   9 project_quota
   9 resilver_defer
  10 edonr
  10 encryption
  11 large_dnode
  11 userobj_accounting
  18 spacemap_v2
  20 device_removal
  20 obsolete_counts
  20 zpool_checkpoint
  31 sha512
  31 skein
  32 multi_vdev_crash_dump
  36 filesystem_limits
  36 large_blocks
  37 bookmarks
  37 embedded_data
  37 enabled_txg
  37 extensible_dataset
  37 hole_birth
  37 spacemap_histogram
  38 async_destroy
  38 empty_bpobj
  38 lz4_compress

As the GNU GRUB is very outdated when it comes to ZFS support it should be pretty bulletproof to use it as a starting point of limited ZFS Feature Flags support.

% cat /usr/share/zfs/compatibility.d/grub2
# Features which are supported by GRUB2
async_destroy
bookmarks
embedded_data
empty_bpobj
enabled_txg
extensible_dataset
filesystem_limits
hole_birth
large_blocks
lz4_compress
spacemap_histogram

To make sure we are compatible we will now cross-link the GNU GRUB data.

First we will ‘generate’ the grep(1) command arguments that we will use in the next command.

% grep '^[^#]' /usr/share/zfs/compatibility.d/grub2 \
  | while read I
    do
      echo "-e ' ${I}' \\"
    done
-e ' async_destroy' \
-e ' bookmarks' \
-e ' embedded_data' \
-e ' empty_bpobj' \
-e ' enabled_txg' \
-e ' extensible_dataset' \
-e ' filesystem_limits' \
-e ' hole_birth' \
-e ' large_blocks' \
-e ' lz4_compress' \
-e ' spacemap_histogram' \

Lets now use these arguments to filter the ZFS features.

% grep -h '^[^#]' /usr/share/zfs/compatibility.d/grub2/* \
    | sort -n \
    | uniq -c \
    | sort -n \
    | grep -e ' async_destroy' \
           -e ' bookmarks' \
           -e ' embedded_data' \
           -e ' empty_bpobj' \
           -e ' enabled_txg' \
           -e ' extensible_dataset' \
           -e ' filesystem_limits' \
           -e ' hole_birth' \
           -e ' large_blocks' \
           -e ' lz4_compress' \
           -e ' spacemap_histogram' \
    | wc -l
      11

% grep -h '^[^#]' /usr/share/zfs/compatibility.d/grub2 | wc -l
      11

So if seems that GRUB list of ZFS Feature Flags seems pretty compatible.

Lets now cross-reference that GRUB data with the data from the OpenZFS Feature Flags page.

I will create new /usr/share/zfs/compatibility.d/OZFF file that has these ZFS Feature Flags as content.

% cat /usr/share/zfs/compatibility.d/OZFF
async_destroy
bookmarks
empty_bpobj
enabled_txg
filesystem_limits
lz4_compress
hole_birth
multi_vdev_crash_dump
spacemap_histogram
embedded_data
large_blocks
sha512
skein

% wc -l /usr/share/zfs/compatibility.d/OZFF
      13

So there are 11 GRUB ZFS Feature Flags and 13 OpenZFS ‘Compatible’ Feature Flags.

Lets see how it they compare.

% cat /usr/share/zfs/compatibility.d/OZFF \
    | grep -e async_destroy \
           -e bookmarks \
           -e embedded_data \
           -e empty_bpobj \
           -e enabled_txg \
           -e extensible_dataset \
           -e filesystem_limits \
           -e hole_birth \
           -e large_blocks \
           -e lz4_compress \
           -e spacemap_histogram \
    | wc -l
      10

I expected 11 here instead of 10 … we will not have to compare GRUB results to the OpenZFS Feature Flags section.

% grep -h '^[^#]' /usr/share/zfs/compatibility.d/{grub2,OZFF} \
    | sort -n \
    | uniq -c \
    | sort -n
   1 extensible_dataset
   1 multi_vdev_crash_dump
   1 sha512
   1 skein
   2 async_destroy
   2 bookmarks
   2 embedded_data
   2 empty_bpobj
   2 enabled_txg
   2 filesystem_limits
   2 hole_birth
   2 large_blocks
   2 lz4_compress
   2 spacemap_histogram

Seems that we finally got our 10 most compatible OpenZFS Feature Flags set.

Its this set:

• async_destroy
• bookmarks
• embedded_data
• empty_bpobj
• enabled_txg
• filesystem_limits
• hole_birth
• large_blocks
• lz4_compress
• spacemap_histogram

To make it more comfortable to use we will put them it into the separate /usr/share/zfs/compatibility.d/COMPATIBLE file.

# cat /usr/share/zfs/compatibility.d/COMPATIBLE
async_destroy
bookmarks
embedded_data
empty_bpobj
enabled_txg
filesystem_limits
hole_birth
large_blocks
lz4_compress
spacemap_histogram

Lets now try to make that best effort most-compatible ZFS pool.

I will use one of my scripts – mdconfig.sh – to easy manipulate md(4) memory disks on FreeBSD.

# truncate -s 1g FILE
# mdconfig.sh -c FILE
IN: created vnode at /dev/md0
# zpool create -o compatibility=COMPATIBLE compatible /dev/md0
# zpool list
NAME         SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
compatible   960M   116K   960M        -         -     0%     0%  1.00x    ONLINE  -
zroot        118G  48.6G  69.4G        -         -    36%    41%  1.00x    ONLINE  -

Now lets see what is zpool upgrade command showing us.

# zpool upgrade
This system supports ZFS pool feature flags.

All pools are formatted using feature flags.


Some supported features are not enabled on the following pools. Once a
feature is enabled the pool may become incompatible with software
that does not support the feature. See zpool-features(7) for details.

Note that the pool 'compatibility' feature can be used to inhibit
feature upgrades.

POOL  FEATURE
---------------
compatible
      multi_vdev_crash_dump
      large_dnode
      sha512
      skein
      userobj_accounting
      encryption
      project_quota
      device_removal
      obsolete_counts
      zpool_checkpoint
      spacemap_v2
      allocation_classes
      resilver_defer
      bookmark_v2
      redaction_bookmarks
      redacted_datasets
      bookmark_written
      log_spacemap
      livelist
      device_rebuild
      zstd_compress
      draid
zroot
      userobj_accounting
      encryption
      project_quota
      allocation_classes
      resilver_defer
      bookmark_v2
      redaction_bookmarks
      redacted_datasets
      bookmark_written
      log_spacemap
      livelist
      device_rebuild
      zstd_compress
      draid

There are LOTS of ZFS Feature Flags to be activated but if we want to have our ZFS pool keep compatible – we will have to stay away from it

You may get impression that you miss a lot … but you do not miss that much. Here are the ZFS Feature Flags you can fully utilize.

# zpool get all compatible | grep -v disabled
NAME        PROPERTY                       VALUE                          SOURCE
compatible  size                           960M                           -
compatible  capacity                       0%                             -
compatible  altroot                        -                              default
compatible  health                         ONLINE                         -
compatible  guid                           5395735446052695775            -
compatible  version                        -                              default
compatible  bootfs                         -                              default
compatible  delegation                     on                             default
compatible  autoreplace                    off                            default
compatible  cachefile                      -                              default
compatible  failmode                       wait                           default
compatible  listsnapshots                  off                            default
compatible  autoexpand                     off                            default
compatible  dedupratio                     1.00x                          -
compatible  free                           960M                           -
compatible  allocated                      116K                           -
compatible  readonly                       off                            -
compatible  ashift                         0                              default
compatible  comment                        -                              default
compatible  expandsize                     -                              -
compatible  freeing                        0                              -
compatible  fragmentation                  0%                             -
compatible  leaked                         0                              -
compatible  multihost                      off                            default
compatible  checkpoint                     -                              -
compatible  load_guid                      17463015630652190527           -
compatible  autotrim                       off                            default
compatible  compatibility                  COMPATIBLE                     local
compatible  feature@async_destroy          enabled                        local
compatible  feature@empty_bpobj            enabled                        local
compatible  feature@lz4_compress           active                         local
compatible  feature@spacemap_histogram     active                         local
compatible  feature@enabled_txg            active                         local
compatible  feature@hole_birth             active                         local
compatible  feature@extensible_dataset     enabled                        local
compatible  feature@embedded_data          active                         local
compatible  feature@bookmarks              enabled                        local
compatible  feature@filesystem_limits      enabled                        local
compatible  feature@large_blocks           enabled                        local

You get the very decent LZ4 compression and also ZFS Bookmarks feature which are very useful feature for sync|recv mechanism.

Keep in mind that the -o compatibility= switch for zpool(8) is available on OpenZFS 2.1 or newer. The 2.1 version is already available on the FreeBSD 13.1-BETA* releases and will be part of the FreeBSD 13.1-RELEASE systems. To make use of it on older FreeBSD releases you will have to use openzfs and openzfs-kmod packages and also use the following settings in the /boot/loader.conf file.

From that one:

zfs_load=YES

Into that one:

zfs_load=NO
openzfs_load=YES

With these openzfs and openzfs-kmod packages and above settings in the /boot/loader.conf file you can use this OpenZFS 2.1 on FreeBSD 12.2 – on FreeBSD 12.3 – and on FreeBSD 13.0 … and of course on upcoming FreeBSD 13.1 release.

Not sure if I should have add anything more here. Feel free to remind me in the commends