LarskoOrg : ZfsUbuntu

HomePage :: Categories :: PageIndex :: RecentChanges :: RecentlyCommented :: Login/Register

Encrypted ZFS with Ubuntu

ZFS is one of the most advanced filesystems. Now it can be used natively on Linux. The only major drawback is that native ZFS encryption is not available. This article shows how use Linux' disk encryption to install Ubuntu onto an encrypted disk with ZFS.

This article originally appeared in Ubuntu User? issue 16.


The ZFS filesystem was developed by Sun Microsystems and released as Open Source to the public in November 2005 as part of OpenSolaris. Then as now it offers features that are not found in any other filesystems. One of the main design aims was to make it as robust and reliable as possible. ZFS stores not only the data, but also cryptographic checksums of the data. These enable it to identify data that has been altered because of bit-rot, power failures or people playing with magnets. Today, after years of development, improvements and
additional features, there is hardly anything it does not do.

While many features, such as support for various RAID modes, hot-swapping of disks and very large limits on data size, number of files and similar attributes, are more geared towards the enterprise user, there are plenty that are of interest to home users. ZFS supports transparent compression for example with a choice of different compression algorithms and levels. Data can be deduplicated automatically, making all those backup copies less of a space hog. Speaking of backups, ZFS not only supports block-level snapshots that only consume space when files actually change, but also the ability to send and receive those snapshots as filesystem streams over the network. Anybody who has had to run rsync on large numbers of files will much appreciate the performance improvement this brings.

ZFS on Linux

While ZFS has been the main filesystem on OpenSolaris since 2005 and fully supported on FreeBSD and other BSDs for several years, its adoption by the Linux community has been somewhat slower. The reason for this is not a lack of enthusiasm, but rather legal issues. ZFS was released under the Common Development and Distribution License (CDDL). This license is not compatible with the GPL, meaning that ZFS cannot be bundled with the Linux kernel. One way around this restriction was to implement ZFS support through a Filesystem in Userspace (FUSE) module. This comes with a number of limitations however. Most notably, there is a significant performance penalty for running through FUSE and ZFS cannot be used as a root filesystem.

More recently, the ZFS on Linux project has provided another solution to this problem. It implements ZFS support as a kernel module that does not rely on FUSE, but exploits a licensing loophole. As long as the user compiles the ZFS kernel module herself and does not distribute it, there are no legal problems. ZFS on Linux makes it feasible to use ZFS as a root filesystem on Linux and an increasing number of distributions are providing ZFS on Linux packages.

Among the alternatives to ZFS are Btrfs for Linux and HAMMER for DragonFly BSD. They both aim to support similar feature sets, but neither is currently as stable and mature as ZFS. Especially Btrfs still has a number of stability issues. The mainstream adoption of HAMMER is further inhibited by the fact that it is only fully supported in a niche operating system.

There is one major feature missing in the current Open Source version of ZFS: filesystem encryption. While implemented and available in Solaris, the code has not made its way to the public yet. With the acquisition of Sun Microsystems by Oracle and the subsequent change of attitude towards derivative Open Source projects, it is questionable if and when this code will be released.

Even though native ZFS encryption is not available at the time of writing, there is no reason to expose ZFS data to anybody who cares to look. There are many block-level disk encryption solutions available and they are all compatible with ZFS. PC BSD (a FreeBSD derivative) even offers support for an encrypted ZFS root filesystem in the installer -- a simple click is enough to set it up. On Linux, this procedure is not as simple. No installer offers the option to use ZFS at all at the moment. With some additional effort, this is not too difficult to achieve though.

In the remainder of this article, we are assuming that you want to install Ubuntu 12.10. Most of the steps described are not specific to this particular distribution, but it provides packages for all the relevant software (if not already included in the installer). For other distributions, the installation will be similar, but additional or different steps may be required to retrieve, compile and install software.

A word of warning

You should be familiar with how Linux works and comfortable with using the command line. The instructions below describe how to set up the basic system with respect to the filesystem and related matters. You may need to set up some additional things like support for specific hardware yourself.

After following the steps in this article, you will have only a very basic installation that will require significant additional setup. If you are not comfortable with configuring and installing Ubuntu Linux from scratch, you should not try to do this without help from somebody who is.

Running a pure ZFS Linux system is still somewhat experimental. While the implementation of ZFS itself can be considered stable and mature, the interface to the Linux kernel is relatively new. Furthermore, there is very little support for the setup described here -- things may break when upgrading the system and if they do, your choice of tools to recover and fix your system will be limited.

If you still feel adventurous enough to undertake this installation despite all the caveats, read on.

Getting started

Before starting the actual installation, get the latest Ubuntu 12.10 boot image. While it is possible to install a 32 Bit ZFS system, it is highly recommended to use the 64 Bit image. Boot the machine you want to install with the image you got and choose "Try Ubuntu" at the initial screen. Once the live distribution is up and running, open a terminal and become root.

The first thing you should do is to set up the network. You will require a network connection during the installation. After that, partition the hard drive you want to install Ubuntu on. You will need a small partition to boot from. The rest of the disk can be allocated to the encrypted ZFS partition. Remember to set the "bootable" flag on your boot partition. Now create a filesystem of your choice on that partition. We will assume that the disk you want to use for the installation is /dev/sda with boot partition /dev/sda1 and the rest of the disk in /dev/sda2. If the device name differs on your system or you prefer to use UUIDs, change the instructions below accordingly.

Setting up the disk and bootstrapping

First, we are going to set up the encrypted disk. Enter the following into your terminal.

# cryptsetup luksFormat -l 512 -c aes-xts-plain64 -h sha512 /dev/sda2
# cryptsetup luksOpen /dev/sda2 cryptroot

The first command will set up the disk as an encrypted partition with a keyfile size of 512 Bytes using sha512 as a passphrase hash and aes-xts-plain64 as the encryption cipher. You are of course free to adapt these options to your liking. If you want to increase the level of security, overwrite /dev/sda2 with random data before setting it up as an encrypted partition. Note that with a large disk this may take days however. The second command opens the encrypted partition we just created as "cryptroot". You will need to enter the password that you gave when the container was set up here.

These commands will set up the entire partition for ZFS. If you want to separate it into multiple slices (for example to have encrypted but non-ZFS swap), you can create multiple volumes on top of the encrypted container with the commands provided by lvm2. Note that if you choose this route, you will need to install lvm2 in the system we are about to set up. In the setup described here, we will create a ZFS volume for swap later.

After creating the encryption layer, we can now proceed to setting up the actual ZFS filesystem. The Ubuntu installer image does not come with ZFS support. Fortunately, there is a personal package archive (PPA) for ZFS that makes installing the kernel modules and other required software easy. In your terminal, enter the following.

# apt-add-repository --yes ppa:zfs-native/stable
# apt-get update
# apt-get install debootstrap ubuntu-zfs

This will add the PPA, update the package index and install ZFS support as well as the debootstrap utility that we will use later to install the system. As the ZFS kernel module cannot be distributed in binary form and needs to be compiled, this step will take a few minutes. Now we can set up the ZFS pool.

# zpool create -O mountpoint=none -o ashift=12 rpool /dev/mapper/cryptroot
# zfs create -o mountpoint=/ rpool/root
# zpool set bootfs=rpool/root rpool
# zpool export rpool
# zpool import -R /mnt rpool

The first command sets up a pool named "rpool" for a disk with a block size of 4 KiB (ashift=12). Then we create a single filesystem for the entire Ubuntu installation. Note that this command will cause an error message because zfs will attempt to mount the new filesystem at the specified mount point. This is of no consequence and saves us having to specify the mount point later.

In this article, we are creating only one filesystem for everything for the sake of simplicity. In a real setup, it is advisable to add a number of additional filesystems for the various parts of the system. A more complex setup allows to take better advantage of the many features ZFS offers. For each filesystem, we can specify whether to compress and at what level, reserve a minimum amount of space to be available or restrict the size. Snapshots can be created per filesystem and immensely simplify common backup tasks. It is however not necessary to anticipate and create all the filesystems one might need at this point. They can be created easily later.

After setting the "bootfs" property on the root filesystem, we export the pool. It is reimported with an alternative root of /mnt, meaning that the filesystem will be mounted at that location. Exporting the pool allows us to specify the alternate root, but also makes sure that all metadata of the newly created pool and filesystem is written to disk.

Now we can proceed with the actual installation.

# debootstrap quantal /mnt

This will take a while to complete. Make a cup of tea while you wait. To speed things up a bit, we could have given the appropriate directory on the Ubuntu live image as source for the packages to install; the given command will fetch them over the network.

Setting up the system

After the bootstrapping is done, we can set up the newly installed system. chroot into it.

# mount --bind /dev/ mnt/dev
# chroot /mnt /bin/bash --login

To make it easier to differentiate between commands to be run in the live system and in the newly installed one, all commands run in the chroot will be prefixed with two hashes. The first thing to do in the new system is to mount all the filesystems we need.

## mount /dev/sda1 /boot
## mount -t proc proc /proc
## mount -t sysfs sysfs /sys
## ln -sf /dev/mapper/cryptroot /dev/cryptroot

Do not mount the special filesystems from the live system as we did for /dev. If you do this, the bootloader will be unable to identify the root filesystem and will not create a boot configuration file. The last command is needed for the same reason. Next, generate the default locale.

## locale-gen en_US.UTF-8

While this is not strictly necessary, it will save you from error messages during the rest of the setup. Now it is time to install some additional software.

## apt-get update
## apt-get install ubuntu-minimal software-properties-common

If you used the boot image as a package source during the installation and it is out of date, you might want to upgrade the already installed software as well. The package software-properties-common contains the apt-add-repository command that we will need to add the ZFS PPA.

## apt-add-repository --yes ppa:zfs-native/stable
## apt-get update
## apt-get install ubuntu-zfs cryptsetup

In addition to installing ZFS support like we did in the live system, cryptsetup is added so we can handle the encrypted partition. The last command will install a whole slew of additional packages, including the Grub boot loader. When asked where to install it, choose the system disk (here /dev/sda). Next, create a ZFS volume for swap.

## zfs create -V 4G rpool/swap
## mkswap /dev/rpool/swap
## swapon /dev/rpool/swap

In this example, the size of the swap partition is 4 GiB. Adapt this figure according to your particular needs. This is a good time to set up /etc/fstab. You will need entries for /boot and swap.

Making it boot

The steps described so far are reasonably straightforward and do not deviate significantly from a more or less standard installation. In order to make the new system boot however, some more elaborate steps are required.

Booting a ZFS system requires a special initial RAM disk. While there is a PPA and package for this, the version for 12.10 (Quantal) has broken dependencies and cannot be installed. The reason for this is that booting used to require a specially-patched Grub. With the inclusion of Grub 2.00 in Ubuntu 12.10, this is no longer necessary but the dependencies have not been updated to reflect this yet. While not the prettiest solution, a feasible workaround is to install the files that this special initramfs package provides manually.

## apt-get install wget
## wget
## dpkg -x zfs-initramfs_0.6.0.91-0ubuntu1~quantal1_amd64.deb zfs-initramfs
## cp -r zfs-initramfs/* /

The only files the zfs-initramfs package contains are hooks for the tool that creates the initial RAM disk. These hooks copy the required binaries and configuration into the initramfs. Next, set up the configuration files to tell the system about the encrypted partition the system is installed on.

## echo 'cryptroot UUID=<UUID> none luks' >> /etc/crypttab

The first component in the configuration file designates the name of the device that provides access to the encrypted container. It is important that you choose the same name you used to set up the ZFS pool and the rest of the system. The UUID of the encrypted partition can be obtained using the blkid command. You could use a filesystem path instead of the UUID here, but as the paths are assigned dynamically by udev and it is possible for them to change, using the UUID is the recommended way. Similar configuration must be provided for the
tool that creates the initial RAM disk.

## echo 'target=cryptroot,source=UUID=<UUID>,key=none,rootdev' >> /etc/initramfs/conf.d/cryptroot

The name of this file is arbitrary; "cryptroot" was chosen here to be consistent with the rest of the setup. The name given as "target" however must be the same as for the rest of the setup again. The UUID is the same as before. This concludes the configuration required to create the initial RAM disk. Run the following command to create it.

## update-initramfs -c -k all

The other part required to make the system bootable is the bootloader. The Grub configuration needs to be changed to boot the ZFS system. In /etc/default/grub, find the command line passed to the kernel (GRUB_CMDLINE_LINUX_DEFAULT). Add the following options to it.

root=ZFS=rpool/root boot=zfs

This will tell Grub to boot a ZFS system and where to find the root filesystem. Also remove the "splash" option from the command line -- the splash screens are not installed in this basic installation and enabling this option will prevent you from seeing the prompt for the passphrase of the encrypted partition. Now let Grub generate a new configuration file.

## update-grub

This is the main configuration done. Before we can boot into the new system, only a few loose ends need to be tied up. The symlink to the encrypted device we created earlier in /dev needs to be persistent in order for future updates of the Grub configuration to succeed. Tell udev to create it as follows.

## echo 'ENV{DM_NAME}=="cryptroot", SYMLINK+="cryptroot"' > /etc/udev/rules.d/99-local.rules

The particular number at the beginning of the file name does not matter as long as it ensures that this rule file is run after the mappings for the encrypted containers have been set up. The file that does this has number 55 on the image I used. Again the name given in the file needs to match the name for the encrypted mapping specified earlier.

All that remains to do to make the new system usable is to set the root password.

## passwd root

Do not forget this step. You will not be able to use the new system unless you do this. Optionally, set up network interfaces and other basic system configuration. Finally, unmount the filesystems and exit the chroot.

## umount /boot
## umount /sys
## umount /proc
## exit

In the live system, unmount the chroot's /dev directory, export the pool and reboot.

# umount /mnt/dev
# zpool export rpool
# reboot

Congratulations! You should now have a pure encrypted ZFS system! You can now proceed to set up the rest of the system, add users and install a desktop environment.

The support for this kind of setup by Ubuntu is still somewhat lacking. In particular you should take care that after kernel upgrades the ZFS modules and the initial RAM disk are recreated correctly. You will not be able to boot if they are not!

Adding a cache device

ZFS pools can have cache devices that are used to speed up the filesystem operations. This will usually be solid-state disk. Many modern systems (even some laptops) have SSDs in addition to normal harddisks now. Adding a cache device can be done by executing a simple command, but in our case we should encrypt the cache device of course.

Having another encrypted device would require to enter a second passphrase when booting. This would be annoying and is unnecessary, as the Ubuntu cryptsetup package ships with a script that allows to derive a passphrase from data on a disk. We are going to use the decrypted main disk as source for the derived passphrase. While not as secure as an independent passphrase, it does not constitute a security risk here because the main disk still needs to be decrypted before the passphrase can be derived and the second disk is only a cache device.

Assuming that the cache device is /dev/sdb, set up the encrypted partition as follows.

# /lib/cryptsetup/scripts/decrypt_derived cryptroot > /tmp/key
# cryptsetup luksFormat -l 512 -c aes-xts-plain64 -h sha512 /dev/sdb --key-file /tmp/key
# rm /tmp/key
# /lib/cryptsetup/scripts/decrypt_derived cryptroot | cryptsetup luksOpen /dev/sdb cryptcache

This will set up the new encrypted container as "cryptcache" with the same parameters as the other one. Adding the encrypted device as cache to the pool is straightforward.

# zpool add rpool cache /dev/mapper/cryptcache

Now we just need to add the information about the encrypted device to the system. No additional steps are required to make the ZFS configuration persistent.

# echo 'cryptcache UUID=<UUID> cryptroot luks,keyscript=/lib/cryptsetup/scripts/decrypt_derived' >> /etc/crypttab

You can verify that the cache device is present and obtain usage statistics by typing zpool iostat -v.


The ZFS on Linux project has made it possible to use the advanced ZFS filesystem for the system root on Linux installations. Even though this is still a somewhat exotic setup and support for it is lacking, the tools that are currently available are entirely adequate for setting up a pure ZFS Linux system and even throw full disk encryption into the bargain.

This article demonstrated all that is needed to set up such a system. While some experience with Linux is necessary to perform the steps confidently, you do not need to be a Guru to complete them.

In the future, using ZFS on Linux will hopefully become much easier and be supported by standard installers. Some BSD distributions are one step ahead in this respect, but no doubt Linux will catch up soon.


There is one comment on this page. [Display comment]

Valid XHTML 1.0 Transitional? :: Valid CSS? :: Powered by Wikka Wakka Wiki
Page was generated in 0.1114 seconds