My goal is to install FreeBSD on a system with 2 disks, keeping the mirroring of the disks for data security. You can find several tutorials on how to install with one encrypted ZFS pool, and in 11.0-RELEASE, the BSDinstall partitionner can do this for you. However I wanted to use a more complex setting, that I will explain below.

Hardware used

I use a Dell server (bare metal) with two 1 To disks, for disk encryption, the CPU has the AES-NI instruction set.

Rational

This server is located in a datacenter (I rent it) and I have no physical access to it.[1] But I want that my data be encrypted so that if a disk is removed from the server (disk failure, seizure or error) the data is protected. So I need to be able to boot a minimal system, which allows me a remote access (through ssh), then allows me to enter the decryption password, mount the encrypted volume(s) and start the remaining services. For this I will partition my disks and use those partitions to directly define a root ZFS pool or (through geli) an encrypted ZFS pool.

Installation

I do the installation through the IDRac remote console, mounting the FreeBSD ISO image in the virtual CD. I use the standard installer, but at the partitionning stage I choose the "Shell Partitionning". This gives me a shell, and (in order for the installer to work) I must setup things so that system filetree is found under /mnt and necessary config files are defined in /tmp/bsd_install_etc. In this directory we set up the following files:

loader.conf
zfs_load="YES"
geom_mirror_load="YES"
fdescfs_load="YES"
nullfs_load="YES"
#
#
geom_eli_load="YES"
crypto_load="YES"
aesni_load="YES"
fstab
/dev/mirror/swap.eli    none    swap    sw      0 0
rc.conf
zfs_enable="YES"

Base partitioning

I use gpart for this partitioning and I based it on Ollivier Robert’s Documentation. My two disks are seen as /dev/ada0 and /dev/ada1 by the installer.

  • First step is to reset the disks partition table

    dd if=/dev/zero of=/dev/ada0 bs=512 count=10
dd if=/dev/zero of=/dev/ada1 bs=512 count=10

  • Then create a gpt partition table

    gpart create -s gpt ada0
gpart create -s gpt ada1

  • We then create our partitions. We need a small partition for the boot code, one for the system (root) pool, one for swap and the rest for data (which will be encrypted). Since we want to mirror the partitions, we need to use the same partitioning scheme on both disks. The leads to

    gpart add -s 100K -a 4k -t freebsd-boot ada0
gpart add -s 70G -a 4k -t freebsd-zfs -l sys0 ada0
gpart add -s 32G -a 4k -t freebsd-swap -l swap0 ada0
gpart add -a 4k -t freebsd-zfs -l data0 ada0
    gpart add -s 100K -a 4k -t freebsd-boot ada1
gpart add -s 70G -a 4k -t freebsd-zfs -l sys1 ada1
gpart add -s 32G -a 4k -t freebsd-swap -l swap1 ada1
gpart add -a 4k -t freebsd-zfs -l data1 ada1

We can now see the partitions

#ls /dev/gpt
data0  data1  swap0  swap1  sys0   sys1

Giving their role to the partitions

We first define both disks as bootable, to be able to boot when any of the disks is down.

gpart set -a bootme -i 2 ada0
gpart set -a bootme -i 2 ada1

Now we manually load the necessary modules to continue

kldload zfs
kldload geom_mirror
kldload geom_eli
kldload crypto
kldload aesni

Then we create the system pool, that we chroot on /mnt for the installer, and the datasets ins this pool

zpool create -R /mnt -O mountpoint=none system mirror gpt/sys0 gpt/sys1
zfs set compression=lz4 system
zfs set checksum=fletcher4 system
zfs create -o mountpoint=/ system/root
zfs create system/root/usr
zfs create system/root/usr/src
zfs create system/root/usr/obj
zfs create system/root/usr/local
zfs create system/root/tmp
zfs create system/root/var
zpool set bootfs=system/root system

Now that we have the basic tree, we add the boot code in the first partition of the disks

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1

The partitioning is now the following one :

# gpart show
=>        40  1953525088  ada0  GPT  (932G)
          40         200     1  freebsd-boot  (100K)
         240   146800640     2  freebsd-zfs  [bootme]  (70G)
   146800880    67108864     3  freebsd-swap  (32G)
   213909744  1739615376     4  freebsd-zfs  (830G)
  1953525120           8        - free -  (4.0K)

=>        40  1953525088  ada1  GPT  (932G)
          40         200     1  freebsd-boot  (100K)
         240   146800640     2  freebsd-zfs  [bootme]  (70G)
   146800880    67108864     3  freebsd-swap  (32G)
   213909744  1739615376     4  freebsd-zfs  (830G)
  1953525120           8        - free -  (4.0K)

The data partition (number 4) will be configured after the system is installed, as base for the encrypted pool.

Defining swap

We use a standard geom mirror for the swap, defined by

gmirror label swap gpt/swap0 gpt/swap1

The line we put in /tmp/bsdinstall_etc/fstab will define an encrypted partition over this mirror as the swap of our system.

We can now exit the shell, let the installer do its job and reboot the system.

The encrypted pool

After reboot we have (hopefully) a working system on which we can ssh to define our new encrypted pool that we will use for our sensitive data.

This a quite standard setting

  • First create the key file :

    root@tgv:~ # mkdir /root/keys
root@tgv:~ # dd if=/dev/random of=/root/keys/data.key bs=128k count=1

  • Then we encrypt the partitions

    root@tgv:~ # geli init -K /root/keys/data.key -s 4096 -l 256 /dev/gpt/dataO
root@tgv:~ # geli init -K /root/keys/data.key -s 4096 -l 256 /dev/gpt/data1

  • We now attach them to the system

    root@tgv:~ # geli attach -k /root/keys/data.key /dev/gpt/data0
root@tgv:~ # geli attach -k /root/keys/data.key /dev/gpt/data1

  • And now we create the pool

    root@tgv:~ # zpool create data mirror gpt/data0.eli gpt/data1.eli

Usage

After this we use the data pool creating datasets with their mountpoints inside, setting the compression, and all the standard zfs parameters. In my case I also use ezjail and I defined the jails in the data zpool, so my script simply starts the jail after mounting :

/usr/local/sbin/Mount-secure.sh
#! /bin/sh

KEY_FILE=/root/keys/data.key
POOL_NAME=data
LBL_NAMES="data0 data1"

for disk in $LBL_NAMES;do

    if [ ! -c /dev/gpt/${disk}.eli ]; then
	echo "decrypting disk $disk"
	if ! geli attach -k $KEY_FILE gpt/$disk ; then
	    echo "Error decrypting disk, please retry"
	    exit 1
	fi
    fi
done

if (zpool status $POOL_NAME >/dev/null); then
    zfs mount -a
else
    zpool import $POOL_NAME
fi

##
# Starting jails
#
#
ezjail-admin onestart
1. I have access to the IDrac console, but I found it not to be very reliable thus I prefer a solution where I can dispense myself of its use