Notes on debian wheezy kfreebsd and zfs

In late 2012, I purchased a new system with the intent of migrating my home storage (primarily several hundred GB of photos and a few dozen GB of email) to it. I felt it would be desirable to boot from a mirrored ZFS root and store my photos on a raidz1 pool.

The Hardware

The Software

Debian "Wheezy" kfreebsd-amd64

The hiccoughs

I've run into a number of bumps on the way to a working system.

Bad drives, bad service from newegg

One of the three drives did not spin up. A second drive had an IN_THE_PAST SMART failure attribute, and the last was just 1 point away from the failure threshold in the same attribute. I RMA'd the first drive, which newegg refused to replace claiming that I had damaged it. Meanwhile, their page of reviews for the drive had 3 or 4 reports similar to my experience—a very likely sign of a damaged batch of disks. When I called this to their attention and asked that they replace all three of my drives, they refused. I disputed the charge with my credit card company and eventually got back the full cost of the drives, but newegg inactivated my account. I'll be shopping elsewhere from now on.

Booting the Debian installer via PXE

On the server side, "tftpd 0.17-17ubuntu1" doesn't seem to work, while "tftpd-hpa 5.0-11ubuntu2" does. The symptom is that grub can't find grub.cfg.

Unless you insert 'pxe_unload' after the last kfreebsd_module line in grub.cfg, the network is "slightly wonky" in the installer—more specifically, it appears that all packets to the IP address of the TFTP server are gobbled up and userspace never sees them. The symptom is that the installer can't bring up the ethernet connection.

Once these details are worked out, it is very handy to start the debian installer without any media.

By the way, unless your internet connection is very fat, or you are sure you'll get all the installer choices right the first time, you're almost certain to want some kind of HTTP proxy that will cache the stuff from the debian package servers. I use apt-cacher-ng operating on http://10.x.x.x:3142/ and manually enter this setting in the installer.

Advanced Format drives and "ashift=12"

I've read on the internet and heard from a colleague that it's important to set up a zfs pool to use the disks' physical (as opposed to logical) block size. Unfortunately, the Debian installer offers no way to do this, and the "gnop" program used as a common workaround isn't available in the debian installer. After considering a number of alternatives, I elected to ignore the need for "ashift=12".

With a few hundred gigs of data on the ashift=12 raidz pool, "zpool scrub" proceeds at a rate of 250-300MB/s. Here are the results of a bonnie++ run:

Version 1.96Sequential OutputSequential InputRandom
Seeks
Sequential CreateRandom Create
SizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU
zaphod64G1529820871943121629244499925104223167.26161518951++++++++27149942377981++++++++2826395
Latency338ms2134ms2796ms65508us1195ms1727msLatency9181us2198us2329us22052us81us66us

Parted 'Could not get identity of device'

You have to ignore this warning every time. I should look for a bug about it on bugs.debian.org. This may have to do with the fact that /dev/adaX are character devices in kfreebsd; I'm not sure.

Booting

I am not an expert on UEFI vs BIOS booting or MBR vs GPT partitioning, but this was the biggest bump I ran into.

For disks larger than 2TiB (~2.2TB), the old-fashioned MBR partition table cannot define partitions that span the whole disk. As a consequence, Debian rightly selects GPT partitioning for the disks. However, in this circumstance it's necessary to have a "bios_grub"-flagged partition for bootloader data. If you do not have this, the symptom is that the bootloader hangs after printing "GRUB". It may be possible to create this partition in the installer, but I did it after the fact from a rescue shell using parted's "set <N> bios_grub on".

After this, my system still did not boot, presenting me with a "GRUB rescue" prompt. I ultimately determined that the problem was now that the right modules were not built into the grub core.img. Unfortunately, I did not keep the commandline that I used to fix this issue. I know it involved "grub-mkimage" and specifying to include the "part_gpt" and "zfs" modules. After that, I did "grub-install" to each disk.

I also used the program gptsync to try to synchronize the MBR partitions with the GPT partitions. However, I don't think that this was important.

I still have not tested that the system will still boot with any one drive disconnected, but I expect that it will.

As far as I can tell, I am doing BIOS (not GPT) booting, as I don't have any partitions of type "gpt_boot", so I don't have to tangle with UEFI and everything that entails.

Networking

My /etc/network/interfaces was generated with the line "iface re0 inet6 auto", which ifup did not like. I had to remove this line before "ifup re0" would work.

Why not...

Why not freebsd?

I have so much familiarity with the Linux userspace and with Debian packaging that I am keen to avoid learning the BSD userspace and Ports packaging if I can.

Why not Debian + zfsonlinux?

A tiny bit of play with zfsonlinux on Ubuntu 10.04 seemed to indicate that this was not yet stable. It was also unclear whether zfs root and boot partitions worked, though since this depends mostly on grub support it probably is. Finally, zfs root is not supported by the debian linux installer to the best of my knowledge.

Entering service

Mostly due to inertia, it took 10 months to transition services from the old Ubuntu 10.04 LTS system either onto my web hosting provider or to the new system.

Shortcomings in the Debian GNU/kFreeBSD userspace

There are a couple of major headaches:
Website Copyright © 2004-2014 Jeff Epler