In late 2012, I purchased a new system with the intent of migrating my home storage (primarily several hundred GB of photos and a few dozen GB of email) to it. I felt it would be desirable to boot from a mirrored ZFS root and store my photos on a raidz1 pool.
The Hardware
- AMD Phenom II X4 965 (4 cores @ 3.4GHz)
- BIOSTAR TA970GX motherboard (UEFI and BIOS booting)
- 32GB DDR1333 RAM
- 3 * 3TB SATA hard drives (ST3000DM001)
- Defective, replaced by 3 * 3TB SATA hard drives (WD RED WDC WD30EFRX)
- Antec Sonata Proto
- 530W power supply
The Software
Debian "Wheezy" kfreebsd-amd64The hiccoughs
I've run into a number of bumps on the way to a working system.Bad drives, bad service from newegg
One of the three drives did not spin up. A second drive had an IN_THE_PAST SMART failure attribute, and the last was just 1 point away from the failure threshold in the same attribute. I RMA'd the first drive, which newegg refused to replace claiming that I had damaged it. Meanwhile, their page of reviews for the drive had 3 or 4 reports similar to my experience—a very likely sign of a damaged batch of disks. When I called this to their attention and asked that they replace all three of my drives, they refused. I disputed the charge with my credit card company and eventually got back the full cost of the drives, but newegg inactivated my account. I'll be shopping elsewhere from now on.
Booting the Debian installer via PXE
On the server side, "tftpd 0.17-17ubuntu1" doesn't seem to work, while "tftpd-hpa 5.0-11ubuntu2" does. The symptom is that grub can't find grub.cfg.
Unless you insert 'pxe_unload' after the last kfreebsd_module line in grub.cfg, the network is "slightly wonky" in the installer—more specifically, it appears that all packets to the IP address of the TFTP server are gobbled up and userspace never sees them. The symptom is that the installer can't bring up the ethernet connection.
Once these details are worked out, it is very handy to start the debian installer without any media.
By the way, unless your internet connection is very fat, or you are sure you'll get all the installer choices right the first time, you're almost certain to want some kind of HTTP proxy that will cache the stuff from the debian package servers. I use apt-cacher-ng operating on http://10.x.x.x:3142/ and manually enter this setting in the installer.
Advanced Format drives and "ashift=12"
I've read on the internet and heard from a colleague that it's important to set up a zfs pool to use the disks' physical (as opposed to logical) block size. Unfortunately, the Debian installer offers no way to do this, and the "gnop" program used as a common workaround isn't available in the debian installer. After considering a number of alternatives, I elected to ignore the need for "ashift=12".
With a few hundred gigs of data on the ashift=12 raidz pool, "zpool scrub" proceeds at a rate of 250-300MB/s. Here are the results of a bonnie++ run:
| Version 1.96 | Sequential Output | Sequential Input | Random Seeks | Sequential Create | Random Create | |||||||||||||||||||||
| Size | Per Char | Block | Rewrite | Per Char | Block | Num Files | Create | Read | Delete | Create | Read | Delete | ||||||||||||||
| K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | |||
| zaphod | 64G | 152 | 98 | 208719 | 43 | 121629 | 24 | 449 | 99 | 251042 | 23 | 167.2 | 6 | 16 | 15189 | 51 | +++++ | +++ | 27149 | 94 | 23779 | 81 | +++++ | +++ | 28263 | 95 |
| Latency | 338ms | 2134ms | 2796ms | 65508us | 1195ms | 1727ms | Latency | 9181us | 2198us | 2329us | 22052us | 81us | 66us | |||||||||||||
Parted 'Could not get identity of device'
You have to ignore this warning every time. I should look for a bug about it on bugs.debian.org. This may have to do with the fact that /dev/adaX are character devices in kfreebsd; I'm not sure.
Booting
I am not an expert on UEFI vs BIOS booting or MBR vs GPT partitioning, but this was the biggest bump I ran into.
For disks larger than 2TiB (~2.2TB), the old-fashioned MBR partition table cannot define partitions that span the whole disk. As a consequence, Debian rightly selects GPT partitioning for the disks. However, in this circumstance it's necessary to have a "bios_grub"-flagged partition for bootloader data. If you do not have this, the symptom is that the bootloader hangs after printing "GRUB". It may be possible to create this partition in the installer, but I did it after the fact from a rescue shell using parted's "set <N> bios_grub on".
After this, my system still did not boot, presenting me with a "GRUB rescue" prompt. I ultimately determined that the problem was now that the right modules were not built into the grub core.img. Unfortunately, I did not keep the commandline that I used to fix this issue. I know it involved "grub-mkimage" and specifying to include the "part_gpt" and "zfs" modules. After that, I did "grub-install" to each disk.
I also used the program gptsync to try to synchronize the MBR partitions with the GPT partitions. However, I don't think that this was important.
I still have not tested that the system will still boot with any one drive disconnected, but I expect that it will.
As far as I can tell, I am doing BIOS (not GPT) booting, as I don't have any partitions of type "gpt_boot", so I don't have to tangle with UEFI and everything that entails.
Networking
My /etc/network/interfaces was generated with the line "iface re0 inet6 auto", which ifup did not like. I had to remove this line before "ifup re0" would work.
Why not...
Why not freebsd?
I have so much familiarity with the Linux userspace and with Debian packaging that I am keen to avoid learning the BSD userspace and Ports packaging if I can.Why not Debian + zfsonlinux?
A tiny bit of play with zfsonlinux on Ubuntu 10.04 seemed to indicate that this was not yet stable. It was also unclear whether zfs root and boot partitions worked, though since this depends mostly on grub support it probably is. Finally, zfs root is not supported by the debian linux installer to the best of my knowledge.Entering service
Mostly due to inertia, it took 10 months to transition services from the old Ubuntu 10.04 LTS system either onto my web hosting provider or to the new system.Shortcomings in the Debian GNU/kFreeBSD userspace
There are a couple of major headaches:- Many python + gtk programs crash in pthread_mutex_unlock (this may be better in jessie, but I'm on stable)
- Valgrind and gdb need patches that I have contributed but which haven't made it into official Debian packages yet.
- There's no accelerated virtualization solution
- Occasionally, random-seeming packages are missing from the debian archive for kfreebsd-amd64 (FTBFS)
Website Copyright © 2004-2024 Jeff Epler