This post has been a few weekends in the making, but I’m here today with a clean deduplicating ZFS file system in the lab. I started the project with a relatively basic goal, replace my current Debian Linux backup server with something that will maximize disk utilization and be relatively simple to maintain. The old Debian box had been configured to execute a series of scripts that would take virtual machine files stored on a VMware Server box and 7zip them down to its own local disks. This configuration served me well while I was running VMware Server in the lab. Today the lab looks a bit different and it is running ESXi, and the Debian backup server is not backing up any longer. Tired of running without a backup of the lab machines, I came up with a brilliant idea. ZFS is easy to use right? I’ve got some basic Linux experience, surely Solaris couldn’t be too different. Everyone and their brother in the press has talked about ZFS’s new found dedup features, if they churned out a server to review in short order surely I could do the same. So with this, I was off, ZFS with dedup or bust.
I started by skipping everyone else’s blogs, notes, and reviews and went straight to downloading Solaris Express Community Edition (SXCE). Although not an outright mistake, this was a step in the wrong direction. I later found that SXCE was soon to be EOL’ed and that further development efforts were going to put forth towards OpenSolaris. Oh well, another ISO to download and another installer to chug through.
With OpenSolaris 2009.06 downloaded I was ready to set forth and dedup. Again, this was to be my own experience so I skipped everyone else’s how-to’s and jumped right into the live CD. From within the Gnome interface on the live CD there is a desktop icon that indicates it will install the OS to your hard disks. Perfect! Click. As I worked my way through the installer I was thinking “Where’s the advanced buttons on each page”…..there were none. Time Zone, Check. Whole disk or partition, check. Root and user passwords, check. Your “done”. It is about this time you should know a little bit about my hardware.
- Intel SE7210TP1-E System board
- 2x Western Digital 750GB SATA Drives
- 4GB RAM
- 3Ghz Pentium 4
The idea here was to install the OS to a 10GB partition and have that partition participate in a pool that would be mirrored to the second drive. While I’m sure this is possible, I wasn’t able to figure it out. I tried a dozen different attempts without success, but I’m getting a bit ahead of myself. You see, I don’t like working at the console. The chair is less comfortable, and it feels like I’m using a 13 year old monitor (I am). I wanted a static IP on my newly installed server so that I could reliably SSH over to the machine and destroy it from afar.
My initial attempts involved poking around the Gnome network manager. After a few desktop lockups I was finally able to get the applet to accept the static address, but the system itself would never apply the settings. No biggie, Ubuntu has done this with me a few times and its an easy fix. I set off to pillage the etc directory to uncover the bits responsible for retaining the IP addressing. Nothing jumped out at me. After cat’ing and nano’ing a heap of unhelpful files I retired to the user manual (Google). I found a cool guy by the name of Malachi (makes me think of the NOFX song), who has a nice writeup here on how to set a static address on OpenSolaris. After reading his excellent post, I jumped over to the console and executed about 50% of what he had written. After a reboot, I was wondering. “Why is my server booting directly to a bash prompt, and why can’t I ping it?”. Let’s just say that you need to follow Malachi’s instructions to ensure your server is bootable. Thank you Malachi.
OK, so we’re back to getting the rest of the disk laid out for a main data partition that will be dedup enabled, right? 10GB system pool (called rpool in Solaris), and a bunch of empty disk remaining if you’re keeping score. I insist my server will be configured this way even after reading things like, “Using whole physical disks is the simplest way to create ZFS storage pools. ZFS configurations become progressively more complex, from management, reliability, and performance perspectives, when you build pools from disk slices, LUNs in hardware RAID arrays, or volumes presented by software-based volume managers” (ZFS Administration Guide). Try as I might, I couldn’t make it work. Maybe one day after I’ve built up my ZFS ninja skills I’ll be able to implement the setup, just not today. It is with this concession I set forth and install the OS again for the 12th time with the intent to use one of the entire disks for installation. Surely it would be easier than futzing with the partition headaches. It was.
All along I had been working with DarkStar’s instructions on how to mirror an rpool ZFS pool to a second disk for a mirrored system drive configuration. Unfortunately I wasn’t specifically adhering to his guide because I’d been goofing with the partitioned drive business. With new confidence that using the entire drive for the system install would take care of my problems I followed DarkStar’s instructions to the letter. I made it through step four, which is much further along than I had made it in previous attempts. When I arrived at step 5 to setup GRUB on the mirror disk I was greeted with. “Partition 0 of the disk has an incorrect offset”. Uh oh. Surely I’ve not chased a rabbit here. Hopefully his instructions still applied to my new version. I didn’t know. So, back to the user manual (Google).
Malachi’s back. He says in a write up similar to DarkStar’s that, “…..before doing the fmthard, you have to do fdisk on the new drive. In my case, I did ‘pfexec fdisk /dev/rdsk/c4t1d0s2′, it complained, I said yes then everything was fine. If you skip this step, you’ll get: fmthard: Partition 2 specifies the full disk and is not equal and later: ‘Partition 0 of the disk has an incorrect offset.’” This is what happened on my configuration as well. Once I went back and ran fdisk on the second partition of the second drive DarkStar’s instructions worked. So I’ve finally got 2009.06 installed, I’ve got my mirrored ZFS pools in place, and I’m ready to dedup. Jeff Bonwick’s blog says it’s simple, which is good because I’m ready to get to the simple part.
I start by executing the commands that Jeff mentions in his post. He notes to turn dedup on for a storage pool to execute the following: zfs set dedup=on poolname. I did and it resulted in, “cannot set property for ‘rpool’: invalid property ‘dedup’”. What tha, I thought I was ready to go. After some further searching, I found out that the dedup code has only been recently committed and that I would have to tap into the into the development releases to get the dedup functionality. Peter Galvin has as nice writeup here on how to take your 2009.06 system up to the latest development release, and obtain the dedup functionality. So before going to bed one evening, I kicked off the system upgrade and hoped this would be my last hurdle.
Like a kid on Christmas day I woke the next morning with the hope that the dedup elves had left a gift on my drives. They left a gift alright. Gnome was booting with errors now, but I figured this was no biggie. This is a development release after all, and I didn’t need any stinky gui. So I pulled up the comfortable office chair of my main machine and SSH’ed into the newly upgraded box. After a disturbingly long wait for a prompt I entered my credentials and was greeted with the release info and nothing else. No cursor, no prompt, no nothing. I beat on things for a bit, rebooted a half dozen times, and scratched my head a bunch. None of this made me feel at ease with the state of my server. I didn’t bother researching how to fix the problems because 1) I wasn’t pursuing this to test the OpenSolaris upgrade experience, and 2) Because I stumbled across http://genunix.org/ . I don’t know exactly what Genunix’s intended purpose is just yet, but I do know they maintain current builds of the latest Solaris operating system forks. I found the latest OpenSolaris build 131 here in ISO form. My Window’s background has always told me that a clean install is better than an upgrade, so I decided to apply that philosophy to this situation. I pulled down the latest release and went about the installation.
It is at this point all the cards start to fall into place. I have build 131 in place that seems relatively error free, I know how to properly configure static IP’s, and I know how to build a mirrored ZFS rpool disk configuration. The last step is to try those dedup statements mentioned in Jeff’s post. I did, and they worked. It has been a difficult road, but I’m better now for having traveled it. I’ve got a little OpenSolaris experience now, and I’ve got a backup box to play around with in the lab. Thanks to everyone mentioned in the post, I could have done myself a favour and hounded you and the forums for the answers, but I tend to learn these things by doing them the hard way.
3 Responses to Doing it the hard way – ZFS, Dedup, and OpenSolaris