some thoughts
January 27, 2009 at 2:04 pm
filed under home storage, thoughts
Tagged drobo, iTunes, nas, storage
I realized the other day that I have 110GB of stuff in my iTunes library. I don’t even think I have enough hard drive space left to get the last two seasons of Battlestar Galactica. I guess I should move everything to my 500GB external drive I bought last year.
Oh wait…it died. But I’ll get to that later.
And Karen is running out of space on her external drive, too. (How does she use up space so quickly? Why are Photoshop files so big?)
So it looks like it’s time to find a good solution for all of my home storage needs.
To tell the whole story, I need to go back a bit. Karen’s laptop has a 100GB internal drive. Not bad, but by no means sufficient when doing a lot of graphic design work. A few years ago, we got a Western Digital Passport drive. I really liked it. It has a great form factor–you could literally put it in your pocket. It was powered by the USB connection. It was relatively cheap. All-in-all, I thought we were getting a really good deal.
And then Karen’s files started getting corrupted. You know, the files that she uses to run her business? So much for Western Digital.
Shortly after that saga, Apple released OS 10.5. It included Time Machine, a truly elegant home backup solution that is seamlessly integrated into the desktop. It has saved me from accidental deletes on more than one occasion.
So to take advantage of Time Machine and to replace Karen’s failed Passport drive, we ordered two 500GB LaCie drives. Again, they were a pretty good deal: cheap, lots of connections on the back, lots of space. I used mine for backup and Karen used hers for backup and storage for work documents and the thousands of pictures we have of Ian.
I think you can guess where this is going.
LaCie drives have a problem with their power supplies. Wait, that’s not quite right. I think the LaCie representative on the phone said, “This is a rare problem, but since you have 2 drives, I’ll go ahead and send you the replacement power supply for the broken one and an extra so you will have a spare.” LaCie, you aren’t really giving me a lot of confidence in your product.
Karen is still using her LaCie drive (but running out of space), and it seems to be working just fine, but mine died. It seems like the power cable is fine, but some of the electronics in the case went bad. Now I don’t have any backups on my computer, and I’m running out of space.
So I’ve been in the market, as it were, for something that is a little more reliable. Since I’m looking at something new, I figured I could dream a little and think of what I would really like. What kind of things do I want in a home storage system?
Doing a lot of research online, I discovered that there aren’t a lot of good options out there. One product I looked at is Netgear’s ReadyNAS NV+. It looks like a good product, but I had two qualms about it. First, it is somewhat expensive. Without any disks, it is about $700. Second, it doesn’t have any official support for Time Machine. On the other hand, the ReadyNAS has some really nice features I like. There are a lot of people that use it and love it. It has built-in network support. People have been able to get it to work with Time Machine (although this use is unsupported). Overall, it seems like a good product; my main beef is the price.
I came across another product I liked: Drobo. Data Robotics makes a little toaster-sized box called Drobo that holds four hard drives, protects against disk failure, connects to a computer with USB or Firewire, and makes management about as complicated as using an iPod. It doesn’t have built in network support (but you can add a DroboShare for another $200), and it is a little pricey for what you get. Retail price for a Drobo without any drives is about $500. In my opinion, $300 to $400 would be a much better price for what you get. One thing that is not mentioned in the marketing material for the Drobo is the $50 annual licensing fee that lets you get firmware updates and tech support for the Drobo.
Not liking the price a whole lot, but liking the promise of hands-off storage, I ordered a Drobo and four 750GB hard drives. They were delivered, I took it home, and the tragedy of errors continued.
I spent quite a bit of time looking online about the best way to use the Drobo for Time Machine backups and storage. I connected the Drobo to my computer, thinking that I could set it as a shared drive so Karen could see it on her machine. That saves me the $200 for the DroboShare, right? Well, since Karen’s info is “mission critical” to her business, I figured I would risk my data before using her as a guinea pig. I set up a sparsebundle file on the Drobo to use for Time Machine backups. This lets me limit the amount of space used by Time Machine. I didn’t want to fill up the Drobo with backups and have no space for any media, and since the Drobo is reported to the OS as being bigger than it really is (a nice feature that allows for seamless future expansion), Time Machine, when left in an unlimited state, could try to save backups to hard drive space I don’t have. Everything seemed to be going very well. I started my initial backup of all my data (including my iTunes media), and let it churn. Hmmm….did that really just take the better part of a day to copy 180GB to the Drobo?
Finally, the backup is done. Mount the sparsebundle, browse the Time Machine backup, everything seems to be working well. Yay for the Drobo. So I moved my iTunes library over (another long wait to copy 110GB). Things are going well. I can see my data in iTunes. I can play the music. Well I guess I don’t need to keep using all the space on my hard drive, right? The data is on my Drobo, safe and secure. Move the iTunes folder to the Trash. Gulp. Empty trash. That was kinda scary, but now I’m in the brave new world of redundant data storage. Even if something goes wrong, I have a Time Machine backup of my iTunes music. I can just restore it and go on with my life.
Unless the backup is corrupted.
AUGH!!
The first sign of trouble was when Time Machine reported that it couldn’t save the backup because the target disk was read-only. “That’s weird,” I thought. Everything looks ok to me. So I eject the Drobo and plug it back in.
“This disk has errors that could not be recovered and has been made available to you with limited functionality. Please back up your data and reformat the drive.”
Errors? I looked down at the Drobo. Four soothing green lights stared back saying, “Everything’s ok here.”
A little more digging and the damage is worse than I feared. iTunes is listing all of my music, but dozens of files are reported as missing. I open the folders on the Drobo and searched for them. The folders were there, but they were all empty. Some are files that I ripped from CDs. Some are files I bought on iTunes.
A feeling of dread begins to set in.
Scared of making it worse, I didn’t touch the computer again for three days.
Having steeled myself to the task ahead, I started to look for the missing data. It’s nowhere to be found. I copied what files I had back to my internal hard drive and vowed to never, ever, delete it again. I tried to think back to what I had done with the Drobo that had caused me to loose data. Although little more than a guess, I think I have an explanation for the problem.
At some point in my first adventures with the Drobo, I restarted my computer. However, the Drobo (or something) seemed to hang and not let the computer shut down properly. Seeing that, I used the power button on my Mac to turn the computer off. Having done it many times before with no ill effects, I didn’t think anything of it. My actions seemed to offend the Drobo mightily, and the filesystem on it got corrupted. Unfortunately, this is one of the worst kinds of problems. I know immediately if a drive dies, because I can’t see it anymore. When the filesystem got corrupted, there was no warning. I didn’t know about the problem until it was too late. Although I don’t completely blame the Drobo for this error, it didn’t do anything good for our relationship. After all, isn’t the Drobo supposed to protect against data corruption errors? That’s why I bought the thing.
Several hours and $120 later, I had finished running Disk Warrior on the Drobo. Disk Warrior fixed the corruption of the Time Machine backup and I was able to restore all of my iTunes library. If your data is worth more than about $100 to you, get it. Disk Warrior is one of the best programs I have ever used.
As an aside, I want to write programs like Disk Warrior. I want to be a hero when my programs are run. Those developers have my gratitude and admiration. They have made a wonderful program, and, if they are ever in town, drinks are on me (that’s Dr Pepper for all my conservative family members reading this).
So now that I have my data back, I’m left looking at a little black box, and I can’t help but thinking of the phrase “black hole.” Do I really want to try this again, or was this just a $500 mistake? To say the least, my confidence was shaken.
I spent the rest of the week carefully reseting and using the Drobo. Things seemed to be working better the second time around, but I still wasn’t using it as any sort of primary storage. I copied (not moved) my iTunes files back over to the Drobo and found another problem. The Drobo is slow. I could not watch HD TV shows that were stored on the Drobo without any stuttering of the picture or sound.
The last strike against the Drobo was when one of my drives was reported as bad (it’s a week-old hard drive). I took out the drive, put it back in, and haven’t had any problems with it. Now I’m really unsure about what the Drobo tells me.
Overall, my opnion of the Drobo is that it is a great little box that would be good for backups, but I can’t recommend it as primary storage for data. And, unfortunately, there are other options that offer cheaper backups out there, if all you are looking for is lots of space and hard drive failure protection.
Unfortunately, with the Drobo having struck out, I’m back to square one. I have no backups for my computer. Karen is running out of space on her external drive. I’m running out of space on my hard drive.
I went back to my list of requirements and started dreaming. If I could have anything, what would I have? What I want is a Drobo that is faster and more reliable.
A few years ago, I started hearing about this wonderful new filesystem called ZFS. In a nutshell, it’s free, it offers tremendous file protection that is completely transparent to the user, and it can hold so much information that I could literally record every second of my life and not fill it up. There are several implementations of ZFS, but the official version is included in OpenSolaris, an open-source operating system made by Sun that is similar to Linux.
I’m fairly proficient at technical things, and although I don’t relish the thought of maintaining a file server at home, I know that I have the knowledge to do so. Plus, ZFS looks really easy to use. I should be able to set it up once and not worry about it. So I started looking for a computer that I could use as a home file server running ZFS. I decided that I wanted a little more customization than I get just ordering something from Dell, so I started putting pieces together at newegg.com to see what my costs would be. It turns out, you can get a really nice computer that does everything the Drobo + DroboShare does and more for about the same cost as the Drobo + DroboShare. Looking good so far.
Not wanting to order a computer without knowing for sure if ZFS would meet my needs, I decided to run a little test. Sun makes a great little (free) program called VirtualBox that lets you run different operating systems in a window on your current computer. I installed VirtualBox and made an OpenSolaris machine in it that is just like what I would put together myself. I started it up, followed some simple ZFS tutorials I found online, and quickly realized that I could do everything I want. I can have a large pool of storage for iTunes media and pictures. I can have a second large pool for Karen’s work files. I can set up volumes that look like blank hard drives to our current computers that we can use for Time Machine backups. And everything is protected from disk failure and corruption by ZFS. I think I found the greener pasture.
So this morning I mailed back my Drobo for a refund (minus a restocking fee, grrr), and placed my order for my new computer parts. Let’s hope this one works out.
ushekar
Sorry about your unpleasant experience with the Drobo. Customers have found the Drobo to be very reliable and Data Robotics is working to make it further so. We hope you will give the Drobo another try in the future. Thanks.
Carolyn
Sorry for all the trouble you had! I hope the new solution works out for you. Is this something I should investigate? I don’t know anything about my TimeMachine yet. Maybe we can talk about it this weekend.
Lydia
I can’t believe the Drobo ended up being such a failure! I was going to try to convince Jon to let me have one…. I guess I’ll just stick to LaCies for now.
John
The Drobo wasn’t a “failure” so much as it wasn’t right for my needs. I think the Drobo would be a great solution for 90% of people who just need backup-only storage. Since I have additional needs, and since I have a professional knowledge of computers, I decided that I could do better myself.
Justin A
Wow. You have thought through the whole storage thing much more than I ever have. I embarked upon my own storage journey recently and took a much cheaper and less time-intensive trek. I must say it is much simpler and less effective than yours as well.
I fixed an external esata drive (1TB) to work as a sata hard drive (3.5mb/s access) -just like an interior hard drive. I had to buy $23 worth of parts for that part. Now, my carbonite backup service (not really fast, but there for an emergency only) will backup my whole 1.75TB (if I useed it all) to a secure online server. If my drives fail, I can just download all the stuff from carbonite (it would take a few days admittedly to download that much, and probably tick off my ISP too).
This solution may not be as elegant or as quick as yours, but it is cheaper ($150 for the 1TB drive and parts, plus the $80/2yr carbonite subscription – which has gone up since I subscribed).
Dude, sorry that you’ve had to experience so many computer ills while you weren’t being paid for your time. That’s the worst!
Will
I’m glad I stumbled on your blog. While I was asleep the other night (1 week into ownership) my Drobo decided that it was going to lose all my data…or not. The Drobo Dashboard and Vista both report it as having 645GB of space used but all I can find is the default folder iTunes makes when it is first run (probably a result of running iTunes after the data disappeared).
From my working experience with it I have to say that is has been really good. I didn’t run into the slow transfer speed you noticed, but mine is connected by Firewire 400 to 800 able (I didn’t see if you mentioned how you connected). I did share mine across my network and I was able to run a Super Nintendo emulator off the Drobo across the network and watch 300.
I had an additional problem where my Drobo kept rebooting itself (apparently it prefers to be plugged directly into the wall). Oh and I also got the Green light, Red light, Green light issue, though I did use SpinRite on the “failed” drive before reinserting it. I have a help ticket into Data Robotics. Hopefully this missing data thing is just a growing pain, but Data Robotics should offer some reimbursement if investing in Disk Warrior or a Windows equivalent is required. Any suggestions for a Windows equivalent? Since my Drobo was setup with Windows I don’t dare plug it into my Macbook to use Disk Warrior.
John
I’m glad you have had good experiences with the Drobo. I think that for some use cases (like direct-connect backup storage) it is the best product on the market. In fact, I wish there were other products with the same ease of use. I was using a USB cable to connect to my Drobo. I was not aware of Firewire 800 to 400 cables.
I have nearly finished setting up my file server. I’ll post more about that process in the days to come.
Rudlaf
I kinda came along the same way. once you enter the triple-digits gigabyte realm data-integrity really becomes an issue. enterprise customers knew this a long time, but now we consumers suffer through the learning phase.
most solutions on the market have serious drawbacks.
zfs & opensolaris is the way to go. its free, runs on cheap hardware and it is pretty simple to use if you are willing to spend a little time. and last not least it offers features that are not available anywhere else.
I built an atom-based homeserver with 3 1tb drives under raid-z2 for under 500 bucks and its great. only thing I didn’t get to run yet is the twonky mediaserver in a linux container within opensolaris – need that for linn sneaky ds music player.
other than that this is a totally headache free way to have reliable backups. set and forget.
.45
I bought a Drobo last year after pulling my hair out from having constant Mac hard drive problems. Here are a couple things I learned the hard way:
1.) Disable hard drive sleep function in power options. Especially in a firewire chain; if 1 drive goes to sleep, the chain will break and the system will freeze while trying to resolve I/O errors. This will require a hard reboot.
2.) See 1. If you have a sparseimage/bundle open during an I/O conflict, your image WILL be corrupted. In my case, the corruption was not repairable by DiskWarrior.
.45
Forgot to mention:
Don’t use a sparsebundle for Time Machine. The Drobo can be partitioned using Disk Utility to limit the size of the TM volume in a natural file structure. I created one firm partition for TM, and the remaining partition is available (and expandable) to the Drobo.
John
@.45
These sound like good suggestions. The symptoms you describe sound similar to what happened to me with my Drobo. These are good things to keep in mind for anyone using external attached storage.
Steven H
Sounds like you have the same goals I have in mind currently (and the last few months). I’ve outgrown Time Machine external storage drives and need to expand. Have you considered Drobo’s recent developments (FireWire 800 / Drobo S / iSCSI / Drobo Elite?) Would you recommend still running your own server with ZFS or do you think Drobo is worth another look at this point?
Also, I just noticed that you’re a RackSpace Cloud Files guy – I <3 your work. Do what you can to get me cname support for my CDN files eh? :)
John
Honestly, what worries me most about ZFS at this point is that it is now owned by Oracle :-).
I haven’t had any problems with my file server, so I’ve had no reason to reinvestigate the world of Drobo. I think they have a good product for certain use cases. Unfortunately, it didn’t meet my needs.
Home storage solutions haven’t changed much recently, so if I did it all again today, I’d probably end up with something very similar.
Tom Tomchak
I somehow stumbled on your post while looking for solutions to fix my locked drobo, and was happy when I did. You verified that some of the problems I’m having are not specific to me.
Overall I really liked the drobo when it came out, but I’m finding more and more over time that there are limitations to it, and it’s always been a concern that it’s their own proprietary system.
The speed issue continues to be an issue on pretty much all of their units. It’s painful if you have the original one as I do.
Anyway, thanks for the post and all of the ZFS details. It’s too bad it’s not more widely adopted by all platforms. It’s hard to see a downside to it.
Tarin
Well it seems that things have not got any better in the bast year regardless what the paid marketers are posting in here.
We set up a Drobo in a basic configuration of a MacMini OSX Server (2x500Gb internal drives in RAID1) and the FireWire800 attached Drobo. It has 4×1.5Tb Western Digital drives and all lights are green.
This thing is SLOW.. it averages 20 megabytes per second writing however reading is a bit faster.
The worrying thing though is that it is silently and irrevocably corrupting the stored data. We only find out when Photoshop or InDesign suddenly don’t open a file citing corruption or missing plug-in issues. Yes I know about the Adobe best practice methods of copying to the desktop before working and this is done but it seems the damage is occurring in the “down time” between edits.
Time Machine is faithfully backing up the data in its corrupted state so that becomes useless as well.
I see this this thing as a nightmare for businesses as I have no faith in the device to store data accurately over time.
The diagnostics, and indeed any useful information on this device, are encrypted by default forcing each simple question into a lengthy email conversation where we have to email over encrypted data files and hope for a response.
We literally cannot tell what is the problem or if one even exists.
The last round of corruption got bad enough that DiskWarrior 4.2 (the latest) could not even complete reading the drive. This time around DiskWarrior simply crashes the server cold with a Kernel Panic.
Please be careful with your choice on these units as they will go quietly and take your data with them.
T