NAS

There is a saying that goes: “There are two types of people. Those who have yet to lose data, and those that backup.” Proper backup can be done a number of ways. Options include internal hard disk, external hard disk, internal network, external network, and tape. The external network option has become more popular over the years as WAN pipes get larger and storage gets cheaper. Excellent companies like Box.net and Backblaze are happy to sell you subscriptions for storing your data with them. If you have a reasonably small data set, these services are probably the way to go. Your files will be offsite, which protects you from catastrophic failures such as equipment loss / theft and equipment damage / tornadoes. But for people like me with a bit more data to store…

zomg

I can’t really maintain a 26 Terabyte dataset off-site. Backblaze would technically allow it, as they advertise unlimited storage. I’m pretty sure Comcast wouldn’t like it if I began uploading and didn’t stop for nearly a year… uploading   My only real solution is local network storage. I’ve been using network storage for more than ten years now. I started with 320GB disks on Debian Linux with MDADM. Then it was 750GB disks organized in the same way. Then another rebuild with 10 1.5TB disks using Ubuntu Server. I still have the 10 disk, 13.6TB array going. I have migrated it to BTRFS in order to take advantage of its extensive next-gen feature set. In particular, I would like to avoid the URE issue that can be present when rebuilding the array. I lost a whole array of data to this problem in 2011, and plan to never allow it to happen again. Since the next gen file-systems store bitsumming for both file and metadata, the integrity of the array is much stronger. Errors can be detected and corrected without downtime. The majority of my data lives on a 12 disk ZFS on FreeBSD array. The spindles are Seagate Barracudas at 3TB a piece. To date I have had one drive that was DOA and another that failed after two years in continuous operation. The wiring is a bit monstrous, but with so much hardware packed into a standard chassis there is no way around it.

20140412_031343

I’ll post a follow up going into more detail about the filesystem, the hardware I have used, and the performance aspects of the system.

kyle