KG

So you’ve got an absolute monster of a server, what do you run on it? There are some really great options. If you are like my co-worker and prefer clicky clicky management and a familiar interface, there is always Windows Server. If you are hardcore and want the most stable and most performant option, FreeBSD is a strong contender. Linux is a great all-around performer and has very strong software support for whatever task you are trying to accomplish. These days though, you aren’t really all that limited with any of them as we can leverage virtualization.

In order to keep the natives from getting restless, Windows Server was our first choice. We could leverage the switch-independent NIC teaming that is new to WS 2012. Also available is a native hypervisor for separating all of our services and controlling network access through the vswitch. We both use Hyper-v at work and support numerous customer deployments on the platform. It’s not a bad system, but the management utilities are slightly lacking, especially for advanced features such as clustering. But, as we have only a single server, these issues would have had minimal impact for us.

Windows server installed just fine and getting things up and running was plenty easy. But after placing a load on the machine, we discovered several problems. The first issue was an absolute showstopper. The NIC aggregation would simply not work in a stable manner for our dual Intel NICs. No matter what we did in terms of driver versions or settings would solve the issue. The network could only sustain between 30 seconds and 1 hour of sustained load before it would become entirely unresponsive and the system would need a reboot to become functional again. While this was a dealbreaker, and Windows was dead to us at that point, we were curious about another feature new to Server 2012. Storage Spaces sounds like just a magically awesome technology for pooling drives and managing storage. You just use the GUI to make virtual disks on top of whatever physical disks you want. Then you just carve out partitions on top of the storage as normal and go to town…when it works. For our purposes the only configuration that made sense was going to be dual-parity. We couldn’t just throw away half our disk space going with a mirrored configuration. With dual parity, we would only lose 2 disks worth of space for data redundancy. While I had read that parity spaces perform badly, I was shocked to find out just how poorly it performed. The writes were most concerning:

80MB/s writes, under the most ideal conditions, across 24 7,200 RPM spindles is wildly unacceptable. 4k writes and re-writes were so bad you’d almost get better performance from a 3.5″ floppy drive. I was shocked!

The read performance was sawtoothing all over the place, so I took Windows Server out behind the woodshed and put it out of its misery.

The next choice was Ubuntu Server LTS. It has a nice long 5 year support window, and it’s just a nice sane Linux implementation. We would have gone Centos, but I really needed a newer kernel because I wanted to employ ZFS. I could write an entire post on why ZFS is the most amazing thing since sliced bread. I might even do that sometime, but staying on point…ZFS was lightyears ahead of Spaces. With all the disks in three separate RAID-Z1 vdevs we made a single pool. Benchmarking this pool yields 1.2+ GB (yes, gigabytes) per second for sustained uncached reads. Writes hover around the 800MB/s range. The performance is just phenomenal. On top of that we get to make use of our 48GB of RAM for read caching, which just makes things even faster. I couldn’t be happier with our disk performance.

UFW makes a really simple IPTables frontend for managing our traffic and locking down ports.

We also started using KVM to keep things sane and separated, which leads us into part III.

After renting for a number of years you eventually decide that it is more financially viable to own longterm. And whether you are talking about housing or hardware, the logic is all the same. Everyone needs housing. Everyone also obviously needs a 100TB+ monster of a server with massive Internet connectivity. Following with the analogy of home ownership, you can really customize your own server to suit your particular needs. Shopping for myself, and my co-worker, we decided that storage was most important. You can rent servers with half as much storage as we built, but it will cost you dearly.

If you aren’t worried about getting a fixer-upper, you can save even more. You can get a “turn-key” chassis secondhand on ebay and get dual sockets and redundant power. Something like the Supermicro 846 series is a good choice for that. Or you can do “new construction” and build your own in a serviceable new chassis like a Norco 4224. This allows you to splurge on only the amenities that you are looking for.

We decided to go with a 5 year old Supermicro 846 that came fully furnished with dual 6 core Xeons and 48GB of memory. For the money, it’s an absolute screamer. I mean that in several ways. The noise is really oppressive if you were going to live with it. Fortunately, we planned on shipping ours off and only visiting when it was being very naughty. As the Supermicro included IPMI for out-of-band management, our need to actually visit the datacenter is going to be very minimal.

With dual hyperthreaded Westmere generation Xeons, we have 24 logical cores to play around with. This is especially important as we both plan on utilizing Plex for media streaming. The on-the-fly video transcoding can shape bandwidth for any situation, but it needs plenty of CPU. As Plex recommends 2000 passmarks per stream, we will be “limited” to 6 transcoded 1080p streams at any one time.

The really fun part for me was provisioning storage. It’s hard to really get a good idea of what 24 hard drives are like until you really work directly with them. It took me 45 minutes just to get them slotted in their hot-swap bays. We ended up with 24 Toshiba 5TB monsters as they were 7,200 RPM and had an excellent price at the time of purchase. Each of these things has a 128MB cache, so we have 3GB of caching before we even get to the controllers or OS.

We needed something to drive the drives, so to speak. The Supermicro allowed us to utilize a backplane with a SAS expander so that we had the option of feeding it a single SAS 8087 cable and run everything that way. Unfortunately, we would be limited to hardware RAID and have limited drive monitoring and error recovery options within the OS. The decision was made to utilize three IBM M1015 HBA’s. These are quite cheap on eBay and three all together have enough ports to give a dedicated 300MB/s lane to each hard disk. Flashed to IT firmware, they are perfect for next-generation RAID.

Overall I’m very happy with the hardware that we have, considering that our savings over something like a Dell or HP are enough to purchase a car.