KG – Page 3 – A Net/Sys Admin Blog

Our Colo adventure Pt I – The Hardware

Posted on October 4, 2015 by kyle

After renting for a number of years you eventually decide that it is more financially viable to own longterm. And whether you are talking about housing or hardware, the logic is all the same. Everyone needs housing. Everyone also obviously needs a 100TB+ monster of a server with massive Internet connectivity. Following with the analogy of home ownership, you can really customize your own server to suit your particular needs. Shopping for myself, and my co-worker, we decided that storage was most important. You can rent servers with half as much storage as we built, but it will cost you dearly.

If you aren’t worried about getting a fixer-upper, you can save even more. You can get a “turn-key” chassis secondhand on ebay and get dual sockets and redundant power. Something like the Supermicro 846 series is a good choice for that. Or you can do “new construction” and build your own in a serviceable new chassis like a Norco 4224. This allows you to splurge on only the amenities that you are looking for.

We decided to go with a 5 year old Supermicro 846 that came fully furnished with dual 6 core Xeons and 48GB of memory. For the money, it’s an absolute screamer. I mean that in several ways. The noise is really oppressive if you were going to live with it. Fortunately, we planned on shipping ours off and only visiting when it was being very naughty. As the Supermicro included IPMI for out-of-band management, our need to actually visit the datacenter is going to be very minimal.

With dual hyperthreaded Westmere generation Xeons, we have 24 logical cores to play around with. This is especially important as we both plan on utilizing Plex for media streaming. The on-the-fly video transcoding can shape bandwidth for any situation, but it needs plenty of CPU. As Plex recommends 2000 passmarks per stream, we will be “limited” to 6 transcoded 1080p streams at any one time.

The really fun part for me was provisioning storage. It’s hard to really get a good idea of what 24 hard drives are like until you really work directly with them. It took me 45 minutes just to get them slotted in their hot-swap bays. We ended up with 24 Toshiba 5TB monsters as they were 7,200 RPM and had an excellent price at the time of purchase. Each of these things has a 128MB cache, so we have 3GB of caching before we even get to the controllers or OS.

We needed something to drive the drives, so to speak. The Supermicro allowed us to utilize a backplane with a SAS expander so that we had the option of feeding it a single SAS 8087 cable and run everything that way. Unfortunately, we would be limited to hardware RAID and have limited drive monitoring and error recovery options within the OS. The decision was made to utilize three IBM M1015 HBA’s. These are quite cheap on eBay and three all together have enough ports to give a dedicated 300MB/s lane to each hard disk. Flashed to IT firmware, they are perfect for next-generation RAID.

Overall I’m very happy with the hardware that we have, considering that our savings over something like a Dell or HP are enough to purchase a car.

Look, up in the sky!

Posted on October 3, 2015 by kyle

One of the modern industry terms that never fails to get on my nerves is “the cloud.” And I can understand that, for many people, uploading their images to iCloud is just sort of a magical fluffy concept that looks plenty like this. The term actually developed from a common industry shorthand used to describe The Internet. In countless non-nondescript and most likely poorly lit offices some shabby network engineer was, at one point, tasked with creating a network diagram. While that engineer could succinctly describe all devices that he or she has control over, the Internet just defies pictographic description. It’s not really a word you would want to draw in a game of Pictionary. It’s so obviously complex that reducing it to a single object is hilarious.

Eventually all those network diagrams just included a sort of amorphous blob. This is the magic box of packet pushing into which you dump things and those things are (mostly) recreated out on the other side.

Arthur Clarke once stated that “Any significantly advanced technology is indistinguishable from magic.” The closer you get to the actual machinery that runs the magic, though, the more it starts to look like machinery. It’s power hungry, expensive, loud, and hot. The enormous amount of work that it takes to keep everything running is just sort of dismissed in a hand-wavey manner by calling this amazing thing “the cloud.” If every packet plumber took a lunch at the same time the entire operation would be a trainwreck without the added benefit of cool aerial photos of the carnage.

I’m not the only one that takes offense. A very popular extension makes the web a more amusing place. It’s much more amusing to read about people sending their very large RAW images up to their Cloud now.

So support your local network engineers. Without them, we wouldn’t be able to secretly hate but simultaneously be jealous of Rich kids of Instagram. If you are thinking there must be a German word for that feeling, I’m sure you can pull it out of the cloud.

FreeBSD as a Hypervisor

Posted on May 24, 2015 by kyle

Since the most powerful machine that I own is my storage server, I thought it prudent to leverage its capabilities by running alternate operating systems to provide the services that I use. FreeBSD is awesome in that it is a coherent system coded by a passionate community and refined by decades of production service. Since I have 96TB of raw disk hanging off of the machine it was imperative that I use what is objectively the best filesystem currently in existence: ZFS. While ZFS runs on Linux now and is mostly nice and stable, I wanted the most tested and most performant platform for ZFS. Short of utilizing the slightly crazy and far less supported OpenIndiana, FreeBSD is the best choice. All of the storage functionality and exporting of shares is fairly straightforward under FreeBSD. What wasn’t quite so well documented was the process of getting virtual machines to run. There are some options:

The first option for virtualization on FreeBSD is to not use FreeBSD. VMWare will allow you to use their hypervisor and run FreeBSD as a guest on top. You then pass through the disks directly as long as you have VT-D capability in your environment. I’ve heard of examples of this working just fine. I’ve also heard of it causing problems that were difficult to diagnose. It’s really up in the air depending on what hardware you are using. In the end I decided against this approach mostly because I am a freedom loving American and I like my software to be as free as I am.

Your second option actually uses FreeBSD as the hypervisor. It’s called bhyve but pronounced “beehive.” It’s eventually going to be to BSD as KVM is to Linux. Eventually. It’s still under development and just wasn’t quite ready when I was deploying my server. This will someday be the best option, but for now there is Virtualbox.

Virtualbox is a type 2 hypervisor that runs on just about everything. And while I would never even consider using it for any sort of serious business, in my experience it has been plenty good enough to use in a homelab setting. So how the heck do we get VMs spun up on a headless FreeBSD host?

Install virtualbox using the fancy new package management system:

pkg install virtualbox-ose

Before we begin creating the VM we need to set up FreeBSD to load the necessary kernel modules. There are three. Load them:

kldload vboxdrv
kldload vboxnetflt
kldload vboxnetadp

Make sure they load on startup every time by adding them to /etc/rc.conf

vboxdrv_load="YES"
vboxnet_enable="YES"
vboxheadless_enable="YES"

Now we create a VM and name it. This also registers the VM with Virtualbox. You can find your particular ostype by running VBoxManage list ostypes.

VBoxManage createvm --name w8 --ostype Windows81 --register

Then we need to define some attributes of the VM. The memory specification, hardware type, and networking setup. If you want the NIC for the VM to just show up as its own discreet NIC on the network, bridged mode is what you are looking for. My NIC is named “igb1.” You can find the one you need to use with ifconfig.

VBoxManage modifyvm w8 --memory 8048 --ioapic on --cpus 8 --chipset ich9 --nic1 bridged --nictype1 82540EM --bridgeadapter1 igb1

Define the storage in MegaBytes.

VBoxManage createhd --filename /path/to/store/HDDcontainer/w8.vdi --size 50000

Create a storage controller to attach it to.

VBoxManage storagectl w8 --name "SATA Controller" --add sata --controller IntelAhci --portcount 4

Attach it.

VBoxManage storageattach "w8" --storagectl "SATA Controller" --port 0 --device 0 --type hdd --medium /path/to/store/HDDcontainer/w8.vdi

We need to define an optical disk emulator to mount our ISO so that we can install to our VM.

VBoxManage storagectl w8 --name "IDE Controller" --add ide --controller PIIX4

Attach our .iso

VBoxManage storageattach w8 --storagectl "IDE Controller" --port 1 --device 0 --type dvddrive --medium /path/to/iso/file.iso

Start the VM. The option --vrde on starts a VNC session on the IP of the HOST NIC on port 3389 by default. You will need a VNC viewer to see the console output and install from the mounted media. I used TightVNC, which worked just fine.

VBoxHeadless --startvm w8 --vrde on

After you are done installing set the VM to boot from the “HDD” by default to prevent boot device confusion. If you fail to do this it will just boot up from the install media again and again.

VBoxManage modifyvm w8 --boot1 disk

It’s best to remove the VNC console after everything is set up the way you want it. Just enable your remote access option of choice from within the VM OS for management.

VBoxHeadless --startvm w8 --vrde off

One last thing to keep in mind is the NIC type that you assign to the VM. You are going to get the best performance by utilizing the virtio driver maintained by Red Hat. The package for the driver is available here. If you are installing another BSD or Linux VM then the driver is already there. If you installing a Windows guest then I found that it is best to use the Intel NIC from above and then download the driver .iso. Then change the --nictype1 to virtio. After booting use the VNC console to assign the driver to your new NIC. This allows for the best performance and, from what I have experienced, improved system stability.

Subsonic

Posted on April 20, 2015 by kyle

The question of “How do I access my music collection over the Internet” has a few good solutions. I previously talked about Ampache, which is quite good but gave me some trouble due to the extreme size of my media library. Ampache had to be tweaked to allow an unlimited execution time and higher memory limits in php.ini. While that is easy enough, it never finished cataloging all of my music. While it completed most of it and I was mostly happy with it I am not a “mostly” kind of guy. The hunt was on for something better.

Subsonic is a standalone media streaming application focused around music and built on Java. It has almost total feature overlap with Ampache in that they both have cover art display, tag editing, multiple user support, transcoding, and more. After the library import, they both use around 550MB of active memory with my library. Unlike Ampache, subsonic has a smartphone app that you can use to stream to your mobile device.

The install for subsonic is much easier and has far fewer steps, as the HTTP server and all other necessary software is included in the RPM or DEB. I made an Ubuntu Server VM and, after updating, I just had to navigate to the download page and choose the correct option for the OS. In this case a DEB. SSH into the VM and wget the download link to get it on the local host. Then it is just a matter of dpkg -i subsonic.deb. The install is complete at this point. It’s worth noting though that, unless you wrangle with it, it wants to run as root by default. So it’s best to carve out a whole VM or physical machine dedicated to just subsonic. One more thing to do before you are done configuring the VM is to attach your storage. You might want to dump your music to the VM locally, assuming you made a big enough disk. Since I already have a storage server I just exported an NFS share and mounted it to /mnt in the VM.

After the install your server will be running on that host at port 4040. After pulling up that location in the browser you get a glimpse of the web interface:

After you have changed the password and edited the media folders to access your media you will want to initiate a scan under “Settings” —> “Media folders.” When this is complete your media has been added to the database…but where the heck is it? It took me 20 minutes on the forums to figure this one out. Apparently a newly added option is restricting certain media folders to certain users. So go here:

Check the box corresponding to your media folder name and you are off to the races.

I’ve found the search function to work extremely well, and playlists are easy to make. There is even Last.fm scrobbling if you are into the whole “social music” thing.