Opus

The question of audio in any computing environment is one that is going to be rife with complexity, license concerns, and a religious sense of maintaining the status quo of various older formats. Storing audio for mobile usually means two things. Firstly, you are likely going to want smaller files in order to pack more runtime into a device with a limited storage capacity. The second thing to consider is what formats the device you plan on using is capable of playing. For most mobile applications, the use of the old standby Mp3 is probably going to be seen as it has great inertia and subsequently is supported everywhere. When one isn’t constrained by mobile device requirements things can get much more interesting. Without storage requirements, a regular computer user is free to use a lossless codec. With 3TB hard disks in the $100 range, there is little reason to archive music in a lossy format. This leaves us with three general options that work for different use cases.

The first option would be totally uncompressed PCM, which is how music is stored on audio CD’s. While this is going to get you a bit for bit copy of the exact data on an audio CD, there is really no reason not to employ some space saving. This option is objectively inferior for this reason.

Lossless codecs are able to save space with zero reduction in quality, which makes them strong contenders if the goal is to archive music. Both FLAC and Apple Lossless are good options, though FLAC is more popular and arguably superior in terms of licensing. These codecs will squeeze some space savings out of the audio without sacrificing quality in any way. For a desktop application with modern hardware, lossless codecs are going to be a superior option.

While true audiophiles would sneer at the thought of anything less than a lossless copy, some of the more modern lossy codecs are truly impressive pieces of software engineering. Mp3 became a THE lossy standard because it didn’t have much strong competition way back in 1995 when it was first released. Mp3 at high bitrates compresses well and is nearly indistinguishable from a recording source without the use of extremely high end audio equipment. It has hung around for so long because it is “good enough” and has a great deal of market inertia behind it now. These days, there are simply better options. Ogg vorbis and Opus are two open formats that both deliver better audio at comparable bitrates when compared to Mp3. These formats are not patent encumbered, which make them solid options for commercial applications. Vorbis came first and was designed as a higher latency codec for music and multimedia audio applications. Vorbis has a large bitrate range (betweek 45kbit and 500 kbit per second) and is simply much stronger than Mp3 across the entire range. For telephony applications, for example, lower bitrates are perfectly capable of transmitting voice audio which requires less fidelity than a music audio recording. The IETF saw how well Vorbis did and thought that they could make it better. Opus was born. Opus has a substantially wider bitrate range than even Vorbis. Opus can limbo all the way down to 8kbit/s. Combined with a much lower algorithmic delay, Opus is ideal for Voice Over IP and other interactive mediums. What is even more impressive is that, when the bitrate is cranked up, it is objectively superior to alternative formats basically all the way up the bitrate scale.

1010px-Opus_quality_comparison_colorblind_compatible.svg

I did some testing myself with my computer. While I do not have any truly high end audio equipment, my Logitech G930 has a reasonable level of audio fidelity and likely represents a level of audio fidelity slightly above the median across computer users. I acquired an audio source from a great 80’s band and converted the lossless FLAC files to Opus at various bitrates starting with 8kbit/s. I selected “Alone” which is one of my favorite tracks and has a high degree of range between quiet sections and blistering full-range instruments and vocals at once. The 8kbit sample was noticeably degraded in quality. There was obvious background hiss and muddled audio in general. It was clear what the song was, but any sort of fine fidelity just wasn’t happening. At 16kbit, I could very clearly make out all vocals but there was still a slight background noise issue on higher volume. 24kbit was where things began to get interesting. At this bitrate, I had a bit of difficulty noticing any quality degradation until the loudest sections came in. No background noise issues could be detected at any time. I didn’t do any testing beyond 32kbit because I simply couldn’t notice the difference between it and the source. When using random A/B testing, I was able to guess the 32kbit encode correctly only slightly better than average. This 32kbit sample came in at a whopping 865KB in size. Truly remarkable when compared against the more than 24MB source file. Below is a chart comparing the size of the various encodes I did against the source filesize.

bitrate

From this angle, that looks pretty impressive to me.

Mosh

Ever since 1995 SSH has been a stable workhorse for anyone dealing with Linux and Unix hosts and even Windows. There is good reason for this. SSH has a number of powerful features that let you exploit its capabilities. You can, for instance, set up a quick and dirty VPN of sorts for forwarding your traffic over a secure tunnel to another machine. SSH can also be used to transfer files, which is enormously useful in itself, but other programs like rsync add onto this functionality and make syncing directories, for instance, be nearly magical. All this while maintaining the security of default AES ciphers over what just might be the most secure network protocol in wide use.

For all its great strengths, SSH does have a few weaknesses. Computer scientists have solved a few of them by adding on to the protocol. The HPN-SSH additions solved some buffer and SMP problems that were holding back SSH from its true potential in terms of transferring files. More recently, MOSH has come along to solve the issues pertaining to interactivity performance and roaming capabilities. The vagaries of modern life mean that we often have to make do with administering machines over wireless or unstable connections. With old-school SSH, any sort of interference in the line would mean you would have to re-establish your connection. This is a particular annoyance for me on my laptop, for example. Suspending and resuming interrupts the network connection, necessitating a handshake to re-establish my broken pipe. If I connect with MOSH, the terminal will simply try to re-establish a connection on its own, firing off UDP packets that it doesn’t need to keep track of in terms of the acks that TCP would traditionally necessitate. It simply waits for the latest data to paint the terminal. Other neat features include its increased interactivity. Managing my European machines from the Western US used to have noticeable lag. While we can’t change the speed of light, MOSH does a great job of making my terminal sessions much more responsive to input. Also, interrupt sequences are non-blocking. This means that you can make a mistake and, for instance, cat out a huge file. At any time you can Ctrl^c your way out of it. Without MOSH, you may have been stuck waiting for that scrolling output to finish. MOSH was developed by some awesome M.I.T. researchers. You can watch them talk about MOSH below:

If you find yourself re-connecting your SSH sessions all too often, you might want to check it out.

NAS

There is a saying that goes: “There are two types of people. Those who have yet to lose data, and those that backup.” Proper backup can be done a number of ways. Options include internal hard disk, external hard disk, internal network, external network, and tape. The external network option has become more popular over the years as WAN pipes get larger and storage gets cheaper. Excellent companies like Box.net and Backblaze are happy to sell you subscriptions for storing your data with them. If you have a reasonably small data set, these services are probably the way to go. Your files will be offsite, which protects you from catastrophic failures such as equipment loss / theft and equipment damage / tornadoes. But for people like me with a bit more data to store…

zomg

I can’t really maintain a 26 Terabyte dataset off-site. Backblaze would technically allow it, as they advertise unlimited storage. I’m pretty sure Comcast wouldn’t like it if I began uploading and didn’t stop for nearly a year… uploading   My only real solution is local network storage. I’ve been using network storage for more than ten years now. I started with 320GB disks on Debian Linux with MDADM. Then it was 750GB disks organized in the same way. Then another rebuild with 10 1.5TB disks using Ubuntu Server. I still have the 10 disk, 13.6TB array going. I have migrated it to BTRFS in order to take advantage of its extensive next-gen feature set. In particular, I would like to avoid the URE issue that can be present when rebuilding the array. I lost a whole array of data to this problem in 2011, and plan to never allow it to happen again. Since the next gen file-systems store bitsumming for both file and metadata, the integrity of the array is much stronger. Errors can be detected and corrected without downtime. The majority of my data lives on a 12 disk ZFS on FreeBSD array. The spindles are Seagate Barracudas at 3TB a piece. To date I have had one drive that was DOA and another that failed after two years in continuous operation. The wiring is a bit monstrous, but with so much hardware packed into a standard chassis there is no way around it.

20140412_031343

I’ll post a follow up going into more detail about the filesystem, the hardware I have used, and the performance aspects of the system.

Experimenting with 10Gbit

In relation to a storage series which I plan on posting soon, I was scouring the Internet searching for ways of improving network transfer without selling kidneys. One very cool option is to implement a point-to-point Infiniband architecture between servers. Since I have a pair of storage servers that would benefit from >1Gbit/s speeds, this sounded ideal. I got the idea from a poster called “Flain.” Using a pair of Mellanox cards and some short CX4 cabling, it is possible to get 10Gbit between machines for around $100.

Some things to keep in mind should you be interested. First, this is for crazy people. Home users have almost no use for anything beyond 1Gbit networking. Secondly, few systems would even be capable of handling data at beyond 1Gbit speeds. GigE has a theoretical maximum throughput of 125MB/s. This is roughly what modern 7,200 RPM HDDs can push for sustained transfer. High end SSD users, on the other hand, have hardware more than capable of maxing out GigE. Since my use case involves shuffling data between two very fast disk arrays, I have to conclude that I NEED 10Gbit 🙂