KG

For the longest time /etc/resolv.conf has been a very simple little file telling Linux where to go for DNS resolution. These days it’s quite the popular file. System daemons old and new are lining up to have their way with what should be a very simple little config. Leading to memes like this one:

While some will symlink to their own file, others want to set up a local caching resolver and point to that. The problem with most of these is that roaming between networks will break your config should you have a mobile device. And local resolution won’t work if you are using a local resolver like systemd-resolved wants to do. It’s not exactly difficult to get into a situation where more than one is installed in which case they fight to wrest control of the single config file. So how do we unfuck this mess? Let’s look at Ubuntu, which now ships with systemd-resolved.

By default we can see that systemd-resolved has a symlink in place. It’s set up a local resolver that won’t respect your upstream DNS that should be assigned by your DHCP server.

So…let’s kill that.

Now that Lennart Poettering has been sufficiently triggered we can tell NetworkManager to do its job.

The old symlink is still in place, so that needs to be removed. A restart of the NetworkManager service writes out a new resolv.conf with the actually assigned DHCP DNS nameservers.

It only took five minutes to fix something that shouldn’t have been broken in the first place. 🤦

Ubuntu has been in the distro news lately with it’s nascent release of 20.04. Ubuntu has long been touted as the beginner friendly distro. And while there have been some historical gaffes and strange decisions I’ve never been one to actively discourage the use of Ubuntu for those that were interested in it. I never even dinged them for basically doing nothing that Debian didn’t do already. Every one of their releases is basically a pull of Debian testing to fit a time window. Unfortunately, I can no longer defend Ubuntu.

I tried out the latest release on a spare Thinkpad. One I use to check out distros occasionally as they come out. The t450s has a Broadwell generation i7 and 12GB of RAM. It shouldn’t be slow, especially as the installs go to a Intel DC series SSD. But this time around I found Ubuntu to feel laggy. Window animations and the opening of some programs would halt and hitch and generally run poorly. A little digging reveals that part of this is due to Ubuntu’s use of SNAP packages by default. If you install chromium from the official repos, for instance, you aren’t getting a native binary for the browser. You are getting a SNAP application. While SNAPs are a good idea I think they are a poor implementation. Flatpak tend to be much faster. Other applications have been replaced by SNAPs seemingly with no rhyme or reasoning:

• gnome-calculator
• gnome-characters
• gnome-logs
• gnome-system-monitor

Why do this? Why these applications? Why is my calculator taking so long to open? You might want to just remove them. But then they have a nasty habit of coming back. As YouTuber DistroTube discovered after his gaffe in the comments. He uninstalled snapd only to have it reinstall itself as a dependency moments later.

So I’ve prattled on about SNAPs, why I don’t like them, and why I think they were implemented in a sneaky and illogical way. Is that my only problem with Ubuntu? Goodness no.

Another example is the default DNS configuration. Since about 1991 /etc/resolv.conf has been where you go to declare your DNS nameservers. It’s worked just find all that time, so clearly it needed to be broken. Now you get autogen nameservers defined in /run/resolvconf/resolv.conf that need a symbolic link to /etc/resolv.conf. dpkg-reconfigure can fix your broken symbolic link. It took me 10 minutes of searching to fix something that was broken for no well explained reason.

So I’m done with Ubuntu. There are just too many other options that don’t do silly things for silly reasons.

Linux, as beautiful as it is, does have some longstanding problems with resource scheduling. Over the years various attempts to resolve the issue have been hampered due to the nature of the problem. Different applications need different performance tuning parameters. People that deploy only a single application on a server don’t care about desktop responsiveness. And desktop users don’t care as much about a server use case that requires maximum throughput at all cost. So you now have an operating system that is used by millions of people on millions of different types of hardware in millions of different ways. You can’t tune a generic profile that would work for everyone.

Graceful handling of storage IO tasks is the job of the scheduler. There are several. Noop, BFQ, CFQ, deadline. They all have different algorithms and design specifications. And it would be understandable if that was the first place you looked when you notice your Linux desktop becoming unresponsive under heavy disk IO load. While some schedulers are better than others at maintaining responsiveness, BFQ probably being the best fit for this type of use, a larger problem lies elsewhere.

Linux famously caches storage operations in RAM to improve performance when not under memory pressure. Because, why not? It’s basically free. The problem with caching storage operations between volatile memory and non-volatile storage is the possibility for data loss. Writes in flight will be lost if there is a power disruption, for example. So this can create problems that the conservative Kernel and distro maintainers understandably try to tune around. By default the Kernel is allowed a small amount of cache to hold asynchronously before it is forced to stop everything and call a synchronous write to flush its data to NV storage. We have some parameters:

vm.dirty_background_ratio

The percentage of data that is allowed to be in memory and uncommitted to disk. The possibility for data loss exists in this state, hence the name “dirty.” This is set at 10% by default. For many more desktop oriented workloads this is going to be quite low.

vm.dirty_ratio

The dirty ratio sysctl setting is the max the Kernel will keep before halting everything and forcing a synchronous write. If you feel your desktop ‘hitching’ for a period of time during IO load, you may very well be seeing this exact issue playing out. The storage system can’t keep up, throws up its hands, and forces everything to stop so it can catch up. The default 20% here is, again, quite low for what most would want on a desktop.

vm.dirty_expire_centisecs

Essentially a timer, the expire parameter states how long dirty data can sit waiting to be flushed async before the kernel forces a sync call. The value is in centiseconds, so divide by 100 to get normal seconds.

If your data isn’t particularly mission critial you can increase each of these values in order to allow the system to work through IO spikes asynchronously. This will lead to a much smoother experience for other applications.