I bought a Lenovo IdeaPad 3 15ALC6. It features an AMD Ryzen 3 5300U with integrated graphics, 8GB of RAM and 512GB NVMe. These features are adequate for its low price, though an even better deal would have been the slightly more expensive model with AMD Ryzen 5 5500U. The main limitation I can see is the small amount of memory which is shared with the GPU, but this is common in budget laptops.

Its compatibility with Linux turned out to be excellent because Lenovo has apparenly made some effort to ensure Fedora and Ubuntu work smoothly on their laptops. The model I bought didn’t have an OS preinstalled and I decided to install Ubuntu 22.04.

Installation

The first thing to do before installation was to visit BIOS by pressing F2. You can select the boot media by pressing F12.

I disabled Secure Boot because it usually gets in the way of Linux distributions, although it’s not strictly necessary for Ubuntu.

Another option of interest was Set UMA Frame Buffer size. This gives you the option of choosing how much of the shared memory to be allocated to the GPU, either 512MB, 1GB or the default 2GB. If you leave it at the default setting, you will find that only 6GB out of 8GB are available to the main system and that’s not much.

After completing the installation, I wanted to try certain things that could improve the battery life. I took some hints from the Arch wiki. I configured the system to use the newer AMD P-State driver instead of ACPI CPUFreq. Phoronix has some benchmarks that show it to be marginally more energy efficient.

I modified /etc/default/grub to look like:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash amd_pstate=passive"

then ran sudo update-grub.

Phoronix has also benchmarked the power consumption of Gnome and KDE under Wayland and X.Org and found that Wayland is more power efficient in either Gnome or KDE.

Mostly AC power

If you mostly use the laptop connected to a power supply, Lenovo has a feature called Conservation Mode which can prolong the life of the battery. It keeps the battery charged at 50-60% instead of going up to 100% and staying there continuously, which can reduce its capacity over time. Here is a good source of information on the issue.

To enable Conservation Mode:

echo 1 | sudo tee /sys/bus/platform/drivers/ideapad_acpi/VPC2004:00/conservation_mode

To disable Conservation Mode:

echo 0 | sudo tee /sys/bus/platform/drivers/ideapad_acpi/VPC2004:00/conservation_mode

This feature is sticky, you only have to do it once and it survives across reboots.

Mostly battery

I plan to use this laptop now and then, mostly on battery. This means that the laptop will stay a lot of time in suspend mode. Suspend does use up a small amount of the energy over time, so a better idea would be to either Power Off or to enable Hibernate.

I enabled Suspend-Then-Hibernate. This mode combines the best of both options: The system is suspended when I close the lid. It starts again quickly when I open it. If I don’t use for 3 hours, then a RTC trigger wakes it up and hibernates it.

Here is how to enable Hibernate on Ubuntu 22.04:

• Boot a live USB.
• Start gparted
• Shrink the main partition. The laptop has 8GB of RAM so I went with a 10GB partition, following this advice.
• Create a swap partition with type linux-swap
• Reboot into main installation.

Modify /etc/fstab to include the new swap partition:

# Get the UUID of the swap partition
lsblk --fs
sudo swapoff -a

## /etc/fstab
#/swapfile  none    swap    defaults        0       0 
UUID=<YOUR_SWAP_UUID>  none    swap    defaults        0       0

Comment out the line about the swapfile, we are using a swap partition now. You can later remove /swapfile to reclaim its space.

sudo swapon -a

Modify /etc/default/grub again to look like:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash resume=UUID=<YOUR_SWAP_UUID>"

sudo update-grub

Try it:

sudo systemctl hibernate

This also enabled the option to Hibernate in Gnome’s Power Off/Log out menu. I didn’t need to install any package, since it’s all handled by systemd.

And now we can enable Suspend-Then-Hibernate:

Modify /etc/systemd/logind.conf. Change

#HandleLidSwitch=suspend

to

HandleLidSwitch=suspend-then-hibernate

You can also use hybrid-sleep if you prefer, which hibernates the system and then suspends it instead of powering off.

Have also a look at /etc/systemd/sleep.conf and in particular HibernateDelaySec=180min, the time before the system wakes up from suspension to hibernate.

Reboot.

Note that this method only handles suspension when you close the lid. If suspension is triggered by other means e.g. by inactivity after some time, then it will go into normal suspension. However it’s good enough for me. There are other options that may work better for you.

Another tip that will prevent the battery from losing capacity over time: Don’t fully charge it and don’t let it fully discharge. Keep the battery levels at 30%-80%.

Other tweaks

Although I prefer Firefox, I installed Google Chrome. I didn’t use snap but went instead with apt and the .deb straight from Google. The reason I picked Google Chrome, not even Chromium, was because I wanted to make sure video decoding was hardware accelerated (and more power efficient). You can verify that this is the case visiting chrome://gpu.

I also tried TLP and auto-cpufreq. These are often recommended to optimize battery life further. However, I am not sure they do something useful. Gnome has power-profiles-daemon and KDE has the equivalent powerdevil. Bluetooth, which is power hungry, is turned off by default by Gnome. So perhaps TLP isn’t as useful these days as it used to be. As for auto-cpufreq, what it does is toggle the scheduler between powersave and performance as well as turn CPU boost on and off, instead of letting schedutil handle it. It seems a bit messy to me and possibly hurting performance, so I didn’t keep it after all.

So what is the battery life on this Linux laptop compared to Windows? I haven’t run a benchmark but compared to the battery life of a Dell Latitude 3520, a laptop with similar specifications running Windows, my impression is that they last about the same. Linux however uses fewer resources, especially memory and it generally feels snappier.

TP-LINK Archer T4U Plus AC1300 is a WiFi 5 USB adapter. It features 2 antennas which can be rotated in any direction and a USB 3.0 connection (which also works in USB 2). The price is ok. I thought it was a good choice for a spare computer that I don’t use often.

Of course the reason that I don’t use it often is because the previous adapter, TP-LINK TL-WN722N, a 2.4Ghz 802.11 b/g/n performed so badly in a congested environment. It was old and it had to go.

Before you buy

To avoid any bad surprises, I tried to find if it was compatible with Linux before I bought it. TP-LINK doesn’t officially support Linux (it only supports Windows). Nevertheless I found that many people had it working without problems so it was worth buying.

The chipset used by the device is either Realtek RTL8812AU or RTL8812BU depending on the version V1, V2 or V3 and you need a specific driver for each chipset!

lsusb gave me this information:

Bus 001 Device 002: ID 2357:010e TP-Link Archer T4UH v2 [Realtek RTL8812AU]

So mine is a V2 with a Realtek RTL8812AU.

Realtek RTL8812AU

The driver that I ended up using is https://github.com/aircrack-ng/rtl8812au which can be installed on Ubuntu and other distributions. It is also packaged for Arch Linux.

To install on Ubuntu, you first have to install some requirements for building modules e.g.

sudo apt install build-essential dkms git iw

and on Debian and others, also make sure you have the Linux headers installed. Then follow the instructions on the driver’s site to install it with DKMS.

Actually, Ubuntu and Debian do have a packaged driver for RTL8812AU. You can try it with

sudo apt install rtl8812au-dkms

I can’t comment on how well this one works. I didn’t try it much because I had aircrack-ng already working. I believe aircrack-ng is actively maintained and performs better.

Realtek RTL8812BU

On the other hand, if you have an RTL8812BU you can try this driver:

https://github.com/morrownr/88x2bu-20210702

There are other drivers too and Linux 6.2+ probably includes some driver on its own. The above is still recommended because it is more tried for the time being.

What about WiFi 6

Maybe you noticed that I bought a WiFi 5 adapter while WiFi 6 is available. One reason is that I thought WiFi 5 would be enough for these specific needs: I only need to saturate the Internet connection and I don’t use that computer very often. So I thought I could keep the costs down and buy a WiFi 5 extender with the money I spared, to improve the signal.

There is also another important reason. TP-LINK TX20U Plus is another model that I considered, it’s a bit more expensive but supports WiFi 6. It uses the chipset Realtek RTL8852AU and a driver for it is at https://github.com/lwfinger/rtl8852au, however the driver does not perform well and may have other problems. The very useful USB WiFi Chipset reference specifically says to avoid.

From what I figured, if you are looking for an extra hassle free, plug and play WiFi 6, try to buy a PCI Express WiFi with an Intel chipset. Intel does fully support Linux and its drivers are in the kernel: worse thing you have to upgrade the kernel. Of course that’s not the only option, other chipsets may have good support too, but an Intel chipset is a “fail-safe” option for Linux. So do try to find out the chipset used and its maturity of support in Linux before buying any WiFi adapter.

The latest update of Arch Linux informed me that pipewire-media-session, a dependency of kwin, is deprecated and would soon be removed from the repositories. I was adviced to replace it with wireplumber. Now, I did try wireplumber in the past, but it didn’t go so well so I had to revert it. It’s finally time to move on, I guess.

The problem is that this had some cascading effects and as a result I had to let go of the (old trusted) pulseaudio as well.

Not what I planned for Sunday morning

The first step was to replace pipewire-media-session with wireplumber as instructed:

$ sudo pacman --asdeps -S wireplumber

which removed pipewire-media-session. Rebooting to check how well things went:

• System sound (back lineout) works
• System sound (front headphones) works
• System microphone (front) does not work
• HDMI sound through the video card does not work
• Webcam, didn’t test

Obviously, not good enough.

It turns out that pipewire was missing its plugins (backends), but that meant removing pulseaudio since they are in conflict.

$ sudo pacman --asdeps -S pipewire-alsa pipewire-pulse pipewire-jack
resolving dependencies...
looking for conflicting packages...
:: pipewire-alsa and pulseaudio-alsa are in conflict. Remove pulseaudio-alsa? [y/N] y
:: pipewire-pulse and pulseaudio are in conflict. Remove pulseaudio? [y/N] y
:: pipewire-jack and jack2 are in conflict (jack). Remove jack2? [y/N] y

Packages (6) jack2-1.9.21-3 [removal]  pulseaudio-16.1-3 [removal]  pulseaudio-alsa-1:1.2.7.1-1 [removal]
             pipewire-alsa-1:0.3.64-1  pipewire-jack-1:0.3.64-1  pipewire-pulse-1:0.3.64-1

Total Download Size:    0.31 MiB
Total Installed Size:   1.11 MiB
Net Upgrade Size:      -6.70 MiB

:: Proceed with installation? [Y/n]

So we are replacing distinct packages with a monolith to rule-them-all. I am getting systemd vibes here.

Progress (and neverending bug squashing)

After rebooting and testing everything, things now appear to work, including the webcam with its own microphone. Since I have multilib installed, I also installed some lib32 versions

$ sudo pacman --asdeps -S lib32-pipewire lib32-pipewire-jack

For video, some packages to consider are (according to the wiki) gst-plugin-pipewire (for gstreamer) and/or pipewire-v4l2 but I didn’t have a need for them so far… Webex, at least, works with the current configuration.

Another thing to consider: I had some options in /etc/pulse/default.pa to avoid a crackling noise in HDMI output. I am not sure if they are still needed, but now that pulseaudio is gone the place to put pulseaudio related options would be /usr/share/pipewire/pipewire-pulse.conf after copying it somewhere under /etc/pipewire.

I wanted a file share somewhere on the Internet and WebDAV is one of the most widely supported protocols. Before it, I considered the alternatives:

• Give the files to a “cloud” provider to do with them as they please. No. Even if they are just groceries lists.
• SSHFS works nicely without any effort, except that it’s slow and not supported by Windows Explorer.
• Nextcloud would also work, but I think it’s an overkill since it’s a fairly complex PHP application that does many things, most of which I don’t care about, and yet it doesn’t do other things I care about, such as being compatible with rsync.

I wanted to do it my way and KISS. I finally managed to do it my way but KISS is in the eye of the beholder.

There are several guides to setup Nginx or Apache for file sharing with WebDAV. Except… for all the small bugs and incompatibilities, as I found out along the way. You see, I wanted to access the share from several clients…

• Windows Explorer
• KDE Dolphin
• Gnome and MATE
• Android FolderSync

And many of them had their own idiosyncratic way of interpreting the RFCs that describe WebDAV.

Here are some of the bugs, problems, things I couldn’t get the server and clients to agree on:

• Sensitive to trailing slashes.
• Windows Explorer sends an OPTIONS / even when the share is in a subfolder.
• Must support LOCK.
• MOVE and COPY expect a Destination: header to match the Host: scheme (which isn’t necessarily true if you use a reverse proxy).
• KDE Dolphin couldn’t interprete the MIME type when used with a Go based client. This seems to have been fixed in the latest version of Dophin.

I had to try a few solutions until I found something that fully worked.

Nginx on its own

Nginx has rudimentary support for WebDAV and needs a separately maintained extension to support LOCK and other methods. It suffered from incompatibilities with missing trailing slashes. There are some guides that attempt to add all the extra hacks needed to make it work, but I gave up on this for the time being. If you are interested in trying it anyway, be informed that the version of the extension included in Ubuntu 20.04 LTS is not recent enough.

The next thing to try was to use a seperate WebDAV server and put it behind a Nginx reverse server, which is what this blog post is about. I already used Nginx for some web hosting and I simply wanted to add a WebDAV file share.

Dave

It’s a standalone WebDAV server with a simple .yaml configuration. It supports multiple users, each with their own folder and password protection. It is written in Go, using the same semi-standard library that Caddy uses for its WebDAV extension. What it doesn’t do is provide “autoindex”, to be able to access the files using a browser, but we can add this functionaly using our reverse proxy.

Do give it a try, if you just want something to quickly set up and be done with it. However I should let you know that its development has finished successfully, all planned features have been added and all bugs have been squashed, which means it won’t be updated further except to keep its dependencies up to date. Here is a fork I made with updated dependencies and minor cleanup.

Apache

Apache is probably the oldest and most widely used implementation of a WebDAV server, except for the fact that I already use Nginx so why would I want to install Apache as well? Just for its WebDAV functionality, behind Nginx.

I won’t be giving full installalation instructions here, there are enough guides for that. However just for completion and because everybody loves copy-paste here is an example of how it looks like:

DavLockDB /usr/local/apache/var/DavLock
<VirtualHost 127.0.0.1:8080>
    ServerAdmin admin@domain.tld
        ServerName domain.tld
        ServerAlias domain.tld
        DocumentRoot /path/to/webdav
        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined

        #Alias /webdav /path/to/webdav

        <Directory /path/to/webdav>
            DAV On
        </Directory>

        # See https://github.com/BytemarkHosting/docker-webdav/blob/master/2.4/conf/conf-available/dav.conf
        # We turn it on for all User Agents here because Windows Explorer and KDE Dolphin appear affected
        SetEnv redirect-carefully
</VirtualHost>

If you use /srv take a look at apache2.conf to make sure the proper lines are uncommented. You also have to create /usr/local/apache/var/DavLock owned and writable by www-data:www-data (on Ubuntu/Debian).

Nginx Reverse Proxy

Here is the fun part! (not really).

After putting Dave or Apache behind the reverse proxy, it worked as intended except for renaming and copying files. Investigating further and standing on the shoulders of previous sufferers giants, I finally figured the cause: MOVE and COPY methods use a Destination header that must match the scheme of Host e.g. https for https and not http. This isn’t true behind a reverse proxy.

Add to this a long standing bug in Nginx and its suggested workaround and you get this Magic Copypasta:

    set $dest $http_destination;
    if ($http_destination ~ "^https://(?<myvar>(.+))") {
       set $dest http://$myvar;
    }
    proxy_set_header Destination $dest;

It looks ugly, but its a workaround for a bug after all.

Here’s how it ended up looking like after several iterations. Nginx handles the Basic Authentication. There are some optimizations to use keepalive and reduce the number of localhost connections that may or may not matter in practice.

upstream webdav {
    server 127.0.0.1:8080;
    keepalive 32;
}

server {
    server_name domain.tld;

    root /var/www/html;

    access_log /var/log/nginx/domain.tld;

    client_max_body_size 0;

    location / {
        auth_basic           "Restricted area";
        auth_basic_user_file /etc/nginx/.htpasswd;

    # https://mailman.nginx.org/pipermail/nginx/2007-January/000504.html - fix Destination: header
    # https://trac.nginx.org/nginx/ticket/348 - bug, workaround with named capture
    set $dest $http_destination;
    if ($http_destination ~ "^https://(?<myvar>(.+))") {
       set $dest http://$myvar;
    }
    proxy_set_header Destination       $dest;

    #rewrite /webdav/(.*) /$1 break;
    proxy_pass http://webdav;

    proxy_buffering off;

    # Keep-alive
    proxy_http_version                 1.1;
    proxy_set_header Connection        "";

    # Proxy headers
    proxy_set_header Host              $host;
    proxy_set_header X-Real-IP         $remote_addr;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Host  $host;
    proxy_set_header X-Forwarded-Port  $server_port;

    # Proxy timeouts between successive read/write operations, not the whole request.
    proxy_connect_timeout              300s;
    proxy_send_timeout                 300s;
    proxy_read_timeout                 300s;

    listen [::]:443 ssl http2;
    listen 443 ssl http2;
    # ... SSL stuff ...
}

Left as an exercise for the reader:

• Create the .htpasswd file (I used Apache’s htpasswd).
• Use 308 redirects instead of 301 to redirect http to https (because 301 doesn’t support all the request methods of WebDAV).
• If needed, include an extra location with “autoindex”.

ssh is a very versatile tool that many of us depend on for everyday tasks. Other than the basic connection to a remote host and multiplexing the terminal with tmux or screen, it is also used

• together with rsync
• to execute remote commands in scripts
• for port forwarding
• for an adhoc VPN of sorts

That’s a lot of functionality. Yet in some places the connection is restricted behind an HTTP proxy that won’t let ssh do its magic. Fortunately, it’s still possible to configure ssh for these cases and here we’ll be covering such a scenario. HTTP proxies usually only allow connections to specific ports such as 80 and 443, although they allow arbitrary TCP streams with the CONNECT method.

Linux, MacOS and Windows with WSL

We will be using netcat-openbsd, as it’s called in Ubuntu and Debian. Apparently there are two implemenations of netcat and we want the one that supports the -x “connect to proxy” parameter. Here is how ~/.ssh/config would look like:

Host otherside
    HostName example.com
    User torvalds
    Port 443
    IdentityFile ~/.ssh/id_ed25519
    ProxyCommand nc -X connect -x 10.20.30.40:8080 %h %p
    LocalForward 9999 127.0.0.1:5050

We are connecting to torvalds@example.com over an HTTP proxy at 10.20.30.40:8080 . netcat also supports SOCKS proxies and authentication if you need them, but you’ll have to man nc for more information on these topics. As a bonus, we forward the local port 9999 to port 5050 on the remote server.

The tricky part is that the ssh server has to listen to port 443, which is normally used by HTTPS. Don’t worry about that, we’ll fix it later.

Windows native

Windows users are probably used to tools like PuTTY and WinSCP to handle their ssh and sftp connections. These programs do support proxy connections, forwarding ports and the like. One thing to keep in mind is that PuTTY uses it’s own file format for ssh key files, however it’s possible to import an existing openssh key into it. We will not be covering their configuration here though. One limitation of them is that they cannot be used with Visual Studio Code for remote development over ssh and you don’t get the handy rsync either.

Windows come with their own version of OpenSSH which can be enabled as an optional feature. Its configuration files can be found in C:\Users\username\.ssh. We also need to install Nmap, which comes with its own netcat-like program, called ncat.

C:\Users\username\.ssh\config:

Host otherside
    HostName example.com
    User torvalds
    Port 443
    IdentityFile C:\Users\torvalds\.ssh\id_ed25519
    ProxyCommand C:\Program Files (x86)\Nmap\ncat.exe --proxy 10.20.30.40:8080 %h %p
    LocalForward 9999 127.0.0.1:5050

It works the same as described in the previous section.

Some gotchas…

• Nmap needs Administrator rights in order to be installed and used, but ncat doesn’t. If you are unable to install Nmap, then I suggest that you install it on a computer where you do have Administrator rights, then copy ncat.exe, all DLLs and ca-bundle.crt. These are all the files you need.
• Do not use an old, so-called portable version linked by ncat’s site.
• nmap 7.93 has a bug, use 7.92 or a later version instead.

DNAT the incoming connection on the server

As we mentioned before, the ssh client has to connect to port 443 in order to pass through an HTTP proxy. The port may be already in use by a web server. We can work around this requirement by using a DNAT rule on the server.

sysctl net.ipv4.conf.$dev.forwarding=1
iptables -t nat -A PREROUTING -p tcp -i $dev --src $proxy_ip --dport 443 -j DNAT --to-destination $my_ip:$ssh_port

$proxy_ip is the outgoing IP of the HTTP proxy e.g. the address duckduckgo.com gives you when you search for “my ip”. $my_ip is the server’s IP and $ssh_port the port ssh is normally listening to.

Note that both port 443 and $ssh_port must be open in the firewall rules or at least accept connections coming from $proxy_ip.

Final remarks

You may want to encrypt the connection with TLS. This mainly serves to obfuscate the headers sent by ssh and make it look more like a common HTTPS connection. This will not be covered here; for more information, have a look at the program’s guide.

That’s it. Have fun.

I was having this problem for some time. After submitting this site’s sitemap.xml to Google Search Console, I was getting a red warning that it could’t fetch the sitemap, because of some unspecified HTTP error. Not very helpful. Plus there wasn’t any sign in the server logs that it was attempting to fetch it.

Apparently this is a fairly common problem and there can be many causes for it:

• A malformed sitemap.xml
• robots.txt or X-Robots-Tag blocking access to crawlers
• Connectivity problems, such as IPv6

I double checked that the sitemap was correct and it could be fetched by others. Bing Webmaster Tools had no problem accessing and parsing it either.

Some people advice to just wait it out and it will solve itself, but after waiting for a few weeks things hadn’t changed. The cause had to be something more obscure. A hint was in nginx error.log:

2021/09/11 15:15:33 [crit] 35499#35499: *136 SSL_do_handshake() failed (SSL: error:141CF06C:SSL routines:tls_parse_ctos_key_share:bad key share) while SSL handshaking, client: 123.123.123.123, server: 0.0.0.0:443

geoiplookup told me that the client’s IP resolved to a domain that belongs to Google.

After some more investigating using tshark and wireshark, I suspected that the cause might be some TLS cipher incompatibility. So, I tried changing this setting in nginx:

ssl_prefer_server_ciphers on;

which Let’s Encrypt sets to off. And it worked. It worked well enough to allow me to submit the sitemap. Although the initial response was that it still couldn’t fetch the file, I could see in the logs that it was getting it and next time I checked I saw the green success sign.

However, this is more of a workaround. I still get such TLS error messages in the logs from Google’s domains, although the crawler works properly. I am convinced that it isn’t so much a problem in my configuration, since I had it with both vanilla Let’s Encrypt and Caddy’s automatic HTTPS configuration.

The real cause is probably that Googlebot, having to index the whole Internet, supports a large number of old and insecure ciphers. Some of them may be causing a problem in new configurations. I could possibly identify exactly which cipher causes the problem and disable it but I would rather have them take a look at this issue, which may be affecting others as well.

Redirecting a domain and Bing

Bing does things a bit differently. For example, it doesn’t distinguish between http:// and https:// domains and even more confusingly, it doesn’t distinguish between www and non-www domains. At least that’s my understanding. I wonder how it handles multiple subdomains.

It also seems to be getting confused by redirections of files that really belong to a single domain, such as robots.txt and sitemap.xml. If I understand correctly, Googlebot ignores such redirections (at least it does for robots.txt) while Bingbot follows them and…

I was trying to redirect this site from its old domain to its new domain. The first approach was to simply 301 redirect everything. Using Change Of Address in Google Search Console was sufficient to redirect the old pages to their new location and to add new content as well. Bing on the other hand, has removed its equivalent Site Move Tool. The old pages weren’t being reindexed to discover the redirections and new pages on the new domain were ignored.

After further investigation/hair pulling/trying various things, I came to the conclusion that this was caused by the redirection of sitemap.xml and robots.txt (which includes the URL of sitemap.xml). So I tried redirecting everything except for these files, which were served as they were in the old domain. Here’s an example for nginx:

# Redirect old domain: https://www.olddomain.com -> https://www.newdomain.com (canonical)
# except for robots.txt and sitemap.xml
server {
    server_name www.olddomain.com;

    access_log /var/log/nginx/olddomain.log;

    location / {
        return 301 https://www.newdomain.com$request_uri;
    }

    location /robots.txt {
        root /var/www/www.olddomain.com;
    }

    location /sitemap.xml {
        root /var/www/www.olddomain.com;
    }

    # ...
}

It gets more complicated because of http://, https://, non-www and www redirections, but I decided to redirect everything old to the old canonical and from there to the new canonical, with the exceptions that were mentioned.

So we have two versions of sitemap.xml, one served from the old domain with the old URLs and the new one. After reindexing some pages and waiting, it seems that it got unstuck. The old pages have been reindexed, Bingbot found the redirections and removed them and the new ones are slowly getting indexed as well. I will have to wait a bit more to be entirely sure because, apparently, Bingbot crawls slowly and selectively.

I used to not think much about SEO. I thought it was about marketing tricks and misleading users. But putting all that aside, in the end, you do want people to find your site and this means making your site crawlable by the search engine bots such as Googlebot and Bingbot. You have to help them a little do their part.

robots.txt introduces your site to search engine bots

Hugo does create a basic robots.txt by default which allows all crawlers access to everything. It is suboptimal though and we can provide some more information to help search engine bots find their way around our site.

Your Hugo theme likely creates some taxonomy index pages about tags and categories. You probably don’t mean to include these pages in the search results but rather the pages they link to. Another problem these automatically generated pages may cause, is slowing down the crawling of your site, because search engines allocate a crawling budget to you; you had better make the best use of it. Finally search engines just don’t like duplicate pages and they may deduce that your site if of poor quality. I assume you have a /posts/ or similar page anyway, with links of interest listed there.

Another recommended piece of information to put in robots.txt is the sitemap of your site. A sitemap is the list of URLs of your site, so that a crawler doesn’t have to discover them by itself. Hugo also generates a sitemap.xml for you, but this isn’t mentioned in the robots.txt file it creates.

The easiest way to create a custom robots.txt file is to disable Hugo’s automatic generation. Change this setting in config.toml:

enableRobotsTXT = false

And then we can put our custom robots.txt in the /static folder:

User-agent: *
Disallow: /tags/
Disallow: /categories/

Sitemap: https://www.example.com/sitemap.xml

Next time you run hugo, this file will be copied as-is in the /public folder.

Descriptions, because first impressions matter

Next after the title of your page and its URL, the most important thing to consider from a SEO perspective is its description. Although it will not affect itself the ranking of your page, it will affect the click rate since a better written description can attract more clicks. Keep in mind that search engine may or may not actually use your description if it’s too short or misleading. The descriptions are created by your theme and the generated HTML looks like this:

<meta name="description" content="I am a description.">

<meta itemprop="description" content="I am a description.">
<meta name="twitter:description" content="I am a description."/>
<meta property="og:description" content="I am a description." />
...

The first line is the “proper” description. The next lines are generated by Hugo’s internal templates and in particular <meta itemprop="description" ...>, defined in _internal/schema.html, seems to be preferred over <meta name="description" ...> by Bing and the other engines it shares data with (more on that later).

What ends up in the description is up to the theme. If you didn’t specify a description in the front matter of your post, it is likely that it will try to generate a summary and use it as a description. The problem here is that autogenerated summaries aren’t very good, since they are usually just the first lines of your post; this may be not what you intended. So I suggest that you have a look at the HTML that your theme generates and if it’s not satisfactory, specify a description of your own in the front matter. It is suggested that the description is 70–155 characters to fit in the search engine results.

Don’t forget about Bing

After putting the URL of your sitemap in robots.txt it is still a good idea to submit it yourself to Google Search Console. This way, you can monitor the crawling of your page by Googlebot and catch any errors. You can also keep an eye on which searches drive traffic to your site.

When it comes to search engines, Bing is a distant #2. But it’s still important, because its crawler Bingbot also provides data to Duckduckgo and possibly other lesser known engines such as Qwant and Ecosia. Therefore it makes sense to make sure that Bingbot can crawl your site correctly.

Bing Webmaster Tools acknowledge that you probably already used Google Search Console before coming to them, so they make it easy to login by using your Google account (along with other options such as a Microsoft account). After that you are given the option to import the domains and sitemaps you have set up in Google Search Console. I must say it all worked flawlessly and effortlessly. Plus I didn’t have to deal with certain bugs of Google Search Console. You are also given options similar to the ones Google Search Console gives you, such as URL Inspection and Performance metrics.

Inform search engines about site changes

After making changes to your site and posting something new, you can let the search engines know that the sitemap has changed. An automatic way of doing this is pinging them with the sitemap URL.

# Ping Google about changes in the sitemap
curl "https://www.google.com/ping?sitemap=https://www.example.com/sitemap.xml"

# Ping Bing about changes in the sitemap
curl "https://www.bing.com/ping?sitemap=https://www.example.com/sitemap.xml"

You can include it in your deployment method; left as an exercise to the reader.

Patience

Congratulations, you went so far and now… you wait. It can take a few days to a few weeks until the search engines have crawled your site and this assuming nothing unexpected has gone wrong and there are no problems to fix. In the meantime I suggest that you take the opportunity to practice your Zen Buddhism exercises and take your mind away from all that pointless worrying.

A more simple way is to use a terminal that supports OSC 52 control sequences. The more common terminals such as konsole and gnome-terminal do not support them, because they can be a security concern (an application running on the terminal “stealing” your clipboard). On the other hand they are too convenient to pass…

Some terminals that do support OSC 52 are xterm (with a configuration option) and alacritty.

Alacritty is an OpenGL accelerated terminal emulator written in Rust. It doesn’t support tabs and I think it is intended to be used either with a terminal multiplexer such as tmux and screen or with i3. It is cross platform, supporting all major platforms. On Linux, you can get it from your distribution’s package manager. On Windows and MacOS, you can grab the binaries.

In any case, all you need is these options in ~/.tmux.conf:

set -g default-terminal "tmux-256color"
set -g mouse on

And then use the mouse to select a region. It will be copied to clipboard.

Special considerations for ssh connections

• If you are using Debian on the remote, make sure ncurses-term is installed, to provide the extra terminal definitions that include alacritty. In Ubuntu, it should be already installed.
• Some older distributions such as Debian buster did not yet include terminfo entries for alacritty. In that case you can configure alacritty to use TERM=xterm-256color instead of TERM=alacritty, by adding these lines to alacritty’s configuration file ~/.config/alacritty/alacritty.yml:

env:
  TERM: xterm-256color

Making it more comfortable

You can choose a color scheme here. In my case, I picked Breeze. I clicked on its arrow and copy-pasted the scheme in a file called ~/.config/alacritty/breeze.yml.

I also like F11 to toggle full screen. Putting it altogether, append these lines to ~/.config/alacritty/alacritty.yml:

import:
 - ~/.config/alacritty/breeze.yml

key_bindings:
  - { key: F11, action: ToggleFullscreen }

OVH recently announced that it will not offer the option of Arch Linux and FreeBSD for new installations. Although they recommend that you upload your own image if you are using its Public Cloud, this option is not available to VPS users.

Not a problem though. There are a few ways to install any OS you want and they seem ok with it.

Use a cloud image

I have tested this method with Arch and Debian bullseye images and everything worked fine, including IPv6 configured out of the box. The following steps describe an Arch installation but, with small adjustments, they should work for other distributions as well.

Install any available distribution such as e.g. Debian.

Log in the manager and reboot in rescue mode. The rescue system is Debian based and includes backports and ZFS packages if you ever need them.

Use lsblk to orient yourself. The Debian based rescue disk should be mounted at /dev/sda while your target disk is at /dev/sdb.

Make some extra room for the image:

mkdir /tmp/mnt
mount -t tmpfs tmpfs /tmp/mnt
cd /tmp/mnt

Download the image with wget:

• Arch Linux https://mirror.pkgbuild.com/images/latest/, use cloudimg
• Debian https://cloud.debian.org/images/cloud/bullseye/daily/latest/ use genericcloud-amd64

We need qemu-img from qemu-utils:

apt install qemu-utils

Overwrite your target disk /dev/sdb with the image as shown in the arch wiki. Notice we are overwriting the whole disk, not just a partition:

qemu-img convert -f qcow2 -O raw Arch-Linux-x86_64-cloudimg-20210315.17387.qcow2 /dev/sdb

The Arch Linux image uses a GPT partitioning scheme, a btrfs partition compressed with zstd and a BIOS compatibility partition.

To make sure you are able to login after reboot, it’s a good idea to setup a user with sudo abilities and a known password. You may also copy your ssh public key manually.

sfdisk -l /dev/sdb
mount -t btrfs /dev/sdbX /mnt
chroot /mnt
useradd -m -s /bin/bash -G wheel arch
passwd arch
echo "arch ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/arch

Finally use the manager to reboot the virtual machine in normal mode.

Cloud images contain cloud-guest-utils which should take of resizing the partition table to fill the disk, setup networking etc. If for some reason you can’t connect with ssh, you can still try KVM to login.

If you got so far, then it’s time to secure the installation:

• update
• configure ssh to only use public key authentication rather than password authentication
• setup a firewall
• etc.

Final tuning

OVH recommends that cloud-guest-utils and qemu-guest-agent are installed when using a custom image. cloud-guest-utils are already installed, so:

sudo pacman -S qemu-guest-agent
sudo systemctl enable --now qemu-guest-agent.service

Have a look at /etc/default/grub.

• To have KVM console work, we need to add console=ttyS0 to boot parameters.
• To improve performance, you may be interested in disabling compression by removing rootflags=compress-force=zstd.
• Provide some entropy to the kernel’s random number generator. You may decide that you’d rather trust the CPU manufacturer’s random number generator rather than get warnings in boot messages about using an uninitialized urandom.

These are just some recommendations. So, let’s modify /etc/default/grub to look like this:

GRUB_CMDLINE_LINUX_DEFAULT="random.trust_cpu=on"
GRUB_CMDLINE_LINUX="net.ifnames=0 console=ttyS0"

And then we can apply the changes.

sudo grub-mkconfig -o /boot/grub/grub.cfg
sudo reboot
sudo btrfs filesystem defragment -r /

Perhaps you are interested in converting the system to use a flat hierarchy of subvolumes by using steps similar to the ones described here.

Here is a benchmark comparing the performance of Postgresql on btrfs and other filesystems.

Arch Linux maintainers use several of these options in the system unit files that they ship, while Debian and Ubuntu maintainers generally only use the options that the upstream developer has used, if any. For examples, have a look at the systemd unit files for memcached and mariadb in /usr/lib/systemd/system.

These options generally fall in these categories:

• Filesystem namespace
• Other namespaces such as user
• Capabilities
• seccomp and system call filter

Some of these options have a performance cost, in particular seccomp:

• https://github.com/systemd/systemd/issues/18370
• https://lore.kernel.org/linux-security-module/c22a6c3cefc2412cad00ae14c1371711@huawei.com/T/, see Lennart Poettering’s comments

Although someone could simply enable all available options and take the performance hit when security is critical over anything else, in other cases it makes sense to try to balance security and performance and carefully pick the options that provide most security benefit for an acceptable performance cost.

Get a report on a service’s security score

The first step is to generate a report on the service

sudo systemd-analyze security mydaemon.service --no-pager

This will give as a total score. Notice that the available options have a weight, according to their estimated impact on security.

Then we can add a snippet with extra options using

sudo systemctl edit mydaemon.service

List of options

I had a look at the source of systemd v248.3

Filesystem namespace: low cost, high impact

src/core/namespace.h

• ProtectHome
• ProtectSystem
• ProtectProc
• ProcSubset
• ProtectKernelTunables
• ProtectKernelModules
• BindMount
• MountImage

Capabilities: low cost, high impact (man prctl)

• CapabilityBoundingSet
• AmbientCapabilities
• NoNewPrivileges

Other options to consider for a chroot:

Also namespace thus low cost.

• RootDirectory
• MountAPIVFS
• PrivateUsers
• DynamicUser

Seccomp (high cost, varying impact)

src/core/execute.c #if HAVE_SECCOMP ... #endif

• SystemCallFilter=
• SystemCallLog=
• SystemCallArchitectures=
• RestrictAddressFamilies=
• MemoryDenyWriteExecute=
• RestrictRealtime=
• RestrictSUIDSGID=
• ProtectKernelTunables=
• ProtectKernelModules=
• ProtectKernelLogs=
• ProtectClock=
• PrivateDevices=
• RestrictNamespaces=
• LockPersonality=

I think it makes sense to use most of the filesystem sandboxing options and capabilties and then pick those seccomp options that will have the most impact on the security of the system, using the report of systemd-analyze security as a guide.

See systemd service configuration options for a full list of currently available options. Some options may not be supported by older systemd versions. See analyze/analyze-security.c for the weight of options or the report of systemd-analyze security.

Example systemd unit

# Tuned after:
# sudo systemd-analyze security caddy.service --no-pager

## Filesystem namespace options (cheap)

# Mount most things read-only and set read-write paths
ProtectSystem=strict
ReadWritePaths=/var/lib/caddy /var/log/caddy
InaccessiblePaths=...
ProtectHome=true
PrivateTmp=true
ProtectProc=invisible
ProtectKernelTunables=true
ProtectControlGroups=true

## Capabilities (man prctl) (cheap)
NoNewPrivileges=true
#CapabilityBoundingSet=
CapabilityBoundingSet=CAP_NET_BIND_SERVICE

## Seccomp (expensive)

# High impact
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
RestrictNamespaces=true

# Misc recommended
PrivateDevices=true
ProtectKernelModules=true
ProtectKernelLogs=true
ProtectClock=true

# Other
#RestrictSUIDSGID=true
#RestrictRealtime=true
#MemoryDenyWriteExecute=true
#LockPersonality=true
#SystemCallArchitectures=native
#SystemCallFilter=@system-service
#SystemCallErrorNumber=EPERM

• Pros
  • Snapshots to easily rollback when/if an update goes wrong
  • Easy backups
  • Container friendly
• Cons
  • Weak random read/write performance when used for a database or a virtual machine host

For development, the pros outweigh the cons and I wouldn’t mind the extra features on a web server. However in that case database performance is often the limiting factor. So I was curious, not whether Btrfs would outperform Ext4 and XFS but rather if it’s a viable choice.

Then there is ZFS, which is used by FreeBSD for good reasons (its previous filesystem UFS is about as good as Ext2) and is also promoted by Ubuntu these days.

Here are some benchmarks. First, the tests were run locally in a VM configured close to the specs of the VPS I was planning to use. Then some benchmarks on the VPS itself.

Tests on a VM

I used these instructions to benchmark Postgresql, it’s a basic benchmark with 10 and 100 clients. However, Btrfs and ZFS were tuned for a database as one would normally do.

• Specs
  • KVM, 2 CPU, 4 Gb RAM, 1 Gb disk
• Linux tuning
  • noatime
  • chattr +C for btrfs
  • ZFS tuned for Postgresql according to the arch wiki
  • No other Postgresql tuning
• FreeBSD tuning
  • Filesystem tuned similarly to Linux but without changing primarycache setting according to
    • https://redbyte.eu/en/blog/postgresql-benchmark-freebsd-centos-ubuntu-debian-opensuse/
    • https://news.ycombinator.com/item?id=16022258
  • ZFS uses lz4 by default

Tests were run using a raw preallocated image on a SSD Btrfs filesystem.

• TPS include connection establishing, rounded, higher is better.
• Latency in ms, lower is better.

10 clients

100 clients

Tests on a VPS

A variety of tests were run:

• Arch Linux using its official cloud image (Btrfs, no compression, has metadata and system DUP, noatime, autodefrag, virtio-block)
• Ubuntu as setup by the VPS provider (virtio-scsi)
• Arch Linux using a custom LVM + Btrfs + Ext4 for database
• Arch Linux with the -lts kernel (no preemption) and Ext4, XFS, Btrfs filesystems

Why Btrfs over LVM? It’s not as crazy as it sounds and it’s mentioned in the btrfs wiki as a possibility. The idea is to combine the best parts of a COW and a traditional filesystem at the cost of extra complexity.

10 clients

100 clients

Conclusion

These are a lot of numbers to absorb, but first a warning: This is a small synthetic test designed to measure a specific aspect of performance, database performance. The real performance of an application will depend on whether it is CPU bound, or network bound, or serving many files or finally the database performance. The only realistic benchmark is the one done on a real application in real conditions.

Having said that:

• Ext4 and XFS are the fastest, as expected. But they come with the smallest set of features compared to newer filesystems.
• Btrfs trails the other options for a database in terms of latency and throughput. But it is reasonably easy to setup and comes with useful features, such as snapshots and easy backups.
• ZFS performs ok, at least on a fresh installation. However I would expect it to get very fragmentated with time, as all COW filesystems do. It also uses a lot of memory which is limited on a VPS. And it’s complicated to setup if you do it yourself.
• The LVM+Btrfs+Ext4 combination performs well, but is not supported by cloud-utils and I had to do the resizing manually.

In other words, every choice is a compromise and one has to pick according to their priorities.

As it turns out, in some cases Go does in fact create dynamically linked executables that expect specific versions of libraries. These are not portable. This happens because Go may use the C compiler even when we do not expect it.

Why does this happen?

$ ldd web
        linux-vdso.so.1 (0x00007fffce3d2000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f0d8f33d000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f0d8f171000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f0d8f385000)

The developer of GoatCounter pointed out that certain standard Go libraries use C bindings to provide extra functionality.

• net
• os/user

When these are used directly or indirectly they have the result of gccgo being used and the executable is dynamically linked by default. However, we can ask them to stay in a pure Go implementation.

Static compilation recipe

go build -tags osusergo
go build -tags netgo
go build -tags osusergo,netgo

One of these build flags will instruct the troubling library to stick to pure Go and give us the static executable we expect.

$ ldd web 
        not a dynamic executable

I think that’s good enough for me but there are options to consider:

CGO_ENABLED=0 go build

This should have the same result. Finally, if we actually do intend to use the C library, we can pass the static compilation flag to the C compiler:

go build -ldflags="-extldflags=-static"

But in that case the executable will not be able to pick up any security updates of the linked libraries. We still have the option of building a dynamically linked executable in a container.