Most VPS hosting providers use Ext4 or XFS out of the box. There are other options of course and interestingly, the official Arch Linux cloud image uses Btrfs, with zstd compression. I like Btrfs and I think it’s a good match for a rolling release distribution targeted at developers such as Arch (and OpenSuse and recently Fedora).
- Snapshots to easily rollback when/if an update goes wrong
- Easy backups
- Container friendly
- Weak random read/write performance when used for a database or a virtual machine host
For development, the pros outweigh the cons and I wouldn’t mind the extra features on a web server. However in that case database performance is often the limiting factor. So I was curious, not whether Btrfs would outperform Ext4 and XFS but rather if it’s a viable choice.
Then there is ZFS, which is used by FreeBSD for good reasons (its previous filesystem UFS is about as good as Ext2) and is also promoted by Ubuntu these days.
Here are some benchmarks. First, the tests were run locally in a VM configured close to the specs of the VPS I was planning to use. Then some benchmarks on the VPS itself.
Tests on a VM
I used these instructions to benchmark Postgresql, it’s a basic benchmark with 10 and 100 clients. However, Btrfs and ZFS were tuned for a database as one would normally do.
- KVM, 2 CPU, 4 Gb RAM, 1 Gb disk
- Linux tuning
- chattr +C for btrfs
- ZFS tuned for Postgresql according to the arch wiki
- No other Postgresql tuning
- FreeBSD tuning
- Filesystem tuned similarly to Linux but without changing primarycache setting according to
- ZFS uses lz4 by default
Tests were run using a raw preallocated image on a SSD Btrfs filesystem.
- TPS include connection establishing, rounded, higher is better.
- Latency in ms, lower is better.
Tests on a VPS
A variety of tests were run:
- Arch Linux using its official cloud image (Btrfs, no compression, has metadata and system DUP, noatime, autodefrag, virtio-block)
- Ubuntu as setup by the VPS provider (virtio-scsi)
- Arch Linux using a custom LVM + Btrfs + Ext4 for database
- Arch Linux with the -lts kernel (no preemption) and Ext4, XFS, Btrfs filesystems
Why Btrfs over LVM? It’s not as crazy as it sounds and it’s mentioned in the btrfs wiki as a possibility. The idea is to combine the best parts of a COW and a traditional filesystem at the cost of extra complexity.
|Arch LVM + btrfs + ext4||virtio-scsi||13-15||665-745|
|Arch lts ext4||virtio-scsi||17||568-583|
|Arch lts xfs||virtio-scsi||15||648-659|
|Arch lts btrfs||virtio-scsi||21||460-465|
|Arch btrfs (single metadata)||virtio-scsi||20||505|
|Arch LVM + btrfs + ext4||virtio-scsi||178-179||556-561 tps|
|Arch lts ext4||virtio-scsi||182-183||543-548|
|Arch lts xfs||virtio-scsi||184-190||523-542|
|Arch lts btrfs||virtio-scsi||240-254||392-416|
|Arch btrfs (single metadata)||virtio-scsi||243||410|
These are a lot of numbers to absorb, but first a warning: This is a small synthetic test designed to measure a specific aspect of performance, database performance. The real performance of an application will depend on whether it is CPU bound, or network bound, or serving many files or finally the database performance. The only realistic benchmark is the one done on a real application in real conditions.
Having said that:
- Ext4 and XFS are the fastest, as expected. But they come with the smallest set of features compared to newer filesystems.
- Btrfs trails the other options for a database in terms of latency and throughput. But it is reasonably easy to setup and comes with useful features, such as snapshots and easy backups.
- ZFS performs ok, at least on a fresh installation. However I would expect it to get very fragmentated with time, as all COW filesystems do. It also uses a lot of memory which is limited on a VPS. And it’s complicated to setup if you do it yourself.
- The LVM+Btrfs+Ext4 combination performs well, but is not supported by
cloud-utilsand I had to do the resizing manually.
In other words, every choice is a compromise and one has to pick according to their priorities.
2021-08-19 10:46 (Last updated: 2021-09-24 18:41)