OpenFest Game

StorPool’s Quiz at OpenFest 2018 – A true story from the life of an administrator

In the last few years, StorPool attends OpenFest. To make our stall a bit more interesting and to challenge the IT experts, we create games for the attendees.  This year we had two – a “server lifting challenge” and “fix the problems challenge”. As a result of the huge interest to the “Fix the problem” StorPool’s Quiz at OpenFest 2018 – A true story from the life of an administrator

Read More
Silent data corruption Linux Kernel

Beware: Silent data corruption discovered in Linux kernels 4.10-4.17

While debugging an issue in a new StorPool deployment we re-discovered a kernel bug that exists in some of the latest Linux kernels and causes silent data corruption. The setup is a virtualized environment where the kernel containing the bug was running on the host. The guests were KVM virtual machines with io=threads (the default Beware: Silent data corruption discovered in Linux kernels 4.10-4.17

Read More
The weird interactions of cgroups and linux page cache in hypervisor environments

The weird interactions of cgroups and linux page cache in hypervisor environments

Here’s one case we’ve spent some time debugging, related to StorPool, VMs, cgroups, OOM (out-of-memory) killer and caching. Some of it should be useful to a lot of sysadmins out there. The issue started with a customer complaining: > It’s happening again, on this hypervisor VMs are being killed > by the OOM killer. This The weird interactions of cgroups and linux page cache in hypervisor environments

Read More
Serial Console Pros and Cons

Why does StorPool recommend against serial console on production installations?

TL;DR – try ‘echo t > /proc/sysrq-trigger‘ and see what happens. The serial console and frame buffer are two separate ways for the kernel to display text, either to a remote machine or on the local display. They both suffer from a very similar problem, that they block the CPU they run on for long Why does StorPool recommend against serial console on production installations?

Read More