How to divide the network traffic to have a minimal effect on your storage in case of a DDoS attack
One of our customers had a rather unpleasant experience recently with a few nasty DDoS attacks. The case we had was that the VM networking and the storage traffic were both on the same L2 link, so one could interfere with the other. When setting up the site after doing many observations we had concluded that the network utilization was pretty low, therefore we had decided to use the same network for the storage as well. Currently, with the storage traffic included, the average utilization is below 1% on the dual 10GE links on each node – pretty low indeed.
Here comes the interesting part, though. Even if there is a DDoS attack, the links barely notice it – the network utilization very rarely goes up above 50% and even in extreme cases only as high as 80%. Even so, when we experienced such attacks, the storage network would suffer quite a lot. So if the network could handle more, we had to look for the bottleneck somewhere else.
Not surprisingly, we found it in the CPU. The reason for the behavior was a rather peculiar combination of factors. First of all, we pin our processes on specific cores of the CPU to achieve greater performance and predictable behavior of our services. One reason this has proven to be a good practice is that if a process is not pinned, the kernel may decide at any time to change the core on which that process is being executed, and that is quite an expensive operation. The other reason is that we group some of our processes in such a way that they have quick access to related structures of data stored in the caches of the cores.
Apart from process pinning we also pin the network card interrupts for mostly the same reasons. The additional benefit here is we specify which processor (for multiprocessor machines) handles the network interrupts. We take care to set those interrupts on the processor that the PCI device (e.g. network card) is attached to, in order to avoid expensive transfers of structures to the other processor where possible.
All this is fine but what more could one do? Obviously, a DDoS attack to a VM would saturate the core that we had pinned the network interrupts to and would cripple the performance of one of our processes (related to networking) pinned on the same core (or its hyperthreading virtual core). This would then hinder the operation of the storage on that node.
So a good practice is to separate the processing of VM traffic interrupts and storage network interrupts. Even though we use VLANs to separate our traffic from VM traffic, the interrupts still go to the same place. So what we came up with is the use of Virtual Functions of the NICs (VFs). VFs are a clever way to present a single network interface as several distinct devices with their own configuration parameters (MAC addresses, queue lengths, etc) and their own separate interrupts visible to the OS. By enabling a VF you end up with one PF (Physical Function; think of this as the main interface, let’s say eth1, the one you already had) and a VF (a new interface with its own configuration parameters, let’s say eth1_vf0).
We used VFs to assign one for the storage network and leave the PF for the virtual machines and host networking, effectively separating interrupts for the storage and the VM/host traffic. We, therefore, could pin the VF interrupts to where we needed them for optimal storage operation and assign the VM traffic interrupts to some other cores that have nothing to do with the storage.
The results were astonishing. A heavy DDoS attack to the VMs did not interfere even in the slightest with the storage network. There were no measurable delays in storage operations whatsoever. We tested this configuration with both Mellanox and Intel cards. They had entirely different procedures for enabling VF support and the resulting features with enabled VFs were different (e.g. the number of queues was 4 per VF with Mellanox and 2 with Intel). Even so, the resulting benefits in this case, where we needed the separation, were equivalent so we deemed this experiment a success.
If you have any questions feel free to contact us at info@storpool.slm.dev
Share this Post