The default configuration values for KVM virtual machines in OpenNebula are conservative and are geared towards generality and ease of initial deployment. As StorPool provides a low-latency and high throughput storage system, to fully utilize its capabilities, some changes and tweaks to those settings are recommended. In addition to the storage-related optimizations we’ll also share some optimizations relating to networking and compute.
Some optimizations could be made by changing the defaults in OpenNebula. Others require replacing some of OpenNebula’s scripts with specialized ones, which in some cases are incompatible with other storage backends.
This article will start with the simplest changes to the default values and then continue with the more advanced optimizations.
Improving the defaults
Using a newer version of qemu-kvm is recommended because of some needed extra features and better performance. For CentOS 7, the use of qemu-kvm-ev is highly recommended. The default version of qemu-kvm in CentOS 7 is not recommended.
The default configuration values are used when there is no variable associated for the given option. Variables could be inherited from the Datastore Template, Image Template, etc. It is recommended to define everything in the VM Template, because these variables have higher priority when a VM is created. This way, a change of a value in the inheritance chain will not affect the VM that is being instantiated from this Template.
Following is a list of recommended changes to the defaults.
Configuration file /etc/one/oned.conf
– DEFAULT_DEVICE_PREFIX = “hd”
+ DEFAULT_DEVICE_PREFIX = “sd”
OpenNebula falls back to IDE emulation as default disk type if not defined otherwise. We recommend the
Configuration file /etc/one/vmm_exec/vmm_exec_kvm.conf
– FEATURES = [ PAE = “no”, ACPI = “yes”, APIC = “no”, HYPERV = “no”, GUEST_AGENT = “no”, VIRTIO_SCSI_QUEUES = “0” ]
+ FEATURES = [ PAE = “no”, ACPI = “yes”, APIC = “no”, HYPERV = “no”, GUEST_AGENT = “yes”, VIRTIO_SCSI_QUEUES = “0” ]
The recent images in the OpenNebula Marketplace have
Configuration file /etc/one/vmm_exec/vmm_exec_kvm.conf
– DISK = [ driver = “raw” , cache = “none”]
+ DISK = [ driver = “raw” , cache = “none” , io = “native” , discard = “unmap” ]
The default block devices created by
– #NIC = [ filter = “clean-traffic”, model=”virtio” ]
+ NIC = [ model=”virtio” ]
When not explicitly configured, libvirt uses an emulated Realtek NIC by default, so we change it to a ‘virtio’ NIC, which is much better.
In configuration file /var/lib/one/remotes/etc/vmm/kvm/kvmrc
– #DEFAULT_ATTACH_CACHE=none
+ DEFAULT_ATTACH_CACHE=none
– #DEFAULT_ATTACH_DISCARD=unmap
+ DEFAULT_ATTACH_DISCARD=unmap
– #DEFAULT_ATTACH_IO=
+ DEFAULT_ATTACH_IO=native
– #DEFAULT_ATTACH_NIC_MODEL=virtio
+ DEFAULT_ATTACH_NIC_MODEL=virtio
When hot-attaching disks and NICs, the default values are sourced from this file, so we set them here too.
Extending OpenNebula’s functionality
KVM Deploy-tweaks
The following optimizations are performed by replacing OpenNebula’s default KVM script to deploy VMs with an alternative one. Our deploy-tweaks script provides a simple platform for further tweaking of the VM’s configuration file. The deploy-tweaks script is enabled using OpenNebula’s option to call a “local” version of the VMM_MAD instead of the default one:
– ARGUMENTS = “-t 15 -r 0 kvm”,
+ ARGUMENTS = “-t 15 -r 0 kvm -l deploy=deploy-tweaks”,
Deploy-tweaks is a wrapper which accept the domain XML from OpenNebula, collects the VM’s metadata as XML and calls sequentially helper scripts from the `/var/lib/one/vmm/kvm/deploy-tweaks.d/` directory. Each helper may alter the VM’s domain XML and optionally look up configuration variables in the VM’s metadata XML.
The following deploy-tweak hook scripts are currently implemented:
scsi-queues.py
We have observed that optimal performance is achieved when the number of virtio-scsi queues matches the number of vCPUs. OpenNebula has the option to define VIRTIO_SCSI_QUEUES, but it is hard to enforce it for each VM, so we are using a helper script to set the recommended number of queues in the virtio-scsi controllers.
iothreads.py
By default qemu submits block IO to the host OS from its main thread. It has the option to submit block IO from a dedicated IO thread. This is known as the “iothreads” option. In this way the VM’s disks could be used without having to wait for other events in qemu’s global queue. This helper script defines a global iothread and assigns each virtio-blk disk and virtio-scsi controller to it.
volatile2dev.py
OpenNebula assumes that all block devices in the SYSTEM Datastore are files, including any volatile disks and context images, and configures them as files in the VM’s domain XML. When addon-storpool is used for the SYSTEM Datastore, there are only symlinks to block devices instead of the usual local files.
Recent versions of libvirt have compatibility checks which prevent live VM migration when there are files which are not on a shared filesystem. An “–unsafe” option exists that allows the migration, but other compatibility checks are disabled too.
This helper script alters the domain XML configuration, defining the volatile disks as block devices, which enables live migration of VMs with volatile disks.
hyperv-clock.py
In addition to the OpenNebula’s default HYPERV_OPTIONS this helper script creates/appends a clock element with additional timers. This provides a noticeable performance boost to Windows VMs.
cpu.py
The primary use of this script is to define the CPU topology for the guest VM. To enable this functionality two variables in the VM’s USER_TEMPLATE must be defined:
- ‘T_CPU_THREADS’ defines the number of threads per core;
- ‘T_CPU_SOCKETS’ defines the number of virtual sockets (NUMA nodes).
The recommended values with even number of VCPUs are T_CPU_THREADS=2, T_CPU_SOCKETS=2. This configuration will create a vCPU topology close to a real processor.
The helper script could tweak other elements of the CPU definition that can be used to simulate more complex CPU topologies.
Disk Snapshots and VM Snapshots
OpenNebula has an API call to create a disk snapshot that could be used for simple backups.
A VM with multiple disks with strictly separate purpose could be backed up with disk snapshots issued disk by disk. This will not work for snapshotting a complex VM disk configuration, like a VM which actively uses multiple virtual disks simultaneously, for example one disk may be used for a database journal file, while another disk for tables and indexes. In this case we need snapshots consistent across multiple disks.
OpenNebula also exposes libvirt’s interface to take snapshots of the entire VM, however this doesn’t work for VMs on any type of storage other than qcow. We reconfigure the OpenNebula “VM snapshot” interface to perform an atomic snapshot of all of VM’s disks, which results in crash-consistent snapshot. In addition it is also possible to create application-consistent snapshots by using qemu-guest-agent.
To enable the alternate VM Snapshot functionality, the following is configured in KVM’s VMM_MAD:
+ ARGUMENTS = “-t 15 -r 0 kvm -l deploy=deploy-tweaks,snapshotcreate=snapshot_create-storpool,snapshotrevert=snapshot_revert-storpool,snapshotdelete=snapshot_delete-storpool”
Please note that after recovering a VM from a VM Snapshot it is recommended to power cycle the VM from OpenNebula, because the recovery script cannot handle complex networking (re)configurations.
Are you looking for block storage for your OpenNebula and KVM environment? StorPool is the preferred choice for any OpenNebula cloud!
With StorPool, OpenNebula users get exceptional storage bandwidth, IOPS and low latency, enabling companies to provision many more VMs per host. Combining both products allows for seamless scalability in terms of capacity and performance, as well as increased reliability.
Enter Your Text
If you want to know more about how StorPool and OpenNebula work together seamlessly, book a short meeting with one of our experts.