Linux scheduling
Based on the hardware layout of the physical cores, the Linux® scheduler maintains hierarchically ordered scheduling domains.
Basic scheduling domains consist of those processes that are run on physically adjacent cores, such as the cores on the same chip. Higher level scheduling domains group physically adjacent scheduling domains, such as the chips on the same book.
The Linux scheduler is a multi-queue scheduler, which means that for each of the logical host CPUs, there is a run queue of processes waiting for this CPU. Each virtual CPU waits for its execution in one of these run queues.
Moving a virtual CPU from one run queue to another is called a (CPU)
migration. Be sure not to confuse the term CPU migration
with
a live migration
, which is the migration of a virtual server from one host to another. The Linux scheduler might decide to
migrate a virtual CPU when the estimated wait time until the virtual
CPU will be executed is too long, the run queue where it is supposed
to be waiting is full, or another run queue is empty and needs to
be filled up.
Migrating a virtual CPU within the same scheduling domain is less cost intensive than to a different scheduling domain because of the caches being moved from one core to another. The Linux scheduler has detailed information about the migration costs between different scheduling domains or CPUs. Migration costs are an important factor for the decision if the migration of a virtual CPU to another host CPU is valuable.
libvirt provides means to assign virtual CPUs to groups of host CPUs in order to minimize migration costs. This process is called CPU pinning. CPU pinning forces the Linux scheduler to migrate virtual CPUs only between those host CPUs of the specified group. Likewise, the execution of the user space process or I/O threads can be assigned to groups of host CPUs.
- CPU pinning can lead to the opposite effect of what was desired when the circumstances for which it was designed change. This may occur, for example, when the host reboots, the workload on the host changes, or the virtual servers are modified.
- Deactivating operating CPUs and activating standby CPUs (CPU hotplug) on the host may lead to a situation where host CPUs are no longer available for the execution of virtual server threads after their reactivation.