Features/CPUHotplug

Summary

There are 2 ways to hotplug CPU in QEMU:

• dedicated legacy interface: cpu-add QMP command (removed in QEMU v5.2)
• generic device-add/device-del interface for hot-(un)plugging CPUs

Owner

• Name: Igor Mammedov
• Email: imammedo@redhat.com

cpu-add interface

Note: The cpu-add interface has been removed starting with QEMU v5.2. Use the device_add interface instead (see below).

Summary

{ 'command': 'cpu-add', 'data': {'id': 'int'} }

• ID - a number in range [0..max-cpus)
  
• Available since: 1.5
  
• Supported targets: i386-softmmu, x86_64-softmmu, s390x (since 2.6)

Description

Command is a legacy solution for CPU hot-add and gives a simplified interface for it. It provides an opportunity to implement the feature for targets that currently can't implement CPU hot-add using device_add command due to their present design. Later targets that implement it could rewrite it to become a wrapper over device_add when it becomes usable for target.

Usage example

1. start QEMU with QMP socket available and with startup amount of CPUs less than maxcpus

./qemu-system-x86_64 -qmp unix:/tmp/qmp-sock,server=on,wait=off -smp 1,maxcpus=4

2. Connect to QMP socket using qmp-shell command

./QMP/qmp-shell /tmp/qmp-sock

3. Add CPUs issuing cpu-add command in qmp-shell command prompt

cpu-add id=1

4. Optionally online newly added CPU inside guest

Linux kernel doesn't online hot-added CPUs automatically. Once CPU is hot-added it should be onlined using an appropriate udev script or manually by issuing a following command:

echo 1 > /sys/devices/system/cpu/cpu1/online

Sample udev script: Add the following line to /etc/udev/rules.d/99-hotPlugCPU.rules

SUBSYSTEM=="cpu",ACTION=="add",RUN+="/bin/sh -c '[ ! -e /sys$devpath/online ] || echo 1 > /sys$devpath/online'"

device_add/device_del interface

Summary

Lists possible to hotplug CPUs:
{ 'command': 'query-hotpluggable-cpus', 'returns': ['HotpluggableCPU'] }

CPU hot-add:
{ 'command': 'device_add', 'data': {'driver': 'str', 'id': 'str', ... }}

• mandatory properties for every CPU:
  • driver: cpu model type name
  • id: unique device name
• target/configuration dependent properties
  • socket-id: socket number in range [0..max sockets)
  • core-id: core number in range [0..max cores)
  • thread-id: thread-id in range [..max threads)
  • node-id: NUMA node ID the CPU belongs to

CPU hot-remove:
{ 'command': 'device_del', 'data': {'id': 'str'} }

• id: device name that has been used for CPU hotplug with device_add command or -device CLI option

Supported targets:

• i386, x86_64 (since 2.7)
• spapr (since 2.7)

Description

When hot-plugging a CPU with device_add command, user must specify which CPU instance it needs to hotplug. To help user specify a particular CPU instance target/machine type that supports generic device_add/device_del interface for hot-adding/removing CPUs must implement query-hotpluggable-cpus QMP command. User can query the capability using query-machines QMP command and check if property hotpluggable-cpus is set to true. If machine reports that hotpluggable-cpus are supported, user can use query-hotpluggable-cpus command to list possible to hotplug CPUs with list of necessary to hotplug properties.
For example with following CLI:

qemu-system-x86_64 -smp 1,maxcpus=2 -qmp unix:/tmp/q,server=on,wait=off

user would get following

qmp-shell -p /tmp/q
Welcome to the QMP low-level shell!
Connected to QEMU 2.8.50
(QEMU) query-hotpluggable-cpus
{
   "return": [
       {
           "type": "qemu64-x86_64-cpu", 
           "vcpus-count": 1, 
           "props": {
               "socket-id": 1, 
               "core-id": 0, 
               "thread-id": 0
           }
       }, 
       {
           "qom-path": "/machine/unattached/device[0]", 
           "type": "qemu64-x86_64-cpu", 
           "vcpus-count": 1, 
           "props": {
               "socket-id": 0, 
               "core-id": 0, 
               "thread-id": 0
           }
       }
   ]
}

list of present and possible CPUs for given at startup -cpu and -smp CLI options. Where

• -cpu cpu_model is translated to corresponding CPU type name that could be used with device_add as driver property value
• -smp n,sockets=x,cores=y,threads=z,maxcpus=m is translated to a machine dependent set of present/possible to hotplug CPUs, where per CPU props list provides a set of property/value pairs necessary to hotplug given CPU instance with device_add command.
• qom-path is path to present CPU which is emplty for possible to hot-add CPU entries
  

Now user having list of necessary properties can hot-add a CPU using either monitor or QMP interface, for above example, the command to hot-add the second CPU would look like:

qmp-shell -p /tmp/q
Welcome to the QMP low-level shell!
Connected to QEMU 2.8.50
(QEMU) device_add id=cpu2 driver=qemu64-x86_64-cpu socket-id=1 core-id=0 thread-id=0
{
    "return": {}
}

CPU hot-remove could be done using device_del command like with any other device:

qmp-shell -p /tmp/q
Welcome to the QMP low-level shell!
Connected to QEMU 2.8.50
(QEMU) device_del id=cpu2
{
    "return": {}
}

Note: CPU hot-remove is machine dependent and requires guest cooperation. device_del command does not guarantee CPU removal to actually happen, typically it's a request forwarded to guest using target dependent mechanism.

• x86 uses ACPI to request removal from guest OS

once guest is freed being removed CPU from use guest ejects it using target dependent mechanism and only at that time CPU is actually removed in QEMU and DEVICE_DELETED QMP event is sent to management application.

Migration considerations

cpu-add

• migration target should be started with initial CPU count '-smp XX' that includes hot-added CPUs on migration source side.
• CPU shouldn't be hot-plugged during migration.
• adding CPUs should be done in successive order from lower to higher IDs in [0..max-cpus) range.
  It's possible to add arbitrary CPUs in random order, however that would cause migration to fail on its target side.

device_add/device_del

• '-smp xx,...' on target side of migration must be the same as on source side
• hot-added CPUs must be added to the end of CLI on target side of migration using '-device' options with the same properties that were used for hotplugging CPUs.
• CPU can be hot-added in any order however it's recommended to use the same order when declaring hot-added CPUs on target side of migration with -device
• if earlier hot-added CPU where successfully hot-removed on source side, it shouldn't be present on target side of migration
• hot-added CPUs declarations on target side shall set hotplugged property to on, i.e. if CPU has been hot-added with command

device_add driver=qemu64-x86_64-cpu socket-id=1 core-id=0 thread-id=0 id=cpu2

QEMU implicitly sets hotplugged property of that CPU to on however on target side user shall explicitly set it as it's off for -device created devices by default. i.e. CPU from above example on target side should be declared as following

-device qemu64-x86_64-cpu,socket-id=1,core-id=0,thread-id=0,id=cpu2,hotplugged=on