Summary

There are 2 ways to hotplug CPU in QEMU:

dedicated legacy interface: cpu-add QMP command
generic device-add/device-del interface for hot-(un)plugging CPUs

Owner

Name: Igor Mammedov
Email: imammedo@redhat.com

cpu-add interface

Summary

{ 'command': 'cpu-add', 'data': {'id': 'int'} }

ID - a number in range [0..max-cpus)
Available since: 1.5
Supported targets: i386-softmmu, x86_64-softmmu, s390x (since 2.6)

Description

Command is a legacy solution for CPU hot-add and gives a simplified interface for it. It provides an opportunity to implement the feature for targets that currently can't implement CPU hot-add using device_add command due to their present design. Later targets that implement it could rewrite it to become a wrapper over device_add when it becomes usable for target.

Usage example

1. start QEMU with QMP socket available and with startup amount of CPUs less than maxcpus

./qemu-system-x86_64 -qmp unix:/tmp/qmp-sock,server,nowait -smp 1,maxcpus=4

2. Connect to QMP socket using qmp-shell command

./QMP/qmp-shell /tmp/qmp-sock

3. Add CPUs issuing cpu-add command in qmp-shell command prompt

cpu-add id=1

4. Optionally online newly added CPU inside guest

Linux kernel doesn't online hot-added CPUs automatically. Once CPU is hot-added it should be onlined using an appropriate udev script or manually by issuing a following command:

echo 1 > /sys/devices/system/cpu/cpu1/online

Sample udev script: Add the following line to /etc/udev/rules.d/99-hotPlugCPU.rules

SUBSYSTEM=="cpu",ACTION=="add",RUN+="/bin/sh -c '[ ! -e /sys$devpath/online ] || echo 1 > /sys$devpath/online'"

Current limitations

migration target should be started with initial CPU count '-smp XX' that includes hot-added CPUs on migration source side.
CPU shouldn't be hot-plugged during migration.
adding CPUs should be done in successive order from lower to higher IDs in [0..max-cpus) range.
It's possible to add arbitrary CPUs in random order, however that would cause migration to fail on its target side.

device_add/device_del interface

Summary

Lists possible to hotplug CPUs:
{ 'command': 'query-hotpluggable-cpus', 'returns': ['HotpluggableCPU'] }

CPU hot-add:
{ 'command': 'device_add', 'data': {'driver': 'str', 'id': 'str', ... }}

mandatory properties for every CPU:
- driver: cpu model type name
- id: unique device name
target/configuration dependent properties
- socket-id: socket number in range [0..max sockets)
- core-id: core number in range [0..max cores)
- thread-id: thread-id in range [..max threads)
- node-id: NUMA node ID the CPU belongs to

CPU hot-remove:
{ 'command': 'device_del', 'data': {'id': 'str'} }

id: device name that has been used for CPU hotplug with device_add command or -device CLI option

Supported targets:

i386, x86_64 (since 2.7)
spapr (since 2.7)

Description

When hot-plugging a CPU with device_add command, user must specify which CPU instance it needs to hotplug. To help user specify a particular CPU instance target/machine type that supports generic device_add/device_del interface for hot-adding/removing CPUs must implement query-hotpluggable-cpus QMP command. User can query the capability using query-machines QMP command and check if property hotpluggable-cpus is set to true. If machine reports that hotpluggable-cpus are supported, user can use query-hotpluggable-cpus command to list possible to hotplug CPUs with list of necessary to hotplug properties.
For example with following CLI:

qemu-system-x86_64 -smp 1,maxcpus=2 -qmp unix:/tmp/q,server,nowait

user would get following

qmp-shell -p /tmp/q
Welcome to the QMP low-level shell!
Connected to QEMU 2.8.50
(QEMU) query-hotpluggable-cpus
{
   "return": [
       {
           "type": "qemu64-x86_64-cpu", 
           "vcpus-count": 1, 
           "props": {
               "socket-id": 1, 
               "core-id": 0, 
               "thread-id": 0
           }
       }, 
       {
           "qom-path": "/machine/unattached/device[0]", 
           "type": "qemu64-x86_64-cpu", 
           "vcpus-count": 1, 
           "props": {
               "socket-id": 0, 
               "core-id": 0, 
               "thread-id": 0
           }
       }
   ]
}

list of present and possible CPUs for given at startup -cpu and -smp CLI options. Where

-cpu cpu_model is translated to corresponding CPU type name that could be used with device_add as driver property value
-smp n,sockets=x,cores=y,threads=z,maxcpus=m is translated to a machine dependent set of present/possible to hotplug CPUs, where per CPU props list provides a set of property/value pairs necessary to hotplug given CPU instance with device_add command.
qom-path is path to present CPU which is emplty for possible to hot-add CPU entries

Now user having list of necessary properties can hot-add a CPU using either monitor or QMP interface, for above example, the command to hot-add the second CPU would look like:

qmp-shell -p /tmp/q
Welcome to the QMP low-level shell!
Connected to QEMU 2.8.50
(QEMU) device_add id=cpu2 driver=qemu64-x86_64-cpu socket-id=1 core-id=0 thread-id=0
{
    "return": {}
}

CPU hot-remove could be done using device_del command like with any other device:

qmp-shell -p /tmp/q
Welcome to the QMP low-level shell!
Connected to QEMU 2.8.50
(QEMU) device_del id=cpu2
{
    "return": {}
}

However note that CPU hot-remove is machine dependent and requires guest cooperation. device_del command does not guarantee CPU removal to actually happen, typically it's a request forwarded to guest using target dependent mechanism.

x86 uses ACPI to request removal from guest OS

Once guest is freed being removed CPU from use guest ejects CPU using target dependent mechanism and only at that time CPU is actually removed in QEMU