Features/CPUHotplug

Summary

There are 2 ways to hotplug CPU in QEMU:

• dedicated legacy interface: cpu-add QMP command
• generic device-add/device-del interface for hot-(un)plugging CPUs

Owner

• Name: Igor Mammedov
• Email: imammedo@redhat.com

cpu-add interface

Summary

{ 'command': 'cpu-add', 'data': {'id': 'int'} }

• ID - a number in range [0..max-cpus)
  
• Available since: 1.5
  
• Supported targets: i386-softmmu, x86_64-softmmu, s390x (since 2.6)

Description

Command is a legacy solution for CPU hot-add and gives a simplified interface for it. It provides an opportunity to implement the feature for targets that currently can't implement CPU hot-add using device_add command due to their present design. Later targets that implement it could rewrite it to become a wrapper over device_add when it becomes usable for target.

Usage example

1. start QEMU with QMP socket available and with startup amount of CPUs less than maxcpus

./qemu-system-x86_64 -qmp unix:/tmp/qmp-sock,server,nowait -smp 1,maxcpus=4

2. Connect to QMP socket using qmp-shell command

./QMP/qmp-shell /tmp/qmp-sock

3. Add CPUs issuing cpu-add command in qmp-shell command prompt

cpu-add id=1

4. Optionally online newly added CPU inside guest

Linux kernel doesn't online hot-added CPUs automatically. Once CPU is hot-added it should be onlined using an appropriate udev script or manually by issuing a following command:

echo 1 > /sys/devices/system/cpu/cpu1/online

Sample udev script: Add the following line to /etc/udev/rules.d/99-hotPlugCPU.rules

SUBSYSTEM=="cpu",ACTION=="add",RUN+="/bin/sh -c '[ ! -e /sys$devpath/online ] || echo 1 > /sys$devpath/online'"

Current limitations

1. migration target should be started with initial CPU count '-smp XX' that includes hot-added CPUs on migration source side.
2. CPU shouldn't be hot-plugged during migration.
3. adding CPUs should be done in successive order from lower to higher IDs in [0..max-cpus) range.
   It's possible to add arbitrary CPUs in random order, however that would cause migration to fail on its target side.

device_add/device_del interface

Summary

Lists possible to hotplug CPUs:
{ 'command': 'query-hotpluggable-cpus', 'returns': ['HotpluggableCPU'] }

CPU hot-add:
{ 'command': 'device_add', 'data': {'driver': 'str', 'id': 'str', ... }}

• mandatory properties for every CPU:
  • driver: cpu model type name
  • id: unique device name
• target/configuration dependent properties
  • socket-id: socket number in range [0..max sockets)
  • core-id: core number in range [0..max cores)
  • thread-id: thread-id in range [..max threads)
  • node-id: NUMA node ID the CPU belongs to

CPU hot-remove:
{ 'command': 'device_del', 'data': {'id': 'str'} }

• id: device name that has been used for CPU hotplug with device_add command or -device CLI option

Supported targets:

• i386, x86_64 (since 2.7)
• spapr (since 2.7)

Description

Target/machine type that supports generic device_add/device_del interface for hot-adding/removing CPUs must implement query-hotpluggable-cpus QMP command. User can query the capability using query-machines QMP command and check if property hotpluggable-cpus is set to true. If machine reports that hotpluggable-cpus are supported, user can use query-hotpluggable-cpus command to list possible to hotplug CPUs with list of necessary to hotplug properties. For example for following CLI:

qemu-system-x86_64 -smp 1,maxcpus=4 -qmp unix:/tmp/q,server,nowait

user would get following result:

qmp-shell -p /tmp/q
Welcome to the QMP low-level shell!
Connected to QEMU 2.8.50
(QEMU) query-hotpluggable-cpus
{
   "return": [
       {
           "type": "qemu64-x86_64-cpu", 
           "vcpus-count": 1, 
           "props": {
               "socket-id": 1, 
               "core-id": 0, 
               "thread-id": 0
           }
       }, 
       {
           "qom-path": "/machine/unattached/device[0]", 
           "type": "qemu64-x86_64-cpu", 
           "vcpus-count": 1, 
           "props": {
               "socket-id": 0, 
               "core-id": 0, 
               "thread-id": 0
           }
       }
   ]
}