Features/CPUHotplug: Difference between revisions

From QEMU
Line 79: Line 79:


== Description ==
== Description ==
Target/machine type that supports generic device_add/device_del interface for hot-adding/removing CPUs must implement
When hot-plugging a CPU with device_add command, user must specify which CPU instance it needs to hotplug. To help user specify a particular CPU instance target/machine type that supports generic device_add/device_del interface for hot-adding/removing CPUs must implement '''''query-hotpluggable-cpus''''' QMP command.
'''''query-hotpluggable-cpus
User can query the capability using '''''query-machines''''' QMP command and check if property ''hotpluggable-cpus'' is set to true. If machine reports that ''hotpluggable-cpus'' are supported, user can use '''''query-hotpluggable-cpus''''' command to list possible to hotplug CPUs with list of necessary to hotplug properties.<br/>
QMP command. User can query the capability using '''''query-machines
For example with following CLI:<br/>
QMP command and check if property ''hotpluggable-cpus
  qemu-system-x86_64 -smp 1,maxcpus=2 -qmp unix:/tmp/q,server,nowait
is set to true. If machine reports that ''hotpluggable-cpus
are supported, user can use '''''query-hotpluggable-cpus
command to list possible to hotplug CPUs with list of necessary to hotplug properties.
For example for following CLI:<br/>
  qemu-system-x86_64 -smp 1,maxcpus=4 -qmp unix:/tmp/q,server,nowait
user would get following
user would get following
  qmp-shell -p /tmp/q
  qmp-shell -p /tmp/q
Line 119: Line 114:
list of present and possible CPUs for given at startup ''-cpu'' and ''-smp'' CLI options. Where  
list of present and possible CPUs for given at startup ''-cpu'' and ''-smp'' CLI options. Where  
*''-cpu cpu_model'' is translated to corresponding CPU ''type'' name that could be used with device_add as ''driver'' property value
*''-cpu cpu_model'' is translated to corresponding CPU ''type'' name that could be used with device_add as ''driver'' property value
*''-smp n,sockets=x,cores=y,threads=z,maxcpus=m'' is translated to a machine dependent set of possible to hotplug CPUs, where per CPU ''props'' list provides a set of property/value pairs necessary to hotplug given CPU instance with device_add command.
*''-smp n,sockets=x,cores=y,threads=z,maxcpus=m'' is translated to a machine dependent set of present/possible to hotplug CPUs, where per CPU ''props'' list provides a set of property/value pairs necessary to hotplug given CPU instance with device_add command.
*''qom-path'' is path to present CPU which is emplty for possible to hot-add CPU entries<br>
Now user having list of necessary properties can hot-add CPU using either monitor or QMP interface, for above example command to hot-add the second cpu would look like:
qmp-shell -p /tmp/q
Welcome to the QMP low-level shell!
Connected to QEMU 2.8.50
(QEMU) device_add id=cpu2 driver=qemu64-x86_64-cpu socket-id=1 core-id=0 thread-id=0
{
    "return": {}
}

Revision as of 15:04, 9 January 2017

Summary

There are 2 ways to hotplug CPU in QEMU:

  • dedicated legacy interface: cpu-add QMP command
  • generic device-add/device-del interface for hot-(un)plugging CPUs

Owner

  • Name: Igor Mammedov
  • Email: imammedo@redhat.com

cpu-add interface

Summary

{ 'command': 'cpu-add', 'data': {'id': 'int'} }

  • ID - a number in range [0..max-cpus)
  • Available since: 1.5
  • Supported targets: i386-softmmu, x86_64-softmmu, s390x (since 2.6)

Description

Command is a legacy solution for CPU hot-add and gives a simplified interface for it. It provides an opportunity to implement the feature for targets that currently can't implement CPU hot-add using device_add command due to their present design. Later targets that implement it could rewrite it to become a wrapper over device_add when it becomes usable for target.

Usage example

1. start QEMU with QMP socket available and with startup amount of CPUs less than maxcpus

./qemu-system-x86_64 -qmp unix:/tmp/qmp-sock,server,nowait -smp 1,maxcpus=4

2. Connect to QMP socket using qmp-shell command

./QMP/qmp-shell /tmp/qmp-sock

3. Add CPUs issuing cpu-add command in qmp-shell command prompt

cpu-add id=1

4. Optionally online newly added CPU inside guest

Linux kernel doesn't online hot-added CPUs automatically. Once CPU is hot-added it should be onlined using an appropriate udev script or manually by issuing a following command:

echo 1 > /sys/devices/system/cpu/cpu1/online

Sample udev script: Add the following line to /etc/udev/rules.d/99-hotPlugCPU.rules

SUBSYSTEM=="cpu",ACTION=="add",RUN+="/bin/sh -c '[ ! -e /sys$devpath/online ] || echo 1 > /sys$devpath/online'"


Current limitations

  1. migration target should be started with initial CPU count '-smp XX' that includes hot-added CPUs on migration source side.
  2. CPU shouldn't be hot-plugged during migration.
  3. adding CPUs should be done in successive order from lower to higher IDs in [0..max-cpus) range.
    It's possible to add arbitrary CPUs in random order, however that would cause migration to fail on its target side.

device_add/device_del interface

Summary

Lists possible to hotplug CPUs:
{ 'command': 'query-hotpluggable-cpus', 'returns': ['HotpluggableCPU'] }

CPU hot-add:
{ 'command': 'device_add', 'data': {'driver': 'str', 'id': 'str', ... }}

  • mandatory properties for every CPU:
    • driver: cpu model type name
    • id: unique device name
  • target/configuration dependent properties
    • socket-id: socket number in range [0..max sockets)
    • core-id: core number in range [0..max cores)
    • thread-id: thread-id in range [..max threads)
    • node-id: NUMA node ID the CPU belongs to

CPU hot-remove:
{ 'command': 'device_del', 'data': {'id': 'str'} }

  • id: device name that has been used for CPU hotplug with device_add command or -device CLI option


Supported targets:

  • i386, x86_64 (since 2.7)
  • spapr (since 2.7)

Description

When hot-plugging a CPU with device_add command, user must specify which CPU instance it needs to hotplug. To help user specify a particular CPU instance target/machine type that supports generic device_add/device_del interface for hot-adding/removing CPUs must implement query-hotpluggable-cpus QMP command. User can query the capability using query-machines QMP command and check if property hotpluggable-cpus is set to true. If machine reports that hotpluggable-cpus are supported, user can use query-hotpluggable-cpus command to list possible to hotplug CPUs with list of necessary to hotplug properties.
For example with following CLI:

qemu-system-x86_64 -smp 1,maxcpus=2 -qmp unix:/tmp/q,server,nowait

user would get following

qmp-shell -p /tmp/q
Welcome to the QMP low-level shell!
Connected to QEMU 2.8.50
(QEMU) query-hotpluggable-cpus
{
   "return": [
       {
           "type": "qemu64-x86_64-cpu", 
           "vcpus-count": 1, 
           "props": {
               "socket-id": 1, 
               "core-id": 0, 
               "thread-id": 0
           }
       }, 
       {
           "qom-path": "/machine/unattached/device[0]", 
           "type": "qemu64-x86_64-cpu", 
           "vcpus-count": 1, 
           "props": {
               "socket-id": 0, 
               "core-id": 0, 
               "thread-id": 0
           }
       }
   ]
}

list of present and possible CPUs for given at startup -cpu and -smp CLI options. Where

  • -cpu cpu_model is translated to corresponding CPU type name that could be used with device_add as driver property value
  • -smp n,sockets=x,cores=y,threads=z,maxcpus=m is translated to a machine dependent set of present/possible to hotplug CPUs, where per CPU props list provides a set of property/value pairs necessary to hotplug given CPU instance with device_add command.
  • qom-path is path to present CPU which is emplty for possible to hot-add CPU entries

Now user having list of necessary properties can hot-add CPU using either monitor or QMP interface, for above example command to hot-add the second cpu would look like:

qmp-shell -p /tmp/q
Welcome to the QMP low-level shell!
Connected to QEMU 2.8.50
(QEMU) device_add id=cpu2 driver=qemu64-x86_64-cpu socket-id=1 core-id=0 thread-id=0
{
    "return": {}
}