Features/CPUHotplug: Difference between revisions

From QEMU
(Use wait=off instead of the obsolete nowait parameter)
 
(44 intermediate revisions by 6 users not shown)
Line 1: Line 1:
= Summary =
= Summary =
This feature is about allowing CPU hotplug in QEMU.
There are 2 ways to hotplug CPU in QEMU:
* dedicated legacy interface: cpu-add QMP command (removed in QEMU v5.2)
* generic device-add/device-del interface for hot-(un)plugging CPUs


= Owner =
= Owner =
Line 7: Line 9:
* '''Email:''' imammedo@redhat.com
* '''Email:''' imammedo@redhat.com


= Detailed Summary =
= cpu-add interface =
It was proposed to use device-add/device-del interface for hot-plugging CPUs
instead of using it's own interface. For this to work CPUs should become
child of Device and do not require external calls for creating or
initializing it.


= X86CPU Hot-Add Dependencies and ToDo-s =
Note: The cpu-add interface has been removed starting with QEMU v5.2. Use the device_add interface instead (see below).
<sup>Legend:</sup> <sup><span style="color:green">green</span> - done, <span style="color:blue">blue</span> - in progress, <span style="color:red">red</span> - TBD</sup>


* <span style="color:green">External CPU clean-ups. Move CPU internals inside CPU object</span>
== Summary ==
** <span style="color:green">move tcg init code CPU. commits d65e98, 84e3b60, eeec69d, 130a038</span>
'''''{ 'command': 'cpu-add', 'data': {'id': 'int'} }'''''<br/>
** <span style="color:green">move CPU reset from board level into CPU. commits 65dee3805, dd673288</span>
* ID - a number in range [0..max-cpus)<br/>
** <span style="color:green">move APIC creation/initialization into CPU object. commit [ bdeec8021]</span>
* Available since: 1.5<br/>
* Supported targets: i386-softmmu, x86_64-softmmu, s390x (since 2.6)


* <span style="color:green">CPU as Device [commit: 961f839]</span>
== Description ==
** necessary for converting "CPUID features" into static properties
Command is a legacy solution for CPU hot-add and gives a simplified interface for it.
** allows to use device_add command after CPU subclasses is implemented.
It provides an opportunity to implement the feature for targets that currently can't implement CPU hot-add
using device_add command due to their present design. Later targets that implement it could rewrite
it to become a wrapper over device_add when it becomes usable for target.


* <span style="color:blue">QOM realize, device-only</span>
== Usage example ==
**http://permalink.gmane.org/gmane.comp.emulators.qemu/187533
1. start QEMU with QMP socket available and with startup amount of CPUs less than ''maxcpus''
** <span style="color:red">convert CPUs realizefn() to use DeviceRealize</span>


* <span style="color:blue">CPUID features as properties</span>
./qemu-system-x86_64 -qmp unix:/tmp/qmp-sock,server=on,wait=off -smp 1,maxcpus=4
** provides an ability to set/get features using common FEAT_FOO=VAL property interfaces.
** Features related clean-ups and code reorganisation:
*** move feature flags fix-ups & checks to realize time
**** in OQM model any feature/property could be amended until realize time. Masking out unsupported kvm/tcg features too early could lead to invalid features to be exposed to guest
**** <span style="color:green">9b15cd9 target-i386: Sanitize AMD's ext2_features at realize time</span>
**** <span style="color:green">4586f15 target-i386: Filter out unsupported features at realize time</span>
*** <span style="color:blue">separate features parsing from setting defaults</span>
**** clean-up cpu_x86_parse_featurestr(), leaving in it only custom features parsing that should be set on CPU instance after all defaults are set. Later defaults initialization should be moved to CPU sub-classes and custom legacy features parser cpu_x86_parse_featurestr() could be converted to setting global properties, when CPU features are converted into static properties.
**** Patches: [http://permalink.gmane.org/gmane.comp.emulators.qemu/188062]
*** <span style="color:blue">simplify vendor property</span>
**** current 'vendor' feature implementation has complex semantic depending on if qemu is running in TCG or KVM mode and if vendor is overridden on command-line. ''if (kvm_enabled() == true)  vendor = host's vendor;  else vendor = built-in vendor; if (custom vendor) vendor = custom vendor''
**** patch [http://patchwork.ozlabs.org/patch/211187/] tries to simplify vendor implementation while keeping current semantic.
**** Patches 5, 6: [http://permalink.gmane.org/gmane.comp.emulators.qemu/188062]
** <span style="color:blue">Conversion of features and other properties into static properties</span> provides following benefits:
*** global properties for CPU, generalizing -cpu xxx,features_string template to a set of global properties 
*** latest implementation based on qom-cpu tree: [https://github.com/imammedo/qemu/tree/x86-cpu-properties.WIP]


* <span style="color:blue">CPU models as CPU subclasses</span>
2. Connect to QMP socket using qmp-shell command
** gives ability to create CPUs using CPU subclass name without any ad-hoc calls.
** there are several implementations in qemu-devel with following open questions:
*** <span style="color:red">if running in KVM mode, kvm_init() should be called before sub-classes class_init is called</span>
**** <span style="color:red">issue 1</span>: in KVM mode qemu provides '''host''' CPU model. This model depends on kvm being initialized before CPU could be created due to dependency on kvm_arch_get_supported_cpuid(). Due to lazy type initialization it usually doesn't cause problem because CPUs are created after kvm_init(). However if ''qemu'' is called with options ''-enable-kvm -cpu help'', it should display host model as available. With introduction of CPU sub-classes, '''host''''s cpu model type should be registered only when kvm is enabled, introducing dependency on kvm_init() being called first. The same issue applies to type introspection when it will be available.
**** <span style="color:red">issue 2</span>: when ''qemu'' is started with ''-enable-kvm'' and '''vendor''' feature is not overridden on command line, built-in '''vendor''' is replaced with host's vendor [see commit 8935499831312]. Again with lazy type initialization it doesn't cause problem because kvm_init() is called before CPU's type class_init() is called, so class_init() could overwrite built-in vendor with host's value. But if type introspection wouldn't require instantiated machine /i.e. be like ''-cpu help''/ it could cause problem that even with ''-enable-kvm'' it would return built-in vendor values instead of host's due to the lack of option dependencies which would say that ''-enable-kvm'' should be processed before '-show-types' or something like this over other interfaces.


* <span style="color:red">Allow hot-plug on SysBus</span>
./QMP/qmp-shell /tmp/qmp-sock
** qdev_create() assigns CPU to SysBus and an APIC created along with a CPU also has SysBus as its bus, so to make hotlug work SysBus should allow hotplug.
** hack to do so [https://github.com/imammedo/qemu/commit/1d55a378ec7b6764c45c774d156649e1575778df]


* <span style="color:red">fix kvm apic code path to work with hotplug</span>
3. Add CPUs issuing ''cpu-add''  command in qmp-shell command prompt
** currently crashes on vapic reset
** works with TCG APIC device, though


* <span style="color:red">Add acpi code to handle CPU hot-add event</span>
cpu-add id=1
** send CPU hot-add event to guest 'not yet done'


* <span style="color:blue">Experimental CPU hot-add tree</span> [https://github.com/imammedo/qemu/tree/x86-cpu-hot-add.WIP] <span style="color:red">(ACPI guest notification is not impl. yet)</span>
4. Optionally online newly added CPU inside guest
 
Linux kernel doesn't online hot-added CPUs automatically. Once CPU is hot-added it should be onlined using an appropriate udev script or manually by issuing a following command:
 
echo 1 > /sys/devices/system/cpu/cpu1/online
 
Sample udev script:
Add the following line to /etc/udev/rules.d/99-hotPlugCPU.rules
 
SUBSYSTEM=="cpu",ACTION=="add",RUN+="/bin/sh -c '[ ! -e /sys$devpath/online ] || echo 1 > /sys$devpath/online'"
 
 
= device_add/device_del interface =
== Summary ==
Lists possible to hotplug CPUs:<br/>
''''' { 'command': 'query-hotpluggable-cpus', 'returns': ['HotpluggableCPU'] }
 
CPU hot-add:<br/>
'''''{ 'command': 'device_add',  'data': {'driver': 'str', 'id': 'str', ... }}
* mandatory properties for every CPU:
** driver: cpu model type name
** id: unique device name
* target/configuration dependent properties
** socket-id: socket number in range [0..max sockets)
** core-id: core number in range [0..max cores)
** thread-id: thread-id in range [..max threads)
** node-id: NUMA node ID the CPU belongs to
 
CPU hot-remove:<br/>
''''' { 'command': 'device_del', 'data': {'id': 'str'} }
* id: device name that has been used for CPU hotplug with device_add command or -device CLI option
 
<br/>
Supported targets:
* i386, x86_64 (since 2.7)
* spapr (since 2.7)
 
== Description ==
When hot-plugging a CPU with device_add command, user must specify which CPU instance it needs to hotplug. To help user specify a particular CPU instance target/machine type that supports generic device_add/device_del interface for hot-adding/removing CPUs must implement '''''query-hotpluggable-cpus''''' QMP command.
User can query the capability using '''''query-machines''''' QMP command and check if property ''hotpluggable-cpus'' is set to true. If machine reports that ''hotpluggable-cpus'' are supported, user can use '''''query-hotpluggable-cpus''''' command to list possible to hotplug CPUs with list of necessary to hotplug properties.<br/>
For example with following CLI:<br/>
qemu-system-x86_64 -smp 1,maxcpus=2 -qmp unix:/tmp/q,server=on,wait=off
user would get following
qmp-shell -p /tmp/q
Welcome to the QMP low-level shell!
Connected to QEMU 2.8.50
(QEMU) query-hotpluggable-cpus
{
    "return": [
        {
            "type": "qemu64-x86_64-cpu",
            "vcpus-count": 1,
            "props": {
                "socket-id": 1,
                "core-id": 0,
                "thread-id": 0
            }
        },
        {
            "qom-path": "/machine/unattached/device[0]",
            "type": "qemu64-x86_64-cpu",
            "vcpus-count": 1,
            "props": {
                "socket-id": 0,
                "core-id": 0,
                "thread-id": 0
            }
        }
    ]
}
 
list of present and possible CPUs for given at startup ''-cpu'' and ''-smp'' CLI options. Where
*''-cpu cpu_model'' is translated to corresponding CPU ''type'' name that could be used with device_add as ''driver'' property value
*''-smp n,sockets=x,cores=y,threads=z,maxcpus=m'' is translated to a machine dependent set of present/possible to hotplug CPUs, where per CPU ''props'' list provides a set of property/value pairs necessary to hotplug given CPU instance with device_add command.
*''qom-path'' is path to present CPU which is emplty for possible to hot-add CPU entries<br>
Now user having list of necessary properties can hot-add a CPU using either monitor or QMP interface, for above example, the command to hot-add the second CPU would look like:
qmp-shell -p /tmp/q
Welcome to the QMP low-level shell!
Connected to QEMU 2.8.50
(QEMU) device_add id=cpu2 driver=qemu64-x86_64-cpu socket-id=1 core-id=0 thread-id=0
{
    "return": {}
}
 
CPU hot-remove could be done using device_del command like with any other device:
qmp-shell -p /tmp/q
Welcome to the QMP low-level shell!
Connected to QEMU 2.8.50
(QEMU) device_del id=cpu2
{
    "return": {}
}
Note: CPU hot-remove is machine dependent and requires guest cooperation. device_del command does not guarantee CPU removal to actually happen, typically it's a request forwarded to guest using target dependent mechanism.
 
*x86 uses ACPI to request removal from guest OS
 
once guest is freed being removed CPU from use guest ejects it using target dependent mechanism and only at that time CPU is actually removed in QEMU and DEVICE_DELETED QMP event is sent to management application.
 
== Migration considerations ==
=== cpu-add ===
* migration target should be started with initial CPU count '-smp XX' that includes hot-added CPUs on migration source side.
* CPU shouldn't be hot-plugged during migration.
* adding CPUs should be done in successive order from lower to higher IDs in [0..max-cpus) range.<br/>It's possible to add arbitrary CPUs in random order, however that would cause migration to fail on its target side.
 
=== device_add/device_del ===
* '-smp xx,...' on target side of migration must be the same as on source side
* hot-added CPUs must be added to the end of CLI on target side of migration using '-device' options with the same properties that were used for hotplugging CPUs.
* CPU can be hot-added in any order however it's recommended to use the same order when declaring hot-added CPUs on target side of migration with ''-device''
* if earlier hot-added CPU where successfully hot-removed on source side, it shouldn't be present on target side of migration
* hot-added CPUs declarations on target side shall set ''hotplugged'' property to ''on'', i.e. if CPU has been hot-added with command
device_add driver=qemu64-x86_64-cpu socket-id=1 core-id=0 thread-id=0 id=cpu2
QEMU implicitly sets ''hotplugged'' property of that CPU to ''on'' however on target side user shall explicitly set it as it's ''off'' for ''-device'' created devices by default. i.e. CPU from above example on target side should be declared as following
-device qemu64-x86_64-cpu,socket-id=1,core-id=0,thread-id=0,id=cpu2,hotplugged=on

Latest revision as of 12:40, 22 March 2022

Summary

There are 2 ways to hotplug CPU in QEMU:

  • dedicated legacy interface: cpu-add QMP command (removed in QEMU v5.2)
  • generic device-add/device-del interface for hot-(un)plugging CPUs

Owner

  • Name: Igor Mammedov
  • Email: imammedo@redhat.com

cpu-add interface

Note: The cpu-add interface has been removed starting with QEMU v5.2. Use the device_add interface instead (see below).

Summary

{ 'command': 'cpu-add', 'data': {'id': 'int'} }

  • ID - a number in range [0..max-cpus)
  • Available since: 1.5
  • Supported targets: i386-softmmu, x86_64-softmmu, s390x (since 2.6)

Description

Command is a legacy solution for CPU hot-add and gives a simplified interface for it. It provides an opportunity to implement the feature for targets that currently can't implement CPU hot-add using device_add command due to their present design. Later targets that implement it could rewrite it to become a wrapper over device_add when it becomes usable for target.

Usage example

1. start QEMU with QMP socket available and with startup amount of CPUs less than maxcpus

./qemu-system-x86_64 -qmp unix:/tmp/qmp-sock,server=on,wait=off -smp 1,maxcpus=4

2. Connect to QMP socket using qmp-shell command

./QMP/qmp-shell /tmp/qmp-sock

3. Add CPUs issuing cpu-add command in qmp-shell command prompt

cpu-add id=1

4. Optionally online newly added CPU inside guest

Linux kernel doesn't online hot-added CPUs automatically. Once CPU is hot-added it should be onlined using an appropriate udev script or manually by issuing a following command:

echo 1 > /sys/devices/system/cpu/cpu1/online

Sample udev script: Add the following line to /etc/udev/rules.d/99-hotPlugCPU.rules

SUBSYSTEM=="cpu",ACTION=="add",RUN+="/bin/sh -c '[ ! -e /sys$devpath/online ] || echo 1 > /sys$devpath/online'"


device_add/device_del interface

Summary

Lists possible to hotplug CPUs:
{ 'command': 'query-hotpluggable-cpus', 'returns': ['HotpluggableCPU'] }

CPU hot-add:
{ 'command': 'device_add', 'data': {'driver': 'str', 'id': 'str', ... }}

  • mandatory properties for every CPU:
    • driver: cpu model type name
    • id: unique device name
  • target/configuration dependent properties
    • socket-id: socket number in range [0..max sockets)
    • core-id: core number in range [0..max cores)
    • thread-id: thread-id in range [..max threads)
    • node-id: NUMA node ID the CPU belongs to

CPU hot-remove:
{ 'command': 'device_del', 'data': {'id': 'str'} }

  • id: device name that has been used for CPU hotplug with device_add command or -device CLI option


Supported targets:

  • i386, x86_64 (since 2.7)
  • spapr (since 2.7)

Description

When hot-plugging a CPU with device_add command, user must specify which CPU instance it needs to hotplug. To help user specify a particular CPU instance target/machine type that supports generic device_add/device_del interface for hot-adding/removing CPUs must implement query-hotpluggable-cpus QMP command. User can query the capability using query-machines QMP command and check if property hotpluggable-cpus is set to true. If machine reports that hotpluggable-cpus are supported, user can use query-hotpluggable-cpus command to list possible to hotplug CPUs with list of necessary to hotplug properties.
For example with following CLI:

qemu-system-x86_64 -smp 1,maxcpus=2 -qmp unix:/tmp/q,server=on,wait=off

user would get following

qmp-shell -p /tmp/q
Welcome to the QMP low-level shell!
Connected to QEMU 2.8.50
(QEMU) query-hotpluggable-cpus
{
   "return": [
       {
           "type": "qemu64-x86_64-cpu", 
           "vcpus-count": 1, 
           "props": {
               "socket-id": 1, 
               "core-id": 0, 
               "thread-id": 0
           }
       }, 
       {
           "qom-path": "/machine/unattached/device[0]", 
           "type": "qemu64-x86_64-cpu", 
           "vcpus-count": 1, 
           "props": {
               "socket-id": 0, 
               "core-id": 0, 
               "thread-id": 0
           }
       }
   ]
}

list of present and possible CPUs for given at startup -cpu and -smp CLI options. Where

  • -cpu cpu_model is translated to corresponding CPU type name that could be used with device_add as driver property value
  • -smp n,sockets=x,cores=y,threads=z,maxcpus=m is translated to a machine dependent set of present/possible to hotplug CPUs, where per CPU props list provides a set of property/value pairs necessary to hotplug given CPU instance with device_add command.
  • qom-path is path to present CPU which is emplty for possible to hot-add CPU entries

Now user having list of necessary properties can hot-add a CPU using either monitor or QMP interface, for above example, the command to hot-add the second CPU would look like:

qmp-shell -p /tmp/q
Welcome to the QMP low-level shell!
Connected to QEMU 2.8.50
(QEMU) device_add id=cpu2 driver=qemu64-x86_64-cpu socket-id=1 core-id=0 thread-id=0
{
    "return": {}
}

CPU hot-remove could be done using device_del command like with any other device:

qmp-shell -p /tmp/q
Welcome to the QMP low-level shell!
Connected to QEMU 2.8.50
(QEMU) device_del id=cpu2
{
    "return": {}
}

Note: CPU hot-remove is machine dependent and requires guest cooperation. device_del command does not guarantee CPU removal to actually happen, typically it's a request forwarded to guest using target dependent mechanism.

  • x86 uses ACPI to request removal from guest OS

once guest is freed being removed CPU from use guest ejects it using target dependent mechanism and only at that time CPU is actually removed in QEMU and DEVICE_DELETED QMP event is sent to management application.

Migration considerations

cpu-add

  • migration target should be started with initial CPU count '-smp XX' that includes hot-added CPUs on migration source side.
  • CPU shouldn't be hot-plugged during migration.
  • adding CPUs should be done in successive order from lower to higher IDs in [0..max-cpus) range.
    It's possible to add arbitrary CPUs in random order, however that would cause migration to fail on its target side.

device_add/device_del

  • '-smp xx,...' on target side of migration must be the same as on source side
  • hot-added CPUs must be added to the end of CLI on target side of migration using '-device' options with the same properties that were used for hotplugging CPUs.
  • CPU can be hot-added in any order however it's recommended to use the same order when declaring hot-added CPUs on target side of migration with -device
  • if earlier hot-added CPU where successfully hot-removed on source side, it shouldn't be present on target side of migration
  • hot-added CPUs declarations on target side shall set hotplugged property to on, i.e. if CPU has been hot-added with command
device_add driver=qemu64-x86_64-cpu socket-id=1 core-id=0 thread-id=0 id=cpu2

QEMU implicitly sets hotplugged property of that CPU to on however on target side user shall explicitly set it as it's off for -device created devices by default. i.e. CPU from above example on target side should be declared as following

-device qemu64-x86_64-cpu,socket-id=1,core-id=0,thread-id=0,id=cpu2,hotplugged=on