Features/CPUHotplug: Difference between revisions

From QEMU
(Use wait=off instead of the obsolete nowait parameter)
 
(22 intermediate revisions by 2 users not shown)
Line 1: Line 1:
= Summary =
= Summary =
There are 2 ways to hotplug CPU in QEMU:
There are 2 ways to hotplug CPU in QEMU:
* dedicated legacy interface: cpu-add QMP command
* dedicated legacy interface: cpu-add QMP command (removed in QEMU v5.2)
* generic device-add/device-del interface for hot-(un)plugging CPUs
* generic device-add/device-del interface for hot-(un)plugging CPUs


Line 10: Line 10:


= cpu-add interface =
= cpu-add interface =
Note: The cpu-add interface has been removed starting with QEMU v5.2. Use the device_add interface instead (see below).
== Summary ==
== Summary ==
'''''{ 'command': 'cpu-add', 'data': {'id': 'int'} }'''''<br/>
'''''{ 'command': 'cpu-add', 'data': {'id': 'int'} }'''''<br/>
* ID - a number in range [0..max-cpus)<br/>
* ID - a number in range [0..max-cpus)<br/>
* Available since: 1.5<br/>
* Available since: 1.5<br/>
* Supported targets: i386-softmmu, x86_64-softmmu
* Supported targets: i386-softmmu, x86_64-softmmu, s390x (since 2.6)


== Description ==
== Description ==
Command is an intermediate solution for CPU hot-add and gives a simplified interface for it.
Command is a legacy solution for CPU hot-add and gives a simplified interface for it.
It provides an opportunity to implement the feature for targets that currently can't implement CPU hot-add
It provides an opportunity to implement the feature for targets that currently can't implement CPU hot-add
using device_add command due to their present design. Later targets that implement it could rewrite
using device_add command due to their present design. Later targets that implement it could rewrite
Line 25: Line 28:
1. start QEMU with QMP socket available and with startup amount of CPUs less than ''maxcpus''
1. start QEMU with QMP socket available and with startup amount of CPUs less than ''maxcpus''


  ./qemu-system-x86_64 -qmp unix:/tmp/qmp-sock,server,nowait -smp 1,maxcpus=4
  ./qemu-system-x86_64 -qmp unix:/tmp/qmp-sock,server=on,wait=off -smp 1,maxcpus=4


2. Connect to QMP socket using qmp-shell command
2. Connect to QMP socket using qmp-shell command
Line 47: Line 50:




== Current limitations ==
= device_add/device_del interface =
== Summary ==
Lists possible to hotplug CPUs:<br/>
''''' { 'command': 'query-hotpluggable-cpus', 'returns': ['HotpluggableCPU'] }
 
CPU hot-add:<br/>
'''''{ 'command': 'device_add',  'data': {'driver': 'str', 'id': 'str', ... }}
* mandatory properties for every CPU:
** driver: cpu model type name
** id: unique device name
* target/configuration dependent properties
** socket-id: socket number in range [0..max sockets)
** core-id: core number in range [0..max cores)
** thread-id: thread-id in range [..max threads)
** node-id: NUMA node ID the CPU belongs to
 
CPU hot-remove:<br/>
''''' { 'command': 'device_del', 'data': {'id': 'str'} }
* id: device name that has been used for CPU hotplug with device_add command or -device CLI option


# migration target should be started with initial CPU count '-smp XX' that includes hot-added CPUs on migration source side.
<br/>
# CPU shouldn't be hot-plugged during migration.
Supported targets:
# adding CPUs should be done in successive order from lower to higher IDs in [0..max-cpus) range.<br/>It's possible to add arbitrary CPUs in random order, however that would cause migration to fail on its target side.
* i386, x86_64 (since 2.7)
* spapr (since 2.7)


= device_add/device_del interface =
== Description ==
<sup>Legend:</sup> <sup><span style="color:green">green</span> - done, <span style="color:blue">blue</span> - in progress, <span style="color:red">red</span> - TBD</sup>
When hot-plugging a CPU with device_add command, user must specify which CPU instance it needs to hotplug. To help user specify a particular CPU instance target/machine type that supports generic device_add/device_del interface for hot-adding/removing CPUs must implement '''''query-hotpluggable-cpus''''' QMP command.
User can query the capability using '''''query-machines''''' QMP command and check if property ''hotpluggable-cpus'' is set to true. If machine reports that ''hotpluggable-cpus'' are supported, user can use '''''query-hotpluggable-cpus''''' command to list possible to hotplug CPUs with list of necessary to hotplug properties.<br/>
For example with following CLI:<br/>
qemu-system-x86_64 -smp 1,maxcpus=2 -qmp unix:/tmp/q,server=on,wait=off
user would get following
qmp-shell -p /tmp/q
Welcome to the QMP low-level shell!
Connected to QEMU 2.8.50
(QEMU) query-hotpluggable-cpus
{
    "return": [
        {
            "type": "qemu64-x86_64-cpu",
            "vcpus-count": 1,
            "props": {
                "socket-id": 1,
                "core-id": 0,
                "thread-id": 0
            }
        },
        {
            "qom-path": "/machine/unattached/device[0]",
            "type": "qemu64-x86_64-cpu",
            "vcpus-count": 1,  
            "props": {
                "socket-id": 0,  
                "core-id": 0,
                "thread-id": 0
            }
        }
    ]
}


== Work in progress/TODOs ==
list of present and possible CPUs for given at startup ''-cpu'' and ''-smp'' CLI options. Where
* <span style="color:blue">Conversion of features and other properties into static properties</span> provides following benefits:
*''-cpu cpu_model'' is translated to corresponding CPU ''type'' name that could be used with device_add as ''driver'' property value
** global properties for CPU, generalizing -cpu xxx,features_string template to a set of global properties 
*''-smp n,sockets=x,cores=y,threads=z,maxcpus=m'' is translated to a machine dependent set of present/possible to hotplug CPUs, where per CPU ''props'' list provides a set of property/value pairs necessary to hotplug given CPU instance with device_add command.
** latest implementation doing only conversion to static properties tree: [https://github.com/imammedo/qemu/tree/x86-cpu-properties.WIP] [http://permalink.gmane.org/gmane.comp.emulators.qemu/196850| posted v7 series]
*''qom-path'' is path to present CPU which is emplty for possible to hot-add CPU entries<br>
** conversion to global properties is postponed until CPU sub-classes 
Now user having list of necessary properties can hot-add a CPU using either monitor or QMP interface, for above example, the command to hot-add the second CPU would look like:
* <span style="color:blue">CPU models as CPU subclasses</span>
qmp-shell -p /tmp/q
** gives ability to create CPUs using CPU subclass name without any ad-hoc calls.
Welcome to the QMP low-level shell!
** there are several implementations in qemu-devel with following open questions:
Connected to QEMU 2.8.50
*** <span style="color:red">if running in KVM mode, kvm_init() should be called before sub-classes class_init is called</span>
(QEMU) device_add id=cpu2 driver=qemu64-x86_64-cpu socket-id=1 core-id=0 thread-id=0
**** <span style="color:red">issue 1</span>: in KVM mode qemu provides '''host''' CPU model. <strike>This model depends on kvm being initialized before CPU could be created due to dependency on kvm_arch_get_supported_cpuid(). Due to lazy type initialization it usually doesn't cause problem because CPUs are created after kvm_init(). However if ''qemu'' is called with options ''-enable-kvm -cpu help'', it should display host model as available. With introduction of CPU sub-classes, '''host''''s cpu model type should be registered only when kvm is enabled, introducing dependency on kvm_init() being called first. The same issue applies to type introspection when it will be available.</strike> I think that we are reached consensus here that 'host' CPU type should be registered at kvm_arch_init() time.
{
**** <span style="color:red">issue 2</span>: <strike>when ''qemu'' is started with ''-enable-kvm'' and '''vendor''' feature is not overridden on command line, built-in '''vendor''' is replaced with host's vendor [see commit 8935499831312]. Again with lazy type initialization it doesn't cause problem because kvm_init() is called before CPU's type class_init() is called, so class_init() could overwrite built-in vendor with host's value. But if type introspection wouldn't require instantiated machine /i.e. be like ''-cpu help''/ it could cause problem that even with ''-enable-kvm'' it would return built-in vendor values instead of host's due to the lack of option dependencies which would say that ''-enable-kvm'' should be processed before '-show-types' or something like this over other interfaces.</strike>. 'vendor' issue evolved into a generic problem, there are several ideas how to proceed:
    "return": {}
***** 1. Register all CPU sub-classes at QEMU start-up time and fix them up later if/when kvm_arch_init() called to apply KVM specific to them
}
****** keeps amount of CPU sub-classes == cpu_models
****** it requires enumeration of all CPU sub-classes to fixup default values. Therefore causing type class initialization for types which won't be used.
****** defaults could be deceiving/not valid if class introspection to happen before kvm_arch_init(), and there is no sure way to prevent misuse.
****** CPU sub-classes defaults are mutable depending on -enable-kvm option.
****** possible to get rid of x86_def_t type/array embedding it in class_[model]_init() functions
***** 2. Register *-tcg-* and *-kvm-* subclasses at QEMU start-up time, with TCG/KVM defaults hard-codded in respective class_[tcg|kvm]_init() functions.
****** CPU-subclasses defaults are not mutable depending on -enable-kvm option and doesn't require (now) kvm_init() being called first
****** doubles CPU sub-classes amount
****** exposes *-kvm-* CPU sub-classes to user and allows to create *-kvm-* based CPU in TCG mode and vice/verse.
****** hard to re-factor since users could start use class names instead of cpu_model names and won't allow to eliminate x86_def_t type/array.
***** 3. Register CPU sub-classes after kvm_init() but before machine init and set defaults in class_init() depending on if KVM is available/inited.
****** keeps amount of CPU sub-classes == cpu_models
****** user sees only one immutable set of classes. But set has different defaults depending on mode QEMU was started with (TCG/KVM).
****** possible to get rid of x86_def_t type/array embedding it in class_[model]_init() functions
****** requires global hook in vl.c between kvm_init() and machine_init()
****** CPU classes won't be available before this hook, so QMP, qom-get/set, list_cpus will be forced to be called after it to get access to CPU sub-classes


== Completed dependencies ==
CPU hot-remove could be done using device_del command like with any other device:
<div style="color:green" class="mw-collapsible mw-collapsed">
qmp-shell -p /tmp/q
* External CPU clean-ups. Move CPU internals inside CPU object
Welcome to the QMP low-level shell!
** move tcg init code CPU. commits d65e98, 84e3b60, eeec69d, 130a038
Connected to QEMU 2.8.50
** move CPU reset from board level into CPU. commits 65dee3805, dd673288
(QEMU) device_del id=cpu2
** move APIC creation/initialization into CPU object. commit bdeec8021
{
    "return": {}
}
Note: CPU hot-remove is machine dependent and requires guest cooperation. device_del command does not guarantee CPU removal to actually happen, typically it's a request forwarded to guest using target dependent mechanism.


* CPU as Device [commit: 961f839]
*x86 uses ACPI to request removal from guest OS
** necessary for converting "CPUID features" into static properties
** allows to use device_add command after CPU subclasses is implemented.


* QOM realize, device-only
once guest is freed being removed CPU from use guest ejects it using target dependent mechanism and only at that time CPU is actually removed in QEMU and DEVICE_DELETED QMP event is sent to management application.
** convert CPUs realizefn() to use DeviceRealize [commit: 2b6f294]


* CPUID features as properties
== Migration considerations ==
** provides an ability to set/get features using common FEAT_FOO=VAL property interfaces.
=== cpu-add ===
** Features related clean-ups and code reorganisation:
* migration target should be started with initial CPU count '-smp XX' that includes hot-added CPUs on migration source side.
*** move feature flags fix-ups & checks to realize time
* CPU shouldn't be hot-plugged during migration.
**** in OQM model any feature/property could be amended until realize time. Masking out unsupported kvm/tcg features too early could lead to invalid features to be exposed to guest
* adding CPUs should be done in successive order from lower to higher IDs in [0..max-cpus) range.<br/>It's possible to add arbitrary CPUs in random order, however that would cause migration to fail on its target side.
**** 9b15cd9 target-i386: Sanitize AMD's ext2_features at realize time
**** 4586f15 target-i386: Filter out unsupported features at realize time
**** 5ec01c2 Move kvm_check_features_against_host() check to realize time
*** separate features parsing from setting defaults
**** clean-up cpu_x86_parse_featurestr(), leaving in it only custom features parsing that should be set on CPU instance after all defaults are set. Later defaults initialization should be moved to CPU sub-classes and custom legacy features parser cpu_x86_parse_featurestr() could be converted to setting global properties, when CPU features are converted into static properties. [commits: 077c68c, fa2db3c, 8ba8a69, 99b88a1, 11acfdd, a91987c, 2c728df]
*** simplify vendor property
**** current 'vendor' feature implementation has complex semantic depending on if qemu is running in TCG or KVM mode and if vendor is overridden on command-line. ''if (kvm_enabled() == true)  vendor = host's vendor;  else vendor = built-in vendor; if (custom vendor) vendor = custom vendor''[commit: 99b88a1, 11acfdd]
</div>


[[Category:Obsolete feature pages]]
=== device_add/device_del ===
* '-smp xx,...' on target side of migration must be the same as on source side
* hot-added CPUs must be added to the end of CLI on target side of migration using '-device' options with the same properties that were used for hotplugging CPUs.
* CPU can be hot-added in any order however it's recommended to use the same order when declaring hot-added CPUs on target side of migration with ''-device''
* if earlier hot-added CPU where successfully hot-removed on source side, it shouldn't be present on target side of migration
* hot-added CPUs declarations on target side shall set ''hotplugged'' property to ''on'', i.e. if CPU has been hot-added with command
device_add driver=qemu64-x86_64-cpu socket-id=1 core-id=0 thread-id=0 id=cpu2
QEMU implicitly sets ''hotplugged'' property of that CPU to ''on'' however on target side user shall explicitly set it as it's ''off'' for ''-device'' created devices by default. i.e. CPU from above example on target side should be declared as following
-device qemu64-x86_64-cpu,socket-id=1,core-id=0,thread-id=0,id=cpu2,hotplugged=on

Latest revision as of 12:40, 22 March 2022

Summary

There are 2 ways to hotplug CPU in QEMU:

  • dedicated legacy interface: cpu-add QMP command (removed in QEMU v5.2)
  • generic device-add/device-del interface for hot-(un)plugging CPUs

Owner

  • Name: Igor Mammedov
  • Email: imammedo@redhat.com

cpu-add interface

Note: The cpu-add interface has been removed starting with QEMU v5.2. Use the device_add interface instead (see below).

Summary

{ 'command': 'cpu-add', 'data': {'id': 'int'} }

  • ID - a number in range [0..max-cpus)
  • Available since: 1.5
  • Supported targets: i386-softmmu, x86_64-softmmu, s390x (since 2.6)

Description

Command is a legacy solution for CPU hot-add and gives a simplified interface for it. It provides an opportunity to implement the feature for targets that currently can't implement CPU hot-add using device_add command due to their present design. Later targets that implement it could rewrite it to become a wrapper over device_add when it becomes usable for target.

Usage example

1. start QEMU with QMP socket available and with startup amount of CPUs less than maxcpus

./qemu-system-x86_64 -qmp unix:/tmp/qmp-sock,server=on,wait=off -smp 1,maxcpus=4

2. Connect to QMP socket using qmp-shell command

./QMP/qmp-shell /tmp/qmp-sock

3. Add CPUs issuing cpu-add command in qmp-shell command prompt

cpu-add id=1

4. Optionally online newly added CPU inside guest

Linux kernel doesn't online hot-added CPUs automatically. Once CPU is hot-added it should be onlined using an appropriate udev script or manually by issuing a following command:

echo 1 > /sys/devices/system/cpu/cpu1/online

Sample udev script: Add the following line to /etc/udev/rules.d/99-hotPlugCPU.rules

SUBSYSTEM=="cpu",ACTION=="add",RUN+="/bin/sh -c '[ ! -e /sys$devpath/online ] || echo 1 > /sys$devpath/online'"


device_add/device_del interface

Summary

Lists possible to hotplug CPUs:
{ 'command': 'query-hotpluggable-cpus', 'returns': ['HotpluggableCPU'] }

CPU hot-add:
{ 'command': 'device_add', 'data': {'driver': 'str', 'id': 'str', ... }}

  • mandatory properties for every CPU:
    • driver: cpu model type name
    • id: unique device name
  • target/configuration dependent properties
    • socket-id: socket number in range [0..max sockets)
    • core-id: core number in range [0..max cores)
    • thread-id: thread-id in range [..max threads)
    • node-id: NUMA node ID the CPU belongs to

CPU hot-remove:
{ 'command': 'device_del', 'data': {'id': 'str'} }

  • id: device name that has been used for CPU hotplug with device_add command or -device CLI option


Supported targets:

  • i386, x86_64 (since 2.7)
  • spapr (since 2.7)

Description

When hot-plugging a CPU with device_add command, user must specify which CPU instance it needs to hotplug. To help user specify a particular CPU instance target/machine type that supports generic device_add/device_del interface for hot-adding/removing CPUs must implement query-hotpluggable-cpus QMP command. User can query the capability using query-machines QMP command and check if property hotpluggable-cpus is set to true. If machine reports that hotpluggable-cpus are supported, user can use query-hotpluggable-cpus command to list possible to hotplug CPUs with list of necessary to hotplug properties.
For example with following CLI:

qemu-system-x86_64 -smp 1,maxcpus=2 -qmp unix:/tmp/q,server=on,wait=off

user would get following

qmp-shell -p /tmp/q
Welcome to the QMP low-level shell!
Connected to QEMU 2.8.50
(QEMU) query-hotpluggable-cpus
{
   "return": [
       {
           "type": "qemu64-x86_64-cpu", 
           "vcpus-count": 1, 
           "props": {
               "socket-id": 1, 
               "core-id": 0, 
               "thread-id": 0
           }
       }, 
       {
           "qom-path": "/machine/unattached/device[0]", 
           "type": "qemu64-x86_64-cpu", 
           "vcpus-count": 1, 
           "props": {
               "socket-id": 0, 
               "core-id": 0, 
               "thread-id": 0
           }
       }
   ]
}

list of present and possible CPUs for given at startup -cpu and -smp CLI options. Where

  • -cpu cpu_model is translated to corresponding CPU type name that could be used with device_add as driver property value
  • -smp n,sockets=x,cores=y,threads=z,maxcpus=m is translated to a machine dependent set of present/possible to hotplug CPUs, where per CPU props list provides a set of property/value pairs necessary to hotplug given CPU instance with device_add command.
  • qom-path is path to present CPU which is emplty for possible to hot-add CPU entries

Now user having list of necessary properties can hot-add a CPU using either monitor or QMP interface, for above example, the command to hot-add the second CPU would look like:

qmp-shell -p /tmp/q
Welcome to the QMP low-level shell!
Connected to QEMU 2.8.50
(QEMU) device_add id=cpu2 driver=qemu64-x86_64-cpu socket-id=1 core-id=0 thread-id=0
{
    "return": {}
}

CPU hot-remove could be done using device_del command like with any other device:

qmp-shell -p /tmp/q
Welcome to the QMP low-level shell!
Connected to QEMU 2.8.50
(QEMU) device_del id=cpu2
{
    "return": {}
}

Note: CPU hot-remove is machine dependent and requires guest cooperation. device_del command does not guarantee CPU removal to actually happen, typically it's a request forwarded to guest using target dependent mechanism.

  • x86 uses ACPI to request removal from guest OS

once guest is freed being removed CPU from use guest ejects it using target dependent mechanism and only at that time CPU is actually removed in QEMU and DEVICE_DELETED QMP event is sent to management application.

Migration considerations

cpu-add

  • migration target should be started with initial CPU count '-smp XX' that includes hot-added CPUs on migration source side.
  • CPU shouldn't be hot-plugged during migration.
  • adding CPUs should be done in successive order from lower to higher IDs in [0..max-cpus) range.
    It's possible to add arbitrary CPUs in random order, however that would cause migration to fail on its target side.

device_add/device_del

  • '-smp xx,...' on target side of migration must be the same as on source side
  • hot-added CPUs must be added to the end of CLI on target side of migration using '-device' options with the same properties that were used for hotplugging CPUs.
  • CPU can be hot-added in any order however it's recommended to use the same order when declaring hot-added CPUs on target side of migration with -device
  • if earlier hot-added CPU where successfully hot-removed on source side, it shouldn't be present on target side of migration
  • hot-added CPUs declarations on target side shall set hotplugged property to on, i.e. if CPU has been hot-added with command
device_add driver=qemu64-x86_64-cpu socket-id=1 core-id=0 thread-id=0 id=cpu2

QEMU implicitly sets hotplugged property of that CPU to on however on target side user shall explicitly set it as it's off for -device created devices by default. i.e. CPU from above example on target side should be declared as following

-device qemu64-x86_64-cpu,socket-id=1,core-id=0,thread-id=0,id=cpu2,hotplugged=on