Features/CPUHotplug

Summary

This feature is about allowing CPU hotplug in QEMU.

Owner

• Name: Igor Mammedov
• Email: imammedo@redhat.com

Detailed Summary

It was proposed to use device-add/device-del interface for hot-plugging CPUs instead of using it's own interface. For this to work CPUs should become child of Device and do not require external calls for creating or initializing it.

CPU hotplug using cpu-add QMP command

Summary

cpu-add id=[ID]

• ID - a number in range [0..max-cpus)
  
• Available since: 1.5
  
• Supported targets: i386-softmmu, x86_64-softmmu

Description

Command is an intermediate solution for CPU hot-add and gives a simplified interface for it. It provides an opportunity to implement the feature for targets that currently can't implement CPU hot-add using device_add command due to their present design. Later targets that implement it could rewrite it to become a wrapper over device_add when it becomes usable for target.

Usage example

1. start QEMU with QMP socket available and with startup amount of CPUs less than maxcpus

./qemu-system-x86_64 -qmp unix:/tmp/qmp-sock,server,nowait -smp 1,maxcpus=4

2. Connect to QMP socket using qmp-shell command

./QMP/qmp-shell /tmp/qmp-sock

3. Add CPUs issuing cpu-add command in qmp-shell command prompt

cpu-add id=1

4. Optionally online newly added CPU inside guest

Linux kernel doesn't online hot-added CPUs automatically. Once CPU is hot-added it should be onlined using an appropriate udev script or manually by issuing a following command:

echo 1 > /sys/devices/system/cpu/cpu1/online

X86CPU Hot-Add Dependencies and ToDo-s

^{Legend: green - done, blue - in progress, red - TBD}

• External CPU clean-ups. Move CPU internals inside CPU object
  • move tcg init code CPU. commits d65e98, 84e3b60, eeec69d, 130a038
  • move CPU reset from board level into CPU. commits 65dee3805, dd673288
  • move APIC creation/initialization into CPU object. commit [ bdeec8021]

• CPU as Device [commit: 961f839]
  • necessary for converting "CPUID features" into static properties
  • allows to use device_add command after CPU subclasses is implemented.

• QOM realize, device-only
  • convert CPUs realizefn() to use DeviceRealize [commit: 2b6f294]

• CPUID features as properties
  • provides an ability to set/get features using common FEAT_FOO=VAL property interfaces.
  • Features related clean-ups and code reorganisation:
    • move feature flags fix-ups & checks to realize time
      • in OQM model any feature/property could be amended until realize time. Masking out unsupported kvm/tcg features too early could lead to invalid features to be exposed to guest
      • 9b15cd9 target-i386: Sanitize AMD's ext2_features at realize time
      • 4586f15 target-i386: Filter out unsupported features at realize time
      • 5ec01c2 Move kvm_check_features_against_host() check to realize time
    • separate features parsing from setting defaults
      • clean-up cpu_x86_parse_featurestr(), leaving in it only custom features parsing that should be set on CPU instance after all defaults are set. Later defaults initialization should be moved to CPU sub-classes and custom legacy features parser cpu_x86_parse_featurestr() could be converted to setting global properties, when CPU features are converted into static properties. [commits: 077c68c, fa2db3c, 8ba8a69, 99b88a1, 11acfdd, a91987c, 2c728df]
    • simplify vendor property
      • current 'vendor' feature implementation has complex semantic depending on if qemu is running in TCG or KVM mode and if vendor is overridden on command-line. if (kvm_enabled() == true) vendor = host's vendor; else vendor = built-in vendor; if (custom vendor) vendor = custom vendor[commit: 99b88a1, 11acfdd]
  • Conversion of features and other properties into static properties provides following benefits:
    • global properties for CPU, generalizing -cpu xxx,features_string template to a set of global properties
    • latest implementation doing only conversion to static properties tree: [1] posted v7 series
    • conversion to global proprties is postponed until CPU sub-classes
• CPU models as CPU subclasses
  • gives ability to create CPUs using CPU subclass name without any ad-hoc calls.
  • there are several implementations in qemu-devel with following open questions:
    • if running in KVM mode, kvm_init() should be called before sub-classes class_init is called
      • issue 1: in KVM mode qemu provides host CPU model. This model depends on kvm being initialized before CPU could be created due to dependency on kvm_arch_get_supported_cpuid(). Due to lazy type initialization it usually doesn't cause problem because CPUs are created after kvm_init(). However if qemu is called with options -enable-kvm -cpu help, it should display host model as available. With introduction of CPU sub-classes, host's cpu model type should be registered only when kvm is enabled, introducing dependency on kvm_init() being called first. The same issue applies to type introspection when it will be available. I think that we are reached consensus here that 'host' CPU type should be registered at kvm_arch_init() time.
      • issue 2: when qemu is started with -enable-kvm and vendor feature is not overridden on command line, built-in vendor is replaced with host's vendor [see commit 8935499831312]. Again with lazy type initialization it doesn't cause problem because kvm_init() is called before CPU's type class_init() is called, so class_init() could overwrite built-in vendor with host's value. But if type introspection wouldn't require instantiated machine /i.e. be like -cpu help/ it could cause problem that even with -enable-kvm it would return built-in vendor values instead of host's due to the lack of option dependencies which would say that -enable-kvm should be processed before '-show-types' or something like this over other interfaces.. 'vendor' issue evolved into a generic problem, there are several ideas how to proceed:
        • 1. Register all CPU sub-classes at QEMU start-up time and fix them up later if/when kvm_arch_init() called to apply KVM specific to them
          • keeps amount of CPU sub-classes == cpu_models
          • it requires enumeration of all CPU sub-classes to fixup default values. Therefore causing type class initialization for types which won't be used.
          • defaults could be deceiving/not valid if class introspection to happen before kvm_arch_init(), and there is no sure way to prevent misuse.
          • CPU sub-classes defaults are mutable depending on -enable-kvm option.
          • possible to get rid of x86_def_t type/array embedding it in class_[model]_init() functions
        • 2. Register *-tcg-* and *-kvm-* subclasses at QEMU start-up time, with TCG/KVM defaults hard-codded in respective class_[tcg|kvm]_init() functions.
          • CPU-subclasses defaults are not mutable depending on -enable-kvm option and doesn't require (now) kvm_init() being called first
          • doubles CPU sub-classes amount
          • exposes *-kvm-* CPU sub-classes to user and allows to create *-kvm-* based CPU in TCG mode and vice/verse.
          • hard to re-factor since users could start use class names instead of cpu_model names and won't allow to eliminate x86_def_t type/array.
        • 3. Register CPU sub-classes after kvm_init() but before machine init and set defaults in class_init() depending on if KVM is available/inited.
          • keeps amount of CPU sub-classes == cpu_models
          • user sees only one immutable set of classes. But set has different defaults depending on mode QEMU was started with (TCG/KVM).
          • possible to get rid of x86_def_t type/array embedding it in class_[model]_init() functions
          • requires global hook in vl.c between kvm_init() and machine_init()
          • CPU classes won't be available before this hook, so QMP, qom-get/set, list_cpus will be forced to be called after it to get access to CPU sub-classes

• Allow hot-plug on SysBus
  • qdev_create() assigns CPU to SysBus and an APIC created along with a CPU also has SysBus as its bus, so to make hotlug work SysBus should allow hotplug.
  • hack to do so [2]

• fix kvm apic code path to work with hotplug
  • currently crashes on vapic reset
  • works with TCG APIC device, though

• Add acpi code to handle CPU hot-add event
  • send CPU hot-add event to guest, in progress

• Experimental CPU hot-add tree [3] (ACPI guest notification is not impl. yet)