Features/QCFG: Difference between revisions

From QEMU
No edit summary
No edit summary
 
(2 intermediate revisions by one other user not shown)
Line 1: Line 1:
== Summary ==
== Summary ==


Replace QemuOpts infrastructure with code generation leveraging QAPI.
Replace QEMU command option handling and QemuOpts with a new infrastructure
that supports full introspection and QMP interoperability.


== Owner ==
== Owner ==
Line 8: Line 9:
* '''Email:''' anthony@codemonkey.ws
* '''Email:''' anthony@codemonkey.ws


== Detailed Summary ==
== Background ==


Use code generator to generate demarshalling routines for command line arguments
In order to add a new command line interface to QEMU, we currently do the
into structures.  Pass structures to function calls to process QEMU command
following:
line.


For instance, consider the ''-vnc'' optionWe would first add a type
# Add a new entry to qemu-options.hx with embedded LaTeX documentation. This only describes the option name and whether it takes an option, it does not cover the format of the option.  Documentation about the format of the option is in free form English and is not necessarily exhaustive.
corresponding to the options ''-vnc'' takes:
# The option should make use of QemuOpts--although many don'tTo use QemuOpts, one must define a schema in qemu-config.c (usually).
# A new case statement is added to vl.c which will then parse the option, later in the main routine, a function is iterated over each QemuOpts that has been parsed.
# That function pointer usually is a function defined somewhere else in the code base.  That function now has to interact with QemuOpts in a way that's agreeable with the schema in qemu-config.c
# In many cases, the schema can not be applied because of limitations in the schema format.  In this case, the function pointer is totally responsible for syntax validation.


Besides the considerable complexity of this, there are a lot of places in the
code that must be in agreement.  This makes it very easy to end up with bugs where options aren't properly validated.
Since QemuOpts finds it way deep into the code, for QMP interfaces that need to
interact in the same way, we need to generate a QemuOpts structure from a QMP
data structure.  This currently works by converting the QMP data structures to
strings and then back to QemuOpts.  This results in some very unusual behavior.
== Unifying configuration ==
QCFG uses a schema in the same fashion as QAPI to allow for the expression of
complex data structures and provide a structured way to specify detailed
documentation.
Here is an example of this mechanism applied to the ''-vnc'' option:
##
# @VncConfig:
#
# Configuration options for the built-in VNC server.
#
# @address:  The hostname to bind the VNC server to.  If specified in the form
#            `:display', the server binds to any interface on port
#            (display + 5900).  If specified in the form, `hostname:display',
#            the server is bound to `hostname' interface on port
#            (display + 5900).  If specified in the form, `unix:path', then the
#            server is bound to the unix domain socket `path'.
#
# @password: #optional If true, VNC authentication is enabled.  Use the
#            @set-password command in QMP to set the password.
#
# @reverse:  #optional If true, @address is treated as a client address and the
#            server will connect to @address and then proceed with the server
#            session.  This can be helpful when working through a proxy.
#
# @no-lock-key-sync: #optional If true, disable the built-in heuristics that
#                    attempt to key the Caps and Num Lock keys synchronized
#                    with the host.
#
# @sasl:    #optional If true, enable SASL authentication.
#
# @tls:      #optional If true, use TLS to encrypt the session.
#
# @x509:    #optional The location of an x509 certificate to use to encrypt
#            the session.  This cannot be used in conjunction with @tls.
#
# @x509verify: #optional The location of an x509 certificate to use to encrypt
#              the session.  This is similar to @x509 but it also verifies the
#              clients x509 certificates when they connect.
#
# @acl:      #optional If true, enable VNC ACL support.  ACLs can be set through
#            QMP.
#
# @lossy:    #optional If true, compression will be enabled that can degrade
#            image quality but result in significantly less bandwidth.
#
# Since: 0.14.0
##
  { 'type': 'VncConfig',
  { 'type': 'VncConfig',
   'data': { 'addr': 'str', '*x509': 'str', '*x509verify': 'str',
   'data': { 'address': 'str', '*password': 'bool', '*reverse': 'bool',
             '*password': 'bool', '*tls': 'bool', '*sasl': 'bool' } }
             '*no-lock-key-sync': 'bool', '*sasl': 'bool', '*tls': 'bool',
            '*x509': 'str', '*x509verify': 'str', '*acl': 'bool',
            '*lossy': 'bool' } }
##
# @vnc:
#
# Enable the built-in VNC server for the guest's VGA output.
#
# Since: 0.14.0
##
{ 'option': 'vnc', 'data': 'VncConfig' }
 
To make use of this in QEMU, we simply need to add the following to vl.c:
 
case QEMU_OPTION_vnc:
    qcfg_handle_option("vnc", optarg);
    break;


We can generate an unmarshalling function that will convert a string of comma
The effect of this will be a function call to ''qcfg_handle_vnc()'' passing in
encoded options to a proper structure.  It will generate the following code:
a native format structure.  Below is an example of the function call and the
types generated to support it:


  // qemu/qcfg-marshal.h
  typedef struct VncConfig {
static VncConfig *qcfg_unmarshal_VncConfig(KeyValues *kv, Error **errp)
    char * address;
{
    bool has_password;
     VncConfig *config;
    bool password;
     Error *local_err = NULL;
    bool has_reverse;
     bool has_addr = false;
    bool reverse;
    bool has_no_lock_key_sync;
    bool no_lock_key_sync;
    bool has_sasl;
    bool sasl;
    bool has_x509;
    char * x509;
    bool has_x509verify;
    char * x509verify;
    bool has_acl;
     bool acl;
     bool has_lossy;
     bool lossy;
   
   
     config = qmp_alloc_VncConfig();
     struct VncConfig * next;
} VncConfig;
   
   
    for (; kv; kv = kv->next) {
void qcfg_handle_vnc(VncConfig *config, Error **errp);
        if (strequals(kv->key, "addr")) {
 
            has_addr = true;
The option handler may either process the option immediately (before other
            config->addr = qcfg_unmarshal_str(kv->value, &local_err);
options are parsed) or save the option in a global list for later processing.
        } else if (strequals(kv->key, "x509")) {
 
            config->has_x509 = true;
No extra code is required to expose this over QMP.  We can either expose a new
            config->x509 = qcfg_unmarshal_str(kv->value, &local_err);
QMP function that takes a ''VncConfig'' structure that is implemented by
        } else if (strequals(kv->key, "x509verify")) {
calling this function or we can expose every option handler through
            config->has_x509verify = true;
automatically generated QMP commands.
            config->x509verify = qcfg_unmarshal_str(kv->value, &local_err);
 
        } else if (strequals(kv->key, "password")) {
== Aliases ==
            config->has_password = true;
 
            config->password = qcfg_unmarshal_bool(kv->value, &local_err);
For command line options that have multiple aliases, they can be reduced to
        } else if (strequals(kv->key, "tls")) {
the following:
            config->has_tls = true;
 
            config->tls = qcfg_unmarshal_bool(kv->value, &local_err);
  { 'option': 'm', 'data': 'size_mb', 'aliases': [ 'memory' ] }
        } else if (strequals(kv->key, "sasl")) {
            config->has_sasl = true;
            config->sasl = qcfg_unmarshal_bool(kv->value, &local_err);
        } else {
            error_set(&local_err, QERR_INVALID_PARAMETER, kv->key);
        }
        if (local_err) {
            goto out;
        }
    }
   
    if (!has_addr) {
        error_set(&local_err, QERR_MISSING_PARAMETER, "addr");
        goto out;
    }
    return config;
out:
    qmp_free_vnc_config(config);
    error_propagate(errp, local_err);
    return NULL;
}


There will still be the normal code generated in qmp-types and qmp-marshal-types
This allows the option to be specified as either ''-m'' or ''-memory''.
which means that this type can also be encoded and decode via QMP.


In order to bridge this type to a function that is called to process a command
Aliasing is particularly useful for configuration file support.
line argument, we just need to add the following stanza:


// qemu/qcfg-schema.json
== Configuration File Support ==
{ 'config': 'vnc', 'type': 'VncConfig' }


Which generates:
While the ''qcfg_handle_option()'' interface takes a string, internally, all of
the functions are implemented with ''KeyValues'' lists.  The
''qcfg_handle_option()'' function will parse the string into a list of
''KeyValues''.


// qemu/qcfg.h
This allows all of the infrastructure to be used with a configuration fileThe
  void qcfg_vnc_config(VncConfig * config, Error **errp);
format of the file is as follows:
static void qcfg_handle_vnc(KeyValues *kv, Error **errp)
{
    VncConfig *config;
    Error *local_err = NULL;
    config = qcfg_unmarshal_VncConfig(kv, &local_err);
    if (local_err) {
        error_propagate(errp, local_err);
        return;
    }
    qcfg_vnc(config, errp);
    qmp_free_VncConfig(config);
}
void qcfg_init(void)
{
    qcfg_register_handler("vnc", &qcfg_marshal_vnc);
}


The effect is that we can call a function that takes a string argument and
[vnc]
will decode that to a strongly typed call to a function.
address=localhost:5
sasl=on
tls=on


This function can also be reused by QMP trivially without going through multiple
This translates roughly to:
levels of marshaling and unmarshaling.


This provides a consistent way to document all options including the parameters
  qcfg_handle_option("vnc", "address=localhost:5,sasl=on,tls=on");
that they take. It also provides a consistent way to do introspection of
command line arguments.


This mechanism can also be applied to qdev which will allow for strongly typed
== Complex Data Types ==
qdev initialization functions that can also be trivially called from within
QEMU.


For instance, for the serial device, we would do:
There are certain options that require nested structures and even recursive
structures.  We will express these structures using a dotted syntax.  For
instance, consider ''-blockdev'':


  { 'type': 'ISASerialConfig',
  { 'type': 'BlockdevConfig',
   'data': { '*index': 'int', '*iobase': 'int', '*irq': 'int',
   'data': { 'file': 'str', '*backing-file': 'BlockdevConfig' } }
            'chardev': 'CharDriverState' } }


This would generate a standard qmp-types entry and corresponding marshalling
The actual ''-blockdev'' argument is considerably more complex than this but
functions.  This means the type is usuable via both QMP and through the
this is enough to illustrate the pointThe following invocations are all
command lineTo generate a device factory interface, we just use the
valid:
following schema entry:


  { 'device': 'isa-serial', 'config': 'ISASerialConfig' }
  -blockdev file=myimage.img,backing-file.file=mybase.img
-blockdev file=myimage.img,backing-file={file=mybase.img,backing-file.file=myotherbase.img}


This would generate the following code:
The structure generated for ''-blockdev'' will look like:


  DeviceState *qdev_ctor_isa_serial(ISASerialConfig *config, Error **errp);
  typedef struct BlockdevConfig {
    char * file;
    struct BlockdevConfig * backing_file;
   
   
static DeviceState *qdev_marshal_isa_serial(KeyValues *kv, Error **errp)
     struct BlockdevConfig * next;
{
  } BlockdevConfig;
     ISASerialConfig *config;
 
    Error *local_err = NULL;
The configuration file syntax will work the same:
    DeviceState *dev;
 
   
[blockdev]
    config = qcfg_unmarshal_ISASerialConfig(kv, &local_err);
  file=myimage.img
    if (local_err) {
  backing-file.file=mybase.img
        error_propagate(errp, local_err);
  backing-file.backing-file.file=myotherbase.img
        return;
    }
   
    dev = qdev_ctor_isa_serial(config, errp);
   
    qmp_free_isa_serial_config(config);
   
    return dev;
}


This lets QMP invoke qdev constructors directly without remarshaling to
The curly braces syntax is not supported in the configuration file syntax.
strings. It also provides a convenient interface for QEMU to consume internally
to create devices which is fully type safe.


It also provides a mechanism to do full introspection of qdev and a consistent
== Further Enhancements ==
way to provide documentation.


== Open Questions ==
After every command line parameter has been reduced to just calling
''qcfg_handle_option()'', we can remove the switch statement option handling
entirely and modify the code generator to automatically associate option names
with the appropriate dispatch function.  This eliminates the need to make any
changes to vl.c in order to add options.


* Should we use a structure or marshal parameters as function calls?
[[Category:Obsolete feature pages]]

Latest revision as of 15:59, 17 October 2016

Summary

Replace QEMU command option handling and QemuOpts with a new infrastructure that supports full introspection and QMP interoperability.

Owner

Background

In order to add a new command line interface to QEMU, we currently do the following:

  1. Add a new entry to qemu-options.hx with embedded LaTeX documentation. This only describes the option name and whether it takes an option, it does not cover the format of the option. Documentation about the format of the option is in free form English and is not necessarily exhaustive.
  2. The option should make use of QemuOpts--although many don't. To use QemuOpts, one must define a schema in qemu-config.c (usually).
  3. A new case statement is added to vl.c which will then parse the option, later in the main routine, a function is iterated over each QemuOpts that has been parsed.
  4. That function pointer usually is a function defined somewhere else in the code base. That function now has to interact with QemuOpts in a way that's agreeable with the schema in qemu-config.c
  5. In many cases, the schema can not be applied because of limitations in the schema format. In this case, the function pointer is totally responsible for syntax validation.

Besides the considerable complexity of this, there are a lot of places in the code that must be in agreement. This makes it very easy to end up with bugs where options aren't properly validated.

Since QemuOpts finds it way deep into the code, for QMP interfaces that need to interact in the same way, we need to generate a QemuOpts structure from a QMP data structure. This currently works by converting the QMP data structures to strings and then back to QemuOpts. This results in some very unusual behavior.

Unifying configuration

QCFG uses a schema in the same fashion as QAPI to allow for the expression of complex data structures and provide a structured way to specify detailed documentation.

Here is an example of this mechanism applied to the -vnc option:

##
# @VncConfig:
#
# Configuration options for the built-in VNC server.
#
# @address:  The hostname to bind the VNC server to.  If specified in the form
#            `:display', the server binds to any interface on port
#            (display + 5900).  If specified in the form, `hostname:display',
#            the server is bound to `hostname' interface on port
#            (display + 5900).  If specified in the form, `unix:path', then the
#            server is bound to the unix domain socket `path'.
#
# @password: #optional If true, VNC authentication is enabled.  Use the
#            @set-password command in QMP to set the password.
#
# @reverse:  #optional If true, @address is treated as a client address and the
#            server will connect to @address and then proceed with the server
#            session.  This can be helpful when working through a proxy.
#
# @no-lock-key-sync: #optional If true, disable the built-in heuristics that
#                    attempt to key the Caps and Num Lock keys synchronized
#                    with the host.
#
# @sasl:     #optional If true, enable SASL authentication.
#
# @tls:      #optional If true, use TLS to encrypt the session.
#
# @x509:     #optional The location of an x509 certificate to use to encrypt
#            the session.  This cannot be used in conjunction with @tls.
#
# @x509verify: #optional The location of an x509 certificate to use to encrypt
#              the session.  This is similar to @x509 but it also verifies the
#              clients x509 certificates when they connect.
#
# @acl:      #optional If true, enable VNC ACL support.  ACLs can be set through
#            QMP.
#
# @lossy:    #optional If true, compression will be enabled that can degrade
#            image quality but result in significantly less bandwidth.
#
# Since: 0.14.0
##
{ 'type': 'VncConfig',
  'data': { 'address': 'str', '*password': 'bool', '*reverse': 'bool',
            '*no-lock-key-sync': 'bool', '*sasl': 'bool', '*tls': 'bool',
            '*x509': 'str', '*x509verify': 'str', '*acl': 'bool',
            '*lossy': 'bool' } }

##
# @vnc:
#
# Enable the built-in VNC server for the guest's VGA output.
#
# Since: 0.14.0
##
{ 'option': 'vnc', 'data': 'VncConfig' }

To make use of this in QEMU, we simply need to add the following to vl.c:

case QEMU_OPTION_vnc:
    qcfg_handle_option("vnc", optarg);
    break;

The effect of this will be a function call to qcfg_handle_vnc() passing in a native format structure. Below is an example of the function call and the types generated to support it:

typedef struct VncConfig {
    char * address;
    bool has_password;
    bool password;
    bool has_reverse;
    bool reverse;
    bool has_no_lock_key_sync;
    bool no_lock_key_sync;
    bool has_sasl;
    bool sasl;
    bool has_x509;
    char * x509;
    bool has_x509verify;
    char * x509verify;
    bool has_acl;
    bool acl;
    bool has_lossy;
    bool lossy;

    struct VncConfig * next;
} VncConfig;

void qcfg_handle_vnc(VncConfig *config, Error **errp);

The option handler may either process the option immediately (before other options are parsed) or save the option in a global list for later processing.

No extra code is required to expose this over QMP. We can either expose a new QMP function that takes a VncConfig structure that is implemented by calling this function or we can expose every option handler through automatically generated QMP commands.

Aliases

For command line options that have multiple aliases, they can be reduced to the following:

{ 'option': 'm', 'data': 'size_mb', 'aliases': [ 'memory' ] }

This allows the option to be specified as either -m or -memory.

Aliasing is particularly useful for configuration file support.

Configuration File Support

While the qcfg_handle_option() interface takes a string, internally, all of the functions are implemented with KeyValues lists. The qcfg_handle_option() function will parse the string into a list of KeyValues.

This allows all of the infrastructure to be used with a configuration file. The format of the file is as follows:

[vnc]
address=localhost:5
sasl=on
tls=on

This translates roughly to:

qcfg_handle_option("vnc", "address=localhost:5,sasl=on,tls=on");

Complex Data Types

There are certain options that require nested structures and even recursive structures. We will express these structures using a dotted syntax. For instance, consider -blockdev:

{ 'type': 'BlockdevConfig',
  'data': { 'file': 'str', '*backing-file': 'BlockdevConfig' } }

The actual -blockdev argument is considerably more complex than this but this is enough to illustrate the point. The following invocations are all valid:

-blockdev file=myimage.img,backing-file.file=mybase.img
-blockdev file=myimage.img,backing-file={file=mybase.img,backing-file.file=myotherbase.img}

The structure generated for -blockdev will look like:

typedef struct BlockdevConfig {
    char * file;
    struct BlockdevConfig * backing_file;

    struct BlockdevConfig * next;
} BlockdevConfig;

The configuration file syntax will work the same:

[blockdev]
file=myimage.img
backing-file.file=mybase.img
backing-file.backing-file.file=myotherbase.img

The curly braces syntax is not supported in the configuration file syntax.

Further Enhancements

After every command line parameter has been reduced to just calling qcfg_handle_option(), we can remove the switch statement option handling entirely and modify the code generator to automatically associate option names with the appropriate dispatch function. This eliminates the need to make any changes to vl.c in order to add options.