ToDo/QMPTesting: Difference between revisions

From QEMU
Line 28: Line 28:
C level unit-testing using [http://check.sourceforge.net check]. This is used to check QEMU's internal APIs.
C level unit-testing using [http://check.sourceforge.net check]. This is used to check QEMU's internal APIs.


Currently we have good coverage of QObjects types (QInt, QString, etc) and the JSON parser. But we need to expand this to the Monitor code.
Currently we have good coverage of QObjects types (QInt, QString, etc) and the JSON parser. We do need to expand this to the Monitor code.


There are two difficulties associated with this, though:
There are two difficulties associated with this, though:


# Monitor code is not modular and has not been designed with unit-tests in mind
# Monitor code is not modular and has not been designed with unit-tests in mind
# It's unlikely that it will be possible to test handlers code this way, as most of them depends on other subsystems
# It's unlikely that it will be possible to test handlers code this way, as most of them depend on other subsystems


Item 1 will be solved when the current monitor redesign work is finished.
Item 1 will be solved when the current monitor redesign work is finished.
Line 42: Line 42:
# Introduce a unit-test tree structure in QEMU
# Introduce a unit-test tree structure in QEMU
# Add a 'make test' target, which runs *all* QEMU's unit-tests
# Add a 'make test' target, which runs *all* QEMU's unit-tests
# Add unit-tests for new monitor code
# Add unit-tests for the new monitor code


== libvirt-TCK ==
== libvirt-TCK ==


This is libvirt's automated test framework. Although this goes beyond QMP testing, we should ensure that any libvirt API which uses QMP gets properly covered.
This is libvirt's automated test framework. Although this goes beyond QMP testing, we should ensure that any libvirt API which uses QMP gets properly covered.

Revision as of 22:19, 11 October 2010

QMP Testing TODO

This page describes the work to be done on improving QMP test coverage and quality.

We have a few tools/frameworks available for this. The following sections describe them, more or less in importance order. Please, note that a number of items can be addressed in parallel.

kvm-autotest

Page: http://www.linux-kvm.org/page/KVM-Autotest.

kvm-autotest is going to be used to do all client-side QMP testing.

This work can be split in three parts:

  1. Testing that the basic protocol works as specified. That is, ensuring that the greeting message, success responses and error messages contain the basic information the spec says they do. More importantly, several errors conditions at the protocol level should be tested
  2. Test that each available command behaves as specified by its *own* specification. This is a big effort, as each command should have its own test suite
  3. Asynchronous messages testing. Like command testing, each asynchronous message should be tested separately

TODO

  1. Introduce the basic protocol suite as explained above (Luiz is working on this)
  2. Not all existing kvm-autotest tests work under QMP, fix them (eg. migration exec)
  3. Add tests for all commands available in QMP. Each command should have its own test suite, this is a big effort and should be done continuously
  4. Add tests for asynchronous messages

unit-tests

C level unit-testing using check. This is used to check QEMU's internal APIs.

Currently we have good coverage of QObjects types (QInt, QString, etc) and the JSON parser. We do need to expand this to the Monitor code.

There are two difficulties associated with this, though:

  1. Monitor code is not modular and has not been designed with unit-tests in mind
  2. It's unlikely that it will be possible to test handlers code this way, as most of them depend on other subsystems

Item 1 will be solved when the current monitor redesign work is finished.

TODO:

  1. Fix problems in current unit-tests (try compiling and running them to see what happens)
  2. Introduce a unit-test tree structure in QEMU
  3. Add a 'make test' target, which runs *all* QEMU's unit-tests
  4. Add unit-tests for the new monitor code

libvirt-TCK

This is libvirt's automated test framework. Although this goes beyond QMP testing, we should ensure that any libvirt API which uses QMP gets properly covered.