Latest revision as of 08:17, 12 October 2016

QMP Testing TODO

This page describes the work to be done on improving QMP test coverage and quality.

We have a few tools/frameworks available for this. The following sections describe them, more or less in importance order. Please, note that a number of items can be addressed in parallel.

kvm-autotest

Page: http://www.linux-kvm.org/page/KVM-Autotest.

kvm-autotest is going to be used to do all client-side QMP testing.

This work can be split in three parts:

Testing that the basic protocol works as specified. That is, ensuring that the greeting message, success responses and error messages contain the basic information the spec says they do. More importantly, several errors conditions at the protocol level should be tested
Test that each available command behaves as specified by its *own* specification. This is a big effort, as each command should have its own test suite
Asynchronous messages testing. Like command testing, each asynchronous message should be tested separately

TODO

~~Introduce the basic protocol suite as explained above~~ (Luiz)
Not all existing kvm-autotest tests work under QMP, fix them (eg. migration exec)
Add tests for all commands available in QMP. Each command should have its own test suite, this is a big effort and should be done continuously
Add tests for asynchronous messages

unit-tests

C level unit-testing using check. This is used to check QEMU's internal APIs.

Currently we have good coverage of QObjects types (QInt, QString, etc) and the JSON parser. We do need to expand this to the Monitor code.

There are two difficulties associated with this, though:

Monitor code is not modular and has not been designed with unit-tests in mind
It's unlikely that it will be possible to test handlers code this way, as most of them depend on other subsystems

Item 1 will be solved when the current monitor redesign work is finished.

TODO

Introduce a unit-test tree structure in QEMU
Add a 'make test' target, which runs *all* QEMU's unit-tests
Add unit-tests for the new monitor code

libvirt-TCK

Git repository: git://libvirt.org/libvirt-tck.git

This is libvirt's automated test framework. Although this goes beyond QMP testing, we should ensure that any libvirt API which uses QMP gets properly covered.

TODO

Check what's the current status wrt libvirt APIs coverage using QMP
Help libvirt guys expanding the suite

@@ Line 1: / Line 1: @@
 = QMP Testing TODO =
-This page describes the work to be done on improving QMP test coverage and quality. We have the following tools/framework available:
+This page describes the work to be done on improving QMP test coverage and quality.
-* QEMU's unit-tests
+We have a few tools/frameworks available for this. The following sections describe them, more or less in importance order. Please, note that a number of items can be addressed in parallel.
-* kvm-autotest
-* libvirt-TCK
+== kvm-autotest ==
+Page: http://www.linux-kvm.org/page/KVM-Autotest.
+kvm-autotest is going to be used to do all client-side QMP testing.
+This work can be split in three parts:
+# Testing that the basic protocol works as specified. That is, ensuring that the greeting message, success responses and error messages contain the basic information the spec says they do. More importantly, several errors conditions at the protocol level should be tested
+# Test that each available command behaves as specified by its *own* specification. This is a big effort, as each command should have its own test suite
+# Asynchronous messages testing. Like command testing, each asynchronous message should be tested separately
+=== TODO ===
+# <strike>Introduce the basic protocol suite as explained above</strike> (Luiz)
+# Not all existing kvm-autotest tests work under QMP, fix them (eg. migration exec)
+# Add tests for all commands available in QMP. Each command should have its own test suite, this is a big effort and should be done continuously
+# Add tests for asynchronous messages
 == unit-tests ==
-C level testing, used to check QEMU's internal APIs.
+C level unit-testing using [http://check.sourceforge.net check]. This is used to check QEMU's internal APIs.
+Currently we have good coverage of QObjects types (QInt, QString, etc) and the JSON parser. We do need to expand this to the Monitor code.
-Currently we have a good coverage of QObjects types (QInt, QString, etc) and the JSON parser. We need to test as much Monitor code as possible, however there are two difficulties associated with this:
+There are two difficulties associated with this, though:
 # Monitor code is not modular and has not been designed with unit-tests in mind
-# It's unlikely that it will be possible to test handlers code this way, as most of them depends on other subsystems
+# It's unlikely that it will be possible to test handlers code this way, as most of them depend on other subsystems
 Item 1 will be solved when the current monitor redesign work is finished.
-TODO:
+=== TODO ===
-# Fix problems in current unit-tests (try compiling and running them to see what happens)
 # Introduce a unit-test tree structure in QEMU
 # Add a 'make test' target, which runs *all* QEMU's unit-tests
-# Add unit-tests for new monitor code
+# Add unit-tests for the new monitor code
-== kvm-autotest ==
+== libvirt-TCK ==
-kvm-autotest is going to be used to do all client-side QMP testing.
+Git repository: git://libvirt.org/libvirt-tck.git
-This work can be split in three parts:
+This is libvirt's automated test framework. Although this goes beyond QMP testing, we should ensure that any libvirt API which uses QMP gets properly covered.
-# Testing that the basic protocol works as specified. That is, ensuring that the greeting message, success responses and error messages contain the basic information the spec says they do. More importantly, several errors conditions at the protocol level should be tested
+=== TODO ===
-# Test that each available command behaves as specified by its *own* specification. This is a big effort, as each command should have its own test suite
-# Asynchronous messages testing. Like command testing, each asynchronous message should be tested separately
-== libvirt-TCK ==
+# Check what's the current status wrt libvirt APIs coverage using QMP
+# Help libvirt guys expanding the suite