|
|
| Line 1: |
Line 1: |
| == Status ==
| |
|
| |
| '''This page is now obsolete. qtest has been merged and is part of the qemu.git tree.'''
| |
|
| |
| Please see [http://git.qemu-project.org/?p=qemu.git;a=blob;f=tests/libqtest.h;hb=HEAD tests/libqtest.h] for device test APIs. Search the tests/ directory for examples using libqtest.
| |
|
| |
| == Owners ==
| |
|
| |
| Stefan Hajnoczi (stefanha at linux.vnet.ibm.com)
| |
|
| |
| Michael Roth (mdroth at linux.vnet.ibm.com)
| |
|
| |
| == Summary == | | == Summary == |
|
| |
|
| Modify QEMU so that custom code can be run as VCPUs in place of TCG/KVM threads. By doing so we can increase our ability to more directly exercise various areas of QEMU's device model, aiding in our efforts to identify/address potential scalability and reliability issues, and ultimately providing a framework for implementing extensive unit testing within QEMU.
| | QTest is an internal framework used in the QEMU unit tests. A QTest based test will spin up one or more QEMU binaries and orchestrate the test via a qtest control socket. The binaries themselves are usually controlled using a QMP socket to trigger events. |
|
| |
|
| | == Debugging == |
|
| |
|
| QEMU currently lacks a standard means to do targeted unit testing of the device model. Frameworks like kvm-autotest interact via guest OS, which provide a highly abstracted interface to the underlying machine, and are susceptable to bugs in the guest OS itself. This allows for reasonable test coverage of guest functionality as a whole, but reduces the accuracy and specificity with which we can exercise paths in the underlying devices.
| | Usually the first problem a new developer comes across is understanding how a test fails. The qtest tests are all grouped together is target architectures so the first step is to re-run make check for the particular architecture that failed: |
|
| |
|
| The following patches provide the basic beginnings of a test framework which replaces vcpu threads with test threads that interact with the underlying machine directly, allowing for directed unit/performance testing of individual devices. Test modules are built directly into the qemu binary, and each module provides the following interfaces:
| | make check-qtest-i386 V=1 |
|
| |
|
| init():
| | The test run will now dump a lot more information about how the test is run. This will be a gtester call with a large number of tests included. This can then be repeated with just the tests you want. For example: |
| Called in place of qemu's normal machine initialization to setup up devices explicitly.
| |
| A full machine can be created here by calling into the normal init path, as well as minimal
| |
| machines with a select set of buses/devices/IRQ handlers.
| |
|
| |
| run():
| |
| Test logic that interacts with the now-created machine.
| |
|
| |
|
| == build/example usage == | | QTEST_QEMU_BINARY=i386-softmmu/qemu-system-i386 QTEST_QEMU_IMG=qemu-img MALLOC_PERTURB_=${MALLOC_PERTURB_:-$((RANDOM % 255 + 1))} gtester -k --verbose -m=quick tests tests/vhost-user-test |
|
| |
|
| $ ./configure --target-list=x86_64-softmmu --enable-qtest --enable-io-thread
| | For additional information you can set QTEST_LOG and also use qtester -p to specify the subtest you want to run: |
| $ make
| |
| $ ./x86_64-softmmu/qemu-system-x86_64 -test ?
| |
| Available test modules:
| |
| rtc
| |
| $ ./x86_64-softmmu/qemu-system-x86_64 -test rtc
| |
| ../qtest/qtest_rtc.c:test_drift():L94: hz: 2, duration_ms: 4999, exp_duration: 5000, drift ratio: 0.000200
| |
| ../qtest/qtest_rtc.c:test_drift():L111: hz: 1024, duration_ms: 4999, exp_duration: 5000, drift ratio: 0.000200
| |
|
| |
|
| == Additional Details/Status == | | QTEST_LOG=1 QTEST_QEMU_BINARY=i386-softmmu/qemu-system-i386 QTEST_QEMU_IMG=qemu-img MALLOC_PERTURB_=${MALLOC_PERTURB_:-$((RANDOM % 255 + 1))} gtester -k --verbose -m=quick tests/vhost-user-test -p /i386/vhost-user/migrate |
|
| |
|
| Currently QTest is invoked with a simple command-line parameter:
| | == Implementation == |
|
| |
|
| qemu -test <test module name>
| | Please see [http://git.qemu-project.org/?p=qemu.git;a=blob;f=tests/libqtest.h;hb=HEAD tests/libqtest.h] for device test APIs. Search the tests/ directory for examples using libqtest. |
| | |
| This causes QEMU to skip creation of KVM/TCG threads and normal machine init routines. Instead, the test module's machine init routine is called, and then a thread is started which executes the test module's run routine. The QEMU event loop still drives timers/IO.
| |
| | |
| A likely addition here would be the ability to pass parameters to the test module. One interesting use case suggested on qemu-devel was to have a block test which could be passed a file containing block traces for the test module to replay and compare against expected output. This would also be useful for performance-oriented tests, for which tweakable parameters could be passed in simpler module use/re-use. We also are not strictly limited to testing devices, qtest could be used to drive units tests for other components such as QMP's JSON parsing routines.
| |
| | |
| The granularity with which individual devices can be created/utilized is limited only by the inherent constraints of the device model implementation. Currently we have the ability to do targeted testing with certain items: the qtest "rtc" module for instance creates a "machine" which consists only of memory, an isa bus, and an rtc clock. The need for a PIC is replaced with a custom IRQ handler that probes the rtc. Basic tests such as PIO operations on a PCI host bridge are also demonstrated in the "pci" test module in current tree. However, more complex devices such as network cards will unsurprisingly pull in a lot of additional devices that make sense from a machine emulation perspective, but may call for unwieldy unit tests. It is not yet clear how well QTest will work for these situations.
| |
| | |
| A more coherant test harness such as those offered by glib is also desirable, qtest modules are currently very open-ended in how errors are handled. Also, currently a failed/aborted test will still result in the qemu process exiting successfully, making it different to leverage in automated fashion for large sets of tests or `make test` functionality.
| |
| | |
| == Code/Submissions ==
| |
| | |
| Current devel tree:
| |
| | |
| * git://repo.or.cz/qemu/mdroth.git qtest-dev
| |
| | |
| RFC v1, http://www.mail-archive.com/qemu-devel@nongnu.org/msg54191.html :
| |
|
| |
|
| * git://repo.or.cz/qemu/mdroth.git qtest_v1
| | [[Category:Testing]] |
Summary
QTest is an internal framework used in the QEMU unit tests. A QTest based test will spin up one or more QEMU binaries and orchestrate the test via a qtest control socket. The binaries themselves are usually controlled using a QMP socket to trigger events.
Debugging
Usually the first problem a new developer comes across is understanding how a test fails. The qtest tests are all grouped together is target architectures so the first step is to re-run make check for the particular architecture that failed:
make check-qtest-i386 V=1
The test run will now dump a lot more information about how the test is run. This will be a gtester call with a large number of tests included. This can then be repeated with just the tests you want. For example:
QTEST_QEMU_BINARY=i386-softmmu/qemu-system-i386 QTEST_QEMU_IMG=qemu-img MALLOC_PERTURB_=${MALLOC_PERTURB_:-$((RANDOM % 255 + 1))} gtester -k --verbose -m=quick tests tests/vhost-user-test
For additional information you can set QTEST_LOG and also use qtester -p to specify the subtest you want to run:
QTEST_LOG=1 QTEST_QEMU_BINARY=i386-softmmu/qemu-system-i386 QTEST_QEMU_IMG=qemu-img MALLOC_PERTURB_=${MALLOC_PERTURB_:-$((RANDOM % 255 + 1))} gtester -k --verbose -m=quick tests/vhost-user-test -p /i386/vhost-user/migrate
Implementation
Please see tests/libqtest.h for device test APIs. Search the tests/ directory for examples using libqtest.
