Features/QTest: Difference between revisions
No edit summary |
|||
| Line 35: | Line 35: | ||
../qtest/qtest_rtc.c:test_drift():L111: hz: 1024, duration_ms: 4999, exp_duration: 5000, drift ratio: 0.000200 | ../qtest/qtest_rtc.c:test_drift():L111: hz: 1024, duration_ms: 4999, exp_duration: 5000, drift ratio: 0.000200 | ||
== Additional Details == | == Additional Details/Status == | ||
Currently QTest is invoked with a simple command-line parameter: | Currently QTest is invoked with a simple command-line parameter: | ||
| Line 43: | Line 43: | ||
This causes QEMU to skip creation of KVM/TCG threads and normal machine init routines. Instead, the test module's machine init routine is called, and then a thread is started which executes the test module's run routine. The QEMU event loop still drives timers/IO. | This causes QEMU to skip creation of KVM/TCG threads and normal machine init routines. Instead, the test module's machine init routine is called, and then a thread is started which executes the test module's run routine. The QEMU event loop still drives timers/IO. | ||
A likely addition here would be the ability to pass parameters to the test module. One interesting use case suggested on qemu-devel was to have a block test which could be passed a file containing block traces for the test module to replay and compare against expected output. This would also be useful for performance-oriented tests, for which tweakable parameters could be passed in simpler module use/re-use. | A likely addition here would be the ability to pass parameters to the test module. One interesting use case suggested on qemu-devel was to have a block test which could be passed a file containing block traces for the test module to replay and compare against expected output. This would also be useful for performance-oriented tests, for which tweakable parameters could be passed in simpler module use/re-use. We also are not strictly limited to testing devices, qtest could be used to drive units tests for other components such as QMP's JSON parsing routines. | ||
The granularity with which individual devices can be created/utilized is limited only by the inherent constraints of the device model implementation. Currently we have the ability to do targeted testing with certain items: the qtest "rtc" module for instance creates a "machine" which consists only of memory, an isa bus, and an rtc clock. The need for a PIC is replaced with a custom IRQ handler that probes the rtc. Basic tests such as PIO operations on a PCI host bridge are also demonstrated in the "pci" test module in current tree. However, more complex devices such as network cards will unsurprisingly pull in a lot of additional devices that make sense from a machine emulation perspective, but may call for unwieldy unit tests. It is not yet clear how well QTest will work for these situations. | The granularity with which individual devices can be created/utilized is limited only by the inherent constraints of the device model implementation. Currently we have the ability to do targeted testing with certain items: the qtest "rtc" module for instance creates a "machine" which consists only of memory, an isa bus, and an rtc clock. The need for a PIC is replaced with a custom IRQ handler that probes the rtc. Basic tests such as PIO operations on a PCI host bridge are also demonstrated in the "pci" test module in current tree. However, more complex devices such as network cards will unsurprisingly pull in a lot of additional devices that make sense from a machine emulation perspective, but may call for unwieldy unit tests. It is not yet clear how well QTest will work for these situations. | ||
A more coherant test harness such as those offered by glib is also desirable, qtest modules are currently very open-ended in how errors are handled. Also, currently a failed/aborted test will still result in the qemu process exiting successfully, making it different to leverage in automated fashion for large sets of tests or `make test` functionality. | |||
== Code/Submissions == | |||
Current devel tree: | |||
git://repo.or.cz/qemu/mdroth.git qtest-dev | |||
RFC v1: | |||
git://repo.or.cz/qemu/mdroth.git qtest_v1 | |||
http://www.mail-archive.com/qemu-devel@nongnu.org/msg54191.html | |||
Revision as of 12:10, 9 March 2011
Owners
Michael Roth mdroth at linux.vnet.ibm.com
Stefan Hajnoczi stefanha at linux.vnet.ibm.com
Summary
Modify QEMU so that custom code can be run as VCPUs in place of TCG/KVM threads. By doing so we can increase our ability to more directly exercise various areas of QEMU's device model, aiding in our efforts to identify/address potential scalability and reliability issues, and ultimately providing a framework for implementing extensive unit testing within QEMU.
QEMU currently lacks a standard means to do targeted unit testing of the device model. Frameworks like kvm-autotest interact via guest OS, which provide a highly abstracted interface to the underlying machine, and are susceptable to bugs in the guest OS itself. This allows for reasonable test coverage of guest functionality as a whole, but reduces the accuracy and specificity with which we can exercise paths in the underlying devices.
The following patches provide the basic beginnings of a test framework which replaces vcpu threads with test threads that interact with the underlying machine directly, allowing for directed unit/performance testing of individual devices. Test modules are built directly into the qemu binary, and each module provides the following interfaces:
init(): Called in place of qemu's normal machine initialization to setup up devices explicitly. A full machine can be created here by calling into the normal init path, as well as minimal machines with a select set of buses/devices/IRQ handlers. run(): Test logic that interacts with the now-created machine.
build/example usage
$ ./configure --target-list=x86_64-softmmu --enable-qtest --enable-io-thread $ make $ ./x86_64-softmmu/qemu-system-x86_64 -test ? Available test modules: rtc $ ./x86_64-softmmu/qemu-system-x86_64 -test rtc ../qtest/qtest_rtc.c:test_drift():L94: hz: 2, duration_ms: 4999, exp_duration: 5000, drift ratio: 0.000200 ../qtest/qtest_rtc.c:test_drift():L111: hz: 1024, duration_ms: 4999, exp_duration: 5000, drift ratio: 0.000200
Additional Details/Status
Currently QTest is invoked with a simple command-line parameter:
qemu -test <test module name>
This causes QEMU to skip creation of KVM/TCG threads and normal machine init routines. Instead, the test module's machine init routine is called, and then a thread is started which executes the test module's run routine. The QEMU event loop still drives timers/IO.
A likely addition here would be the ability to pass parameters to the test module. One interesting use case suggested on qemu-devel was to have a block test which could be passed a file containing block traces for the test module to replay and compare against expected output. This would also be useful for performance-oriented tests, for which tweakable parameters could be passed in simpler module use/re-use. We also are not strictly limited to testing devices, qtest could be used to drive units tests for other components such as QMP's JSON parsing routines.
The granularity with which individual devices can be created/utilized is limited only by the inherent constraints of the device model implementation. Currently we have the ability to do targeted testing with certain items: the qtest "rtc" module for instance creates a "machine" which consists only of memory, an isa bus, and an rtc clock. The need for a PIC is replaced with a custom IRQ handler that probes the rtc. Basic tests such as PIO operations on a PCI host bridge are also demonstrated in the "pci" test module in current tree. However, more complex devices such as network cards will unsurprisingly pull in a lot of additional devices that make sense from a machine emulation perspective, but may call for unwieldy unit tests. It is not yet clear how well QTest will work for these situations.
A more coherant test harness such as those offered by glib is also desirable, qtest modules are currently very open-ended in how errors are handled. Also, currently a failed/aborted test will still result in the qemu process exiting successfully, making it different to leverage in automated fashion for large sets of tests or `make test` functionality.
Code/Submissions
Current devel tree: git://repo.or.cz/qemu/mdroth.git qtest-dev
RFC v1: git://repo.or.cz/qemu/mdroth.git qtest_v1 http://www.mail-archive.com/qemu-devel@nongnu.org/msg54191.html