Features/QTest: Difference between revisions
| Line 43: | Line 43: | ||
This causes QEMU to skip creation of KVM/TCG threads and normal machine init routines. Instead, the test module's machine init routine is called, and then a thread is started which executes the test module's run routine. The QEMU event loop still drives timers/IO. | This causes QEMU to skip creation of KVM/TCG threads and normal machine init routines. Instead, the test module's machine init routine is called, and then a thread is started which executes the test module's run routine. The QEMU event loop still drives timers/IO. | ||
The granularity with which individual devices can be created/utilized is limited only by the | A likely addition here would be the ability to pass parameters to the test module. One interesting use case suggested on qemu-devel was to have a block test which could be passed a file containing block traces for the test module to replay and compare against expected output. This would also be useful for performance-oriented tests, for which tweakable parameters could be passed in simpler module use/re-use. | ||
The granularity with which individual devices can be created/utilized is limited only by the inherent constraints of the device model implementation. Currently we have the ability to do targeted testing with certain items: the qtest "rtc" module for instance creates a "machine" which consists only of memory, an isa bus, and an rtc clock. The need for a PIC is replaced with a custom IRQ handler that probes the rtc. Basic tests such as PIO operations on a PCI host bridge are also demonstrated in the "pci" test module in current tree. However, more complex devices such as network cards will unsurprisingly pull in a lot of additional devices that make sense from a machine emulation perspective, but may call for unwieldy unit tests. It is not yet clear how well QTest will work for these situations. | |||
Revision as of 02:17, 9 March 2011
Owners
Michael Roth mdroth at linux.vnet.ibm.com
Stefan Hajnoczi stefanha at linux.vnet.ibm.com
Summary
Modify QEMU so that custom code can be run as VCPUs in place of TCG/KVM threads. By doing so we can increase our ability to more directly exercise various areas of QEMU's device model, aiding in our efforts to identify/address potential scalability and reliability issues, and ultimately providing a framework for implementing extensive unit testing within QEMU.
QEMU currently lacks a standard means to do targeted unit testing of the device model. Frameworks like kvm-autotest interact via guest OS, which provide a highly abstracted interface to the underlying machine, and are susceptable to bugs in the guest OS itself. This allows for reasonable test coverage of guest functionality as a whole, but reduces the accuracy and specificity with which we can exercise paths in the underlying devices.
The following patches provide the basic beginnings of a test framework which replaces vcpu threads with test threads that interact with the underlying machine directly, allowing for directed unit/performance testing of individual devices. Test modules are built directly into the qemu binary, and each module provides the following interfaces:
init(): Called in place of qemu's normal machine initialization to setup up devices explicitly. A full machine can be created here by calling into the normal init path, as well as minimal machines with a select set of buses/devices/IRQ handlers. run(): Test logic that interacts with the now-created machine.
build/example usage
$ ./configure --target-list=x86_64-softmmu --enable-qtest --enable-io-thread $ make $ ./x86_64-softmmu/qemu-system-x86_64 -test ? Available test modules: rtc $ ./x86_64-softmmu/qemu-system-x86_64 -test rtc ../qtest/qtest_rtc.c:test_drift():L94: hz: 2, duration_ms: 4999, exp_duration: 5000, drift ratio: 0.000200 ../qtest/qtest_rtc.c:test_drift():L111: hz: 1024, duration_ms: 4999, exp_duration: 5000, drift ratio: 0.000200
Additional Details
Currently QTest is invoked with a simple command-line parameter:
qemu -test <test module name>
This causes QEMU to skip creation of KVM/TCG threads and normal machine init routines. Instead, the test module's machine init routine is called, and then a thread is started which executes the test module's run routine. The QEMU event loop still drives timers/IO.
A likely addition here would be the ability to pass parameters to the test module. One interesting use case suggested on qemu-devel was to have a block test which could be passed a file containing block traces for the test module to replay and compare against expected output. This would also be useful for performance-oriented tests, for which tweakable parameters could be passed in simpler module use/re-use.
The granularity with which individual devices can be created/utilized is limited only by the inherent constraints of the device model implementation. Currently we have the ability to do targeted testing with certain items: the qtest "rtc" module for instance creates a "machine" which consists only of memory, an isa bus, and an rtc clock. The need for a PIC is replaced with a custom IRQ handler that probes the rtc. Basic tests such as PIO operations on a PCI host bridge are also demonstrated in the "pci" test module in current tree. However, more complex devices such as network cards will unsurprisingly pull in a lot of additional devices that make sense from a machine emulation perspective, but may call for unwieldy unit tests. It is not yet clear how well QTest will work for these situations.