Features/QTest

Owners

Michael Roth mdroth at linux.vnet.ibm.com

Stefan Hajnoczi stefanha at linux.vnet.ibm.com

Summary

Modify QEMU so that custom code can be run as VCPUs in place of TCG/KVM threads. By doing so we can increase our ability to more directly exercise various areas of QEMU's device model, aiding in our efforts to identify/address potential scalability and reliability issues, and ultimately providing a framework for implementing extensive unit testing within QEMU.

QEMU currently lacks a standard means to do targeted unit testing of the device model. Frameworks like kvm-autotest interact via guest OS, which provide a highly abstracted interface to the underlying machine, and are susceptable to bugs in the guest OS itself. This allows for reasonable test coverage of guest functionality as a whole, but reduces the accuracy and specificity with which we can exercise paths in the underlying devices.

The following patches provide the basic beginnings of a test framework which replaces vcpu threads with test threads that interact with the underlying machine directly, allowing for directed unit/performance testing of individual devices. Test modules are built directly into the qemu binary, and each module provides the following interfaces:

init():
  Called in place of qemu's normal machine initialization to setup up devices explicitly.
  A full machine can be created here by calling into the normal init path, as well as minimal
  machines with a select set of buses/devices/IRQ handlers.

run():
  Test logic that interacts with the now-created machine.

build/example usage

$ ./configure --target-list=x86_64-softmmu --enable-qtest --enable-io-thread
$ make
$ ./x86_64-softmmu/qemu-system-x86_64 -test ?
Available test modules:
rtc
$ ./x86_64-softmmu/qemu-system-x86_64 -test rtc
../qtest/qtest_rtc.c:test_drift():L94: hz: 2, duration_ms: 4999, exp_duration: 5000, drift ratio: 0.000200
../qtest/qtest_rtc.c:test_drift():L111: hz: 1024, duration_ms: 4999, exp_duration: 5000, drift ratio: 0.000200

Additional Details

Currently QTest is invoked with a simple command-line parameter:

qemu -test <test module name>

This causes QEMU to skip creation of KVM/TCG threads and normal machine init routines. Instead, the test module's machine init routine is called, and then a thread is started which executes the test module's run routine. The QEMU event loop still drives timers/IO.

A likely addition here would be the ability to pass parameters to the test module. One interesting use case suggested on qemu-devel was to have a block test which could be passed a file containing block traces for the test module to replay and compare against expected output. This would also be useful for performance-oriented tests, for which tweakable parameters could be passed in simpler module use/re-use.

The granularity with which individual devices can be created/utilized is limited only by the inherent constraints of the device model implementation. Currently we have the ability to do targeted testing with certain items: the qtest "rtc" module for instance creates a "machine" which consists only of memory, an isa bus, and an rtc clock. The need for a PIC is replaced with a custom IRQ handler that probes the rtc. Basic tests such as PIO operations on a PCI host bridge are also demonstrated in the "pci" test module in current tree. However, more complex devices such as network cards will unsurprisingly pull in a lot of additional devices that make sense from a machine emulation perspective, but may call for unwieldy unit tests. It is not yet clear how well QTest will work for these situations.