Features/QTest

Owners

Michael Roth mdroth at linux.vnet.ibm.com

Stefan Hajnoczi stefanha at linux.vnet.ibm.com

Summary

Modify QEMU so that custom code can be run as VCPUs in place of TCG/KVM threads. By doing so we can increase our ability to more directly exercise various areas of QEMU's device model, aiding in our efforts to identify/address potential scalability and reliability issues, and ultimately providing a framework for implementing extensive unit testing within QEMU.

QEMU currently lacks a standard means to do targeted unit testing of the device model. Frameworks like kvm-autotest interact via guest OS, which provide a highly abstracted interface to the underlying machine, and are susceptable to bugs in the guest OS itself. This allows for reasonable test coverage of guest functionality as a whole, but reduces the accuracy and specificity with which we can exercise paths in the underlying devices.

The following patches provide the basic beginnings of a test framework which replaces vcpu threads with test threads that interact with the underlying machine directly, allowing for directed unit/performance testing of individual devices. Test modules are built directly into the qemu binary, and each module provides the following interfaces:

init():
  Called in place of qemu's normal machine initialization to setup up devices explicitly.
  A full machine can be created here by calling into the normal init path, as well as minimal
  machines with a select set of buses/devices/IRQ handlers.

run():
  Test logic that interacts with the now-created machine.

build/example usage

$ ./configure --target-list=x86_64-softmmu --enable-qtest --enable-io-thread
$ make
$ ./x86_64-softmmu/qemu-system-x86_64 -test ?
Available test modules:
rtc
$ ./x86_64-softmmu/qemu-system-x86_64 -test rtc
../qtest/qtest_rtc.c:test_drift():L94: hz: 2, duration_ms: 4999, exp_duration: 5000, drift ratio: 0.000200
../qtest/qtest_rtc.c:test_drift():L111: hz: 1024, duration_ms: 4999, exp_duration: 5000, drift ratio: 0.000200

Additional Details

Currently QTest is invoked with a simple command-line parameter:

qemu -test <test module name>

This causes QEMU to skip creation of KVM/TCG threads and normal machine init routines. Instead, the test module's machine init routine is called, and then a thread is started which executes the test module's run routine. The QEMU event loop still drives timers/IO.

The granularity with which individual devices can be created/utilized is limited only by the inherant constraints of the device model implementation. Currently we have the ability to do targetted testing with certain items: the qtest "rtc" module for instance creates a "machine" which consists only of memory, an isa bus, and an rtc clock. The need for a PIC with a custom IRQ handler that probes the rtc. Basic interaction such as PIO on a PCI host bridge are also demonstrated in the current tree. More complex devices however will unsurprisingly pull in a lot of additional devices that make sense from a machine emulation perspective, but may call for unwieldly test cases.