Features/QTest: Difference between revisions

From QEMU
(Update examples)
 
(16 intermediate revisions by 4 users not shown)
Line 1: Line 1:
== Owners ==
== Summary ==
 
QTest is an internal framework used in the QEMU unit tests. A QTest based test will spin up one or more QEMU binaries and orchestrate the test via a qtest control socket. The binaries themselves are usually controlled using a QMP socket to trigger events.
 
== Debugging ==
 
=== Isolating the failing test ===
 
Usually the first problem a new developer comes across is understanding how a test fails. The qtest tests are all grouped together by target architectures so the first step is to re-run make check for the particular architecture that failed, with the V=1 option to make it more verbose:
 
    make check-qtest-i386 V=1
 
You can also run the tests directly with meson:


Stefan Hajnoczi (stefanha at linux.vnet.ibm.com)
    meson test --list | grep qos
    meson test qtest-aarch64/qos-test


Michael Roth (mdroth at linux.vnet.ibm.com)
You can then examine `meson-logs/testlog.tx` to get the invocation details.


== Summary ==
The test run will now dump a lot more information about how the test is run. This will be a gtester call with a large number of tests included. This can then be repeated with just the tests you want. For example:


Modify QEMU so that custom code can be run as VCPUs in place of TCG/KVM threads. By doing so we can increase our ability to more directly exercise various areas of QEMU's device model, aiding in our efforts to identify/address potential scalability and reliability issues, and ultimately providing a framework for implementing extensive unit testing within QEMU.
    env MALLOC_PERTURB_=16 QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon QTEST_QEMU_IMG=./qemu-img G_TEST_DBUS_DAEMON=/home/alex/lsrc/qemu.git/tests/dbus-vmstate-daemon.sh PYTHON=/home/alex/lsrc/qemu.git/builds/all/pyvenv/bin/python3 QTEST_QEMU_BINARY=./qemu-system-aarch64 /home/alex/lsrc/qemu.git/builds/all/tests/qtest/qos-test --tap -k


For additional information you can set QTEST_LOG and also use -l to list all the tests and -p to specify the subtest you want to run:


QEMU currently lacks a standard means to do targeted unit testing of the device model. Frameworks like kvm-autotest interact via guest OS, which provide a highly abstracted interface to the underlying machine, and are susceptable to bugs in the guest OS itself. This allows for reasonable test coverage of guest functionality as a whole, but reduces the accuracy and specificity with which we can exercise paths in the underlying devices.
    env QTEST_LOG=1 MALLOC_PERTURB_=16 QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon QTEST_QEMU_IMG=./qemu-img G_TEST_DBUS_DAEMON=/home/alex/lsrc/qemu.git/tests/dbus-vmstate-daemon.sh PYTHON=/home/alex/lsrc/qemu.git/builds/all/pyvenv/bin/python3 QTEST_QEMU_BINARY=./qemu-system-aarch64 /home/alex/lsrc/qemu.git/builds/all/tests/qtest/qos-test --tap -p /aarch64/virt/generic-pcihost/pci-bus-generic/pci-bus/vhost-user-gpio-pci/vhost-user-gpio/vhost-user-gpio-tests/read-guest-mem/memfile


The following patches provide the basic beginnings of a test framework which replaces vcpu threads with test threads that interact with the underlying machine directly, allowing for directed unit/performance testing of individual devices. Test modules are built directly into the qemu binary, and each module provides the following interfaces:
=== Using debugging tools under the test harness ===


init():
Since the QEMU binary is run as a separate process, running it under the control of debugging tools like gdb or valgrind isn't straightforward; you have to tell the test harness to launch the debug tool for you.
  Called in place of qemu's normal machine initialization to setup up devices explicitly.
  A full machine can be created here by calling into the normal init path, as well as minimal
  machines with a select set of buses/devices/IRQ handlers.
run():
  Test logic that interacts with the now-created machine.


== build/example usage ==
For most cases, the simplest thing is to set QTEST_STOP=1. This will cause the qtest harness to send a SIGSTOP to the QEMU process as soon as it has started it and before it tries to run the test. You can then find the stopped QEMU process's PID and connect to it with gdb. Once you've attached gdb to it you can resume the QEMU process with the gdb 'continue' command.


$ ./configure --target-list=x86_64-softmmu --enable-qtest --enable-io-thread
A different approach is to use a command like this to get the test harness to start a new xterm with a gdb in it for each test it runs:
$ make
$ ./x86_64-softmmu/qemu-system-x86_64 -test ?
Available test modules:
rtc
$ ./x86_64-softmmu/qemu-system-x86_64 -test rtc
../qtest/qtest_rtc.c:test_drift():L94: hz: 2, duration_ms: 4999, exp_duration: 5000, drift ratio: 0.000200
../qtest/qtest_rtc.c:test_drift():L111: hz: 1024, duration_ms: 4999, exp_duration: 5000, drift ratio: 0.000200


== Additional Details/Status ==
    QTEST_QEMU_BINARY="xterm -e gdb --tty $(tty) --args ./qemu-system-aarch64" QTEST_QEMU_IMG=qemu-img gtester -k --verbose -m=quick  tests/device-introspect-test


Currently QTest is invoked with a simple command-line parameter:
or in terminal mode (with tmux multiplexer) something like:


  qemu -test <test module name>
    QTEST_QEMU_BINARY="tmux split-window gdb --args ./qemu-system-aarch64" QTEST_QEMU_IMG=qemu-img gtester -k --verbose -m=quick tests/device-introspect-test


This causes QEMU to skip creation of KVM/TCG threads and normal machine init routines. Instead, the test module's machine init routine is called, and then a thread is started which executes the test module's run routine. The QEMU event loop still drives timers/IO.
The gdb will get control before the QEMU process has done anything, so you can set breakpoints and so on before using 'continue' to make it start execution.


A likely addition here would be the ability to pass parameters to the test module. One interesting use case suggested on qemu-devel was to have a block test which could be passed a file containing block traces for the test module to replay and compare against expected output. This would also be useful for performance-oriented tests, for which tweakable parameters could be passed in simpler module use/re-use. We also are not strictly limited to testing devices, qtest could be used to drive units tests for other components such as QMP's JSON parsing routines.
You can use a similar trick to run the QEMU process under valgrind:


The granularity with which individual devices can be created/utilized is limited only by the inherent constraints of the device model implementation. Currently we have the ability to do targeted testing with certain items: the qtest "rtc" module for instance creates a "machine" which consists only of memory, an isa bus, and an rtc clock. The need for a PIC is replaced with a custom IRQ handler that probes the rtc. Basic tests such as PIO operations on a PCI host bridge are also demonstrated in the "pci" test module in current tree. However, more complex devices such as network cards will unsurprisingly pull in a lot of additional devices that make sense from a machine emulation perspective, but may call for unwieldy unit tests. It is not yet clear how well QTest will work for these situations.
    QTEST_QEMU_BINARY="valgrind --vgdb-error=1 --log-file=vg.log qemu-system-aarch64" QTEST_QEMU_IMG=qemu-img MALLOC_PERTURB_=${MALLOC_PERTURB_:-$((RANDOM % 255 + 1))} gtester -k --verbose -m=quick  tests/device-introspect-test


A more coherant test harness such as those offered by glib is also desirable, qtest modules are currently very open-ended in how errors are handled. Also, currently a failed/aborted test will still result in the qemu process exiting successfully, making it different to leverage in automated fashion for large sets of tests or `make test` functionality.
It should be noted you will see two QEMU's start up as there is a probe stage before the individual tests are run.


== Code/Submissions ==
==== rr ====


Current devel tree:
rr takes some special care due to [https://github.com/rr-debugger/rr/issues/3139#issuecomment-1090775126 an issue with SIGTERM]. The work around is to run the whole sequence under rr and then use "rr ps" and "rr dump" to attach to the correct point in the stream.


* git://repo.or.cz/qemu/mdroth.git qtest-dev
== Implementation ==


RFC v1, http://www.mail-archive.com/qemu-devel@nongnu.org/msg54191.html :
Please see [http://git.qemu-project.org/?p=qemu.git;a=blob;f=tests/qtest/libqos/libqtest.h;hb=HEAD tests/qtest/libqos/libqtest.h] for device test APIs.  Search the tests/ directory for examples using libqtest.


* git://repo.or.cz/qemu/mdroth.git qtest_v1
[[Category:Completed feature pages]]

Latest revision as of 09:21, 5 January 2024

Summary

QTest is an internal framework used in the QEMU unit tests. A QTest based test will spin up one or more QEMU binaries and orchestrate the test via a qtest control socket. The binaries themselves are usually controlled using a QMP socket to trigger events.

Debugging

Isolating the failing test

Usually the first problem a new developer comes across is understanding how a test fails. The qtest tests are all grouped together by target architectures so the first step is to re-run make check for the particular architecture that failed, with the V=1 option to make it more verbose: 

   make check-qtest-i386 V=1

You can also run the tests directly with meson: 

   meson test --list | grep qos
   meson test qtest-aarch64/qos-test

You can then examine `meson-logs/testlog.tx` to get the invocation details.

The test run will now dump a lot more information about how the test is run. This will be a gtester call with a large number of tests included. This can then be repeated with just the tests you want. For example: 

   env MALLOC_PERTURB_=16 QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon QTEST_QEMU_IMG=./qemu-img G_TEST_DBUS_DAEMON=/home/alex/lsrc/qemu.git/tests/dbus-vmstate-daemon.sh PYTHON=/home/alex/lsrc/qemu.git/builds/all/pyvenv/bin/python3 QTEST_QEMU_BINARY=./qemu-system-aarch64 /home/alex/lsrc/qemu.git/builds/all/tests/qtest/qos-test --tap -k

For additional information you can set QTEST_LOG and also use -l to list all the tests and -p to specify the subtest you want to run: 

   env QTEST_LOG=1 MALLOC_PERTURB_=16 QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon QTEST_QEMU_IMG=./qemu-img G_TEST_DBUS_DAEMON=/home/alex/lsrc/qemu.git/tests/dbus-vmstate-daemon.sh PYTHON=/home/alex/lsrc/qemu.git/builds/all/pyvenv/bin/python3 QTEST_QEMU_BINARY=./qemu-system-aarch64 /home/alex/lsrc/qemu.git/builds/all/tests/qtest/qos-test --tap -p /aarch64/virt/generic-pcihost/pci-bus-generic/pci-bus/vhost-user-gpio-pci/vhost-user-gpio/vhost-user-gpio-tests/read-guest-mem/memfile

Using debugging tools under the test harness

Since the QEMU binary is run as a separate process, running it under the control of debugging tools like gdb or valgrind isn't straightforward; you have to tell the test harness to launch the debug tool for you.

For most cases, the simplest thing is to set QTEST_STOP=1. This will cause the qtest harness to send a SIGSTOP to the QEMU process as soon as it has started it and before it tries to run the test. You can then find the stopped QEMU process's PID and connect to it with gdb. Once you've attached gdb to it you can resume the QEMU process with the gdb 'continue' command.

A different approach is to use a command like this to get the test harness to start a new xterm with a gdb in it for each test it runs: 

   QTEST_QEMU_BINARY="xterm -e gdb --tty $(tty) --args ./qemu-system-aarch64" QTEST_QEMU_IMG=qemu-img gtester -k --verbose -m=quick  tests/device-introspect-test

or in terminal mode (with tmux multiplexer) something like: 

   QTEST_QEMU_BINARY="tmux split-window gdb --args ./qemu-system-aarch64" QTEST_QEMU_IMG=qemu-img gtester -k --verbose -m=quick  tests/device-introspect-test

The gdb will get control before the QEMU process has done anything, so you can set breakpoints and so on before using 'continue' to make it start execution.

You can use a similar trick to run the QEMU process under valgrind: 

   QTEST_QEMU_BINARY="valgrind --vgdb-error=1 --log-file=vg.log qemu-system-aarch64" QTEST_QEMU_IMG=qemu-img MALLOC_PERTURB_=${MALLOC_PERTURB_:-$((RANDOM % 255 + 1))} gtester -k --verbose -m=quick  tests/device-introspect-test

It should be noted you will see two QEMU's start up as there is a probe stage before the individual tests are run.

rr

rr takes some special care due to an issue with SIGTERM. The work around is to run the whole sequence under rr and then use "rr ps" and "rr dump" to attach to the correct point in the stream.

Implementation

Please see tests/qtest/libqos/libqtest.h for device test APIs. Search the tests/ directory for examples using libqtest.

Retrieved from "https://wiki.qemu.org/index.php?title=Features/QTest&oldid=11570"