Features/QTest: Difference between revisions

From QEMU
(Update the examples, mention the second QEMU and expand the section on rr)
Line 13: Line 13:
The test run will now dump a lot more information about how the test is run. This will be a gtester call with a large number of tests included. This can then be repeated with just the tests you want. For example:
The test run will now dump a lot more information about how the test is run. This will be a gtester call with a large number of tests included. This can then be repeated with just the tests you want. For example:


     QTEST_QEMU_BINARY=i386-softmmu/qemu-system-i386 QTEST_QEMU_IMG=qemu-img MALLOC_PERTURB_=${MALLOC_PERTURB_:-$((RANDOM % 255 + 1))} gtester -k --verbose -m=quick tests tests/vhost-user-test
     QTEST_QEMU_BINARY=./qemu-system-i386 QTEST_QEMU_IMG=qemu-img MALLOC_PERTURB_=${MALLOC_PERTURB_:-$((RANDOM % 255 + 1))} gtester -k --verbose -m=quick tests tests/vhost-user-test


For additional information you can set QTEST_LOG and also use qtester -p to specify the subtest you want to run:
For additional information you can set QTEST_LOG and also use qtester -p to specify the subtest you want to run:


     QTEST_LOG=1 QTEST_QEMU_BINARY=i386-softmmu/qemu-system-i386 QTEST_QEMU_IMG=qemu-img MALLOC_PERTURB_=${MALLOC_PERTURB_:-$((RANDOM % 255 + 1))} gtester -k --verbose -m=quick tests/vhost-user-test -p /i386/vhost-user/migrate
     QTEST_LOG=1 QTEST_QEMU_BINARY=./qemu-system-i386 QTEST_QEMU_IMG=qemu-img MALLOC_PERTURB_=${MALLOC_PERTURB_:-$((RANDOM % 255 + 1))} gtester -k --verbose -m=quick tests/vhost-user-test -p /i386/vhost-user/migrate


=== Using debugging tools under the test harness ===
=== Using debugging tools under the test harness ===
Line 27: Line 27:
A different approach is to use a command like this to get the test harness to start a new xterm with a gdb in it for each test it runs:
A different approach is to use a command like this to get the test harness to start a new xterm with a gdb in it for each test it runs:


     QTEST_QEMU_BINARY="xterm -e gdb --tty $(tty) --args aarch64-softmmu/qemu-system-aarch64" QTEST_QEMU_IMG=qemu-img gtester -k --verbose -m=quick  tests/device-introspect-test
     QTEST_QEMU_BINARY="xterm -e gdb --tty $(tty) --args ./qemu-system-aarch64" QTEST_QEMU_IMG=qemu-img gtester -k --verbose -m=quick  tests/device-introspect-test


or in terminal mode (with tmux multiplexer) something like:
or in terminal mode (with tmux multiplexer) something like:


     QTEST_QEMU_BINARY="tmux split-window gdb --args aarch64-softmmu/qemu-system-aarch64" QTEST_QEMU_IMG=qemu-img gtester -k --verbose -m=quick  tests/device-introspect-test
     QTEST_QEMU_BINARY="tmux split-window gdb --args ./qemu-system-aarch64" QTEST_QEMU_IMG=qemu-img gtester -k --verbose -m=quick  tests/device-introspect-test


The gdb will get control before the QEMU process has done anything, so you can set breakpoints and so on before using 'continue' to make it start execution.
The gdb will get control before the QEMU process has done anything, so you can set breakpoints and so on before using 'continue' to make it start execution.
Line 38: Line 38:


     QTEST_QEMU_BINARY="valgrind --vgdb-error=1 --log-file=vg.log qemu-system-aarch64" QTEST_QEMU_IMG=qemu-img MALLOC_PERTURB_=${MALLOC_PERTURB_:-$((RANDOM % 255 + 1))} gtester -k --verbose -m=quick  tests/device-introspect-test
     QTEST_QEMU_BINARY="valgrind --vgdb-error=1 --log-file=vg.log qemu-system-aarch64" QTEST_QEMU_IMG=qemu-img MALLOC_PERTURB_=${MALLOC_PERTURB_:-$((RANDOM % 255 + 1))} gtester -k --verbose -m=quick  tests/device-introspect-test
It should be noted you will see two QEMU's start up as there is a probe stage before the individual tests are run.


==== rr ====
==== rr ====


rr takes some special care due to [https://github.com/rr-debugger/rr/issues/3139#issuecomment-1090775126]
rr takes some special care due to [https://github.com/rr-debugger/rr/issues/3139#issuecomment-1090775126 an issue with SIGTERM]. The work around is to run the whole sequence under rr and then use "rr ps" and "rr dump" to attach to the correct point in the stream.


== Implementation ==
== Implementation ==

Revision as of 09:40, 7 April 2022

Summary

QTest is an internal framework used in the QEMU unit tests. A QTest based test will spin up one or more QEMU binaries and orchestrate the test via a qtest control socket. The binaries themselves are usually controlled using a QMP socket to trigger events.

Debugging

Isolating the failing test

Usually the first problem a new developer comes across is understanding how a test fails. The qtest tests are all grouped together by target architectures so the first step is to re-run make check for the particular architecture that failed, with the V=1 option to make it more verbose:

   make check-qtest-i386 V=1

The test run will now dump a lot more information about how the test is run. This will be a gtester call with a large number of tests included. This can then be repeated with just the tests you want. For example:

   QTEST_QEMU_BINARY=./qemu-system-i386 QTEST_QEMU_IMG=qemu-img MALLOC_PERTURB_=${MALLOC_PERTURB_:-$((RANDOM % 255 + 1))} gtester -k --verbose -m=quick tests tests/vhost-user-test

For additional information you can set QTEST_LOG and also use qtester -p to specify the subtest you want to run:

   QTEST_LOG=1 QTEST_QEMU_BINARY=./qemu-system-i386 QTEST_QEMU_IMG=qemu-img MALLOC_PERTURB_=${MALLOC_PERTURB_:-$((RANDOM % 255 + 1))} gtester -k --verbose -m=quick tests/vhost-user-test -p /i386/vhost-user/migrate

Using debugging tools under the test harness

Since the QEMU binary is run as a separate process, running it under the control of debugging tools like gdb or valgrind isn't straightforward; you have to tell the test harness to launch the debug tool for you.

For most cases, the simplest thing is to set QTEST_STOP=1. This will cause the qtest harness to send a SIGSTOP to the QEMU process as soon as it has started it and before it tries to run the test. You can then find the stopped QEMU process's PID and connect to it with gdb. Once you've attached gdb to it you can resume the QEMU process with the gdb 'continue' command.

A different approach is to use a command like this to get the test harness to start a new xterm with a gdb in it for each test it runs:

   QTEST_QEMU_BINARY="xterm -e gdb --tty $(tty) --args ./qemu-system-aarch64" QTEST_QEMU_IMG=qemu-img gtester -k --verbose -m=quick  tests/device-introspect-test

or in terminal mode (with tmux multiplexer) something like:

   QTEST_QEMU_BINARY="tmux split-window gdb --args ./qemu-system-aarch64" QTEST_QEMU_IMG=qemu-img gtester -k --verbose -m=quick  tests/device-introspect-test

The gdb will get control before the QEMU process has done anything, so you can set breakpoints and so on before using 'continue' to make it start execution.

You can use a similar trick to run the QEMU process under valgrind:

   QTEST_QEMU_BINARY="valgrind --vgdb-error=1 --log-file=vg.log qemu-system-aarch64" QTEST_QEMU_IMG=qemu-img MALLOC_PERTURB_=${MALLOC_PERTURB_:-$((RANDOM % 255 + 1))} gtester -k --verbose -m=quick  tests/device-introspect-test

It should be noted you will see two QEMU's start up as there is a probe stage before the individual tests are run.

rr

rr takes some special care due to an issue with SIGTERM. The work around is to run the whole sequence under rr and then use "rr ps" and "rr dump" to attach to the correct point in the stream.

Implementation

Please see tests/qtest/libqos/libqtest.h for device test APIs. Search the tests/ directory for examples using libqtest.