Testing: Difference between revisions

From QEMU
m (added explanations)
No edit summary
 
(54 intermediate revisions by 13 users not shown)
Line 1: Line 1:
== QEMU disk images ==
== [[Testing/CI|Continuous Integration]] ==


Here is a collection of disk images which can be used to test system emulation.
Continuous Integration for the project is a distributed affair spread
across a number of public CI services. However our primary CI loop is
run on GitLab with other services filling in gaps in it's coverage.


{| class="wikitable" border="1"
There is a strong preference for tests to be directly runable with our
!  File
existing make infrastructure. The public services to run tests all
!  Comment
offer free tier accounts for FLOSS developers so people can run their
|-
own tests with the same CI setup although generally these need public
|  [http://wiki.qemu.org/download/linux-0.2.img.bz2 linux-0.2.img.bz2] (8 MB)
repositories.
|  Small Linux disk image containing a 2.6.20 Linux kernel, X11 and various utilities to test QEMU
|-
|  [http://odin.fdos.org/odin2005/odin1440.img odin1440.img]
|  FreeDOS floppy disk image from [http://odin.fdos.org/ ODIN] (Steve Nickolas)
|-
|  [http://nopid.free.fr/small.ffs.bz2 small.ffs.bz2]
|  Small NetBSD Image (thanx to Nicolas Ollinger)
|-
|  [http://wiki.qemu.org/download/minix204.tar.bz2 minix204.tar.bz2]
|  Minix 2.0.4 (thanx to Túlio Almeida Pexoto)
|-
|  [http://wiki.qemu.org/download/efi-bios.tar.bz2 efi-bios.tar.bz2]
|  EFI BIOS for QEMU (thanx to Tristan Gingold)
|-
|  [http://wiki.qemu.org/download/sparc-test-0.2.tar.gz sparc-test-0.2.tar.gz]
|  SPARC Linux 2.6 test kernel and initrd disk image
|-
|  [http://wiki.qemu.org/download/arm-test-0.2.tar.gz arm-test-0.2.tar.gz]
|  ARM Linux 2.6 test kernel and initrd disk image (thanx to Paul Brook)
|-
|  [http://wiki.qemu.org/download/mips-test-0.2.tar.gz mips-test-0.2.tar.gz]
|  MIPS Linux 2.6 test kernel and initrd disk image (thanx to Thiemo Seufer)
|-
|  [http://wiki.qemu.org/download/mipsel-test-0.2.tar.gz mipsel-test-0.2.tar.gz]
|  MIPS little endian Linux 2.6 test kernel and initrd disk image (thanx to Thiemo Seufer)
|-
|  [http://wiki.qemu.org/download/coldfire-test-0.1.tar.bz2 coldfire-test-0.1.tar.bz2]
|  Coldfire Linux 2.6 test kernel and initrd disk image (thanx to Paul Brook)
|-
|  [http://wiki.qemu.org/download/sh-test-0.2.tar.bz2 sh-test-0.2.tar.bz2]
|  SH4 Linux 2.6 test kernel and initrd disk image (thanx to Shin-ichiro KAWASAKI)
|-
|  [http://wiki.qemu.org/download/cris-axisdev88-img-linux2_6_33.tgz cris-axisdev88-img-linux2_6_33.tgz]
|  CRIS AXIS Devboard88 Linux 2.6 test image with selftesting testsuite (Edgar E. Iglesias)
|-
|  [http://wiki.qemu.org/download/mb-s3adsp1800-linux-2_6_34.tgz mb-s3adsp1800-linux-2_6_34.tgz]
|  Microblaze S3ADSP1800 Linux 2.6 test image with selftesting testsuite (Edgar E. Iglesias)
|-
|  [http://wiki.qemu.org/download/ppc-virtexml507-linux-2_6_34.tgz ppc-virtexml507-linux-2_6_34.tgz]
|  PPC-440 Virtex-ML507 Linux 2.6 test image (Edgar E. Iglesias)
|-
|  [http://wiki.qemu.org/download/xtensa-dc232b_kernel_rootfs.tgz xtensa-dc232b_kernel_rootfs.tgz]
|  Xtensa Linux 2.6.29 test image (Max Filippov)
|}


== QEMU Linux user mode emulation tests ==
{{CIStatus}}


These executables can be used to test Linux user mode emulation.
== Tests included in the QEMU source ==


{| class="wikitable" border="1"
QEMU includes a test suite comprising:
!  File
 
!  Comment
* unit tests for library code
|-
* [[Features/QTest|QTest]]-based tests, which inject predefined stimuli into the device emulation code.
[http://wiki.qemu.org/download/linux-user-test-0.3.tar.gz linux-user-test-0.3.tar.gz]
* [[Testing/QemuIoTests|qemu-iotests]], a regression test suite for the block layer code.
|  Distribution of shared libraries and various shell executables for almost all Linux target architectures that QEMU simulates. It is used to make regression tests on the Linux user mode emulation.
 
|}
Run '''make check-help''' for a full breakdown of the various sub-checks that can be run. We also have a {{src|path=docs/devel/testing.rst|description=documentation in the source tree}}.
 
=== <tt>make check</tt> ===
 
The unit tests and QTest-based can be run with "<tt>make check</tt>". Use "<tt>make check-help</tt>" to see a list of other available test targets and parameters (for example, you can use "<tt>make check SPEED=slow V=1</tt>" for a verbose, more thorough test run). These unit tests are used in [[#Continuous Integration|our continuous integration]] systems, based on [[Testing/Travis|Travis]] and [[Testing/CI/Patchew|Patchew]].
 
Currently '''make check''' includes the following:
 
* check-qapi-schema
* check-unit
* check-qtest
* check-decodetree
* check-block
 
=== <tt>make check-tcg</tt> ===
 
This will attempt to build tests to be run under QEMU for all the enabled guest architectures. You can either supply cross compilers to configure or will attempt to fall-back to precanned docker images with them.
 
=== qemu-iotests ===
 
<blockquote>''Main article: [[Testing/QemuIoTests]]''</blockquote>
 
A subset of the qemu-iotests is run from the toplevel build directory with <tt>make check-block</tt>. A full version of the testsuite, taking around half an hour to run, is run with <tt>make check-block SPEED=thorough</tt>. For full control over the iotests, <tt>cd tests/qemu-iotests/</tt> and run <tt>./check --help</tt> for further information.
 
=== <tt>make docker</tt> ===
 
The build system supports a number of Docker build targets which allow the source tree to be built and tested on a number of different Linux distributions regardless of your host. See [[Testing/DockerBuild]] for more information.
 
=== device-crash-test script ===
 
The <tt>scripts/device-crash-test</tt> script can be used to run QEMU with multiple
<tt>-machine</tt> and <tt>-device</tt> combinations, to look for obvious crashes machine or
device code.
 
=== <tt>make check-acceptance</tt> ===
 
This make target runs the tests under <tt>tests/acceptance</tt>, which are higher level functional tests.
 
These tests are written using the Avocado Testing Framework (which will be installed automatically) in conjunction with a the <tt>avocado_qemu.Test</tt>
class, implemented at <tt>tests/acceptance/avocado_qemu</tt>.
 
== System emulation ==
 
Various machines are tested for [[Testing/Acceptance#Machines]], see the test source to see which files are downloaded/used.
 
We also have [[Testing/System Images|a collection of links to disk images]] which can be used to test system emulation.
 
== User mode emulation ==
 
Here are some links to executables that can be used to test Linux user mode emulation:
 
* [https://kos.to/linux-user-busyboxes-0.1.tar.xz linux-user-busyboxes-0.1.tar.xz] - Collection of static busybox binaries for almost all Linux target architectures that QEMU simulates. For quick smoke testing of Linux user mode emulation.
 
It is also possible to [[Testing/LTP|run the Linux Test Project's syscall test suite under the Linux user mode emulation]].
 
== Dynamic code analysis ==
 
This includes any test to detect memory leaks, reads of uninitialised memory,
buffer overflows or other forms of illegal memory access, that needs QEMU to be run, not merely compiled.
 
=== Valgrind ===
 
Typically these kind of tests are done using [[Documentation/Debugging with Valgrind|Valgrind]] on a Linux host.
Any of the disk images and executables listed above can be used in such tests.
 
# Simple i386 boot test (BIOS only) with Valgrind.
valgrind --leak-check=full --track-origins=yes --verbose qemu-system-i386
 
=== clang UBSan ===
 
The [[https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html clang undefined behavior sanitizer]] can be used to warn about accidental uses of C undefined behavior when QEMU is run. To use it you first need to configure and build QEMU with a clang compiler with the right options:
 
mkdir build/clang
(cd build/clang && ../../configure --cc=clang --cxx=clang++ \
    '--extra-cflags=-fsanitize=undefined -fno-sanitize=shift-base')
make -C build/clang -j8
 
(The -fno-sanitize=shift-base is a workaround for [[https://bugs.llvm.org/show_bug.cgi?id=25552 LLVM bug 25552]] where it did not correctly suppress some shift-related warnings when -fwrapv was in use. If you're using a clang where that bug is fixed, likely 3.9 or better, you can drop it.)
 
Then when you run the resulting QEMU binaries messages will be printed when UB is invoked:
 
hw/core/loader.c:67:15: runtime error: null pointer passed as argument 1, which is declared to never be null
 
See the clang documentation for more information including how to produce stack backtraces on errors.
 
== Static code analysis ==
 
There are a number of tools which analyse C code and try to detect typical
errors. None of these tools is perfect, so using different tools with QEMU
will detect more bugs. Be prepared to also get lots of false warnings!
 
=== ccc-analyzer (clang) ===
 
This is an example used on Debian. It needs package clang.
 
# Start from the root directory with QEMU code.
mkdir -f bin/debug/ccc-analyzer
cd bin/debug/ccc-analyzer
../../../configure --enable-debug --enable-trace-backend=stderr \
      --cc=/usr/share/clang/scan-build/ccc-analyzer --disable-docs
make
 
 
=== smatch ===
 
Here is a typical example using smatch (from git://repo.or.cz/smatch.git):
 
# Start from the root directory with QEMU code.
mkdir -f bin/debug/smatch
cd bin/debug/smatch
CHECK="smatch" ../../../configure --enable-debug --cc=cgcc --host-cc=cgcc
make
 
This example expects that smatch and cgcc are installed in your PATH
(if not, you must add absolute paths to the example).
 
=== Coverity ===
 
Periodic scans of QEMU are done on the public Coverity Scan service (scan.coverity.com).  You can request access on their website, and the administrator will grant it if you are an active participant in QEMU development.
 
Coverity is confused slightly by multiple definitions of functions with the same name. For this reason, Coverity scans are done as follows:
 
mkdir cov-int
./configure --audio-drv-list=oss,alsa,sdl,pa --disable-werror
make libqemustub.a
cov-build --dir cov-int make
tar cvf - cov-int | xz > cov-int.tar.xz
 
Notice that libqemustub.a is ignored by Coverity.  This is because some stubs call <tt>abort()</tt> and this causes dead-code false positives. The file cov-int.tar.xz can then be uploaded to [https://scan.coverity.com/projects/378/builds/new Coverity Scan's "Submit build" page]. Customarily, the "project version" is set to the output of <tt>git describe HEAD</tt> and the "description/tag" is set to "commit XYZ" where XYZ is the '''full''' SHA1 hash of the commit.
 
== Avocado and Avocado-VT ==
 
Avocado is a generic testing framework (used in the <tt>make check-acceptance</tt> tests).
 
Avocado-VT is the culmination of the old "virt-test" project (and previously known as KVM autotest) with a compatibility layer with to make it run under Avocado.  Avocado-VT adds extensive support for Virtualization testing, including first level support for testing QEMU.
 
To get started with Avocado-VT please visit:
* http://avocado-vt.readthedocs.io
* https://github.com/avocado-framework/avocado-vt
 
To learn more about Avocado please visit:
* http://avocado-framework.readthedocs.io
* https://github.com/avocado-framework/avocado
 
 
After installing it, you can use Avocado-VT tests with your own build of QEMU:
 
avocado run boot --vt-qemu-bin /path/to/qemu-system-x86_64
 
== Testing related meetings ==
 
There's a regular meeting about QEMU testing automation, Avocado's role in that,
CI efforts and related topics.
 
The meetings will take place every Tuesday from 6:00 AM to 7:00 AM,
(GMT-05:00) Eastern Time - New York.
 
The meeting ID is 2282383352, and it can be used in any of the following
ways to join the meeting:
 
* Using your browser: https://bluejeans.com/2282383352
* Using one of the apps: https://www.bluejeans.com/downloads
* Dialing to one of these numbers: https://www.bluejeans.com/numbers
 
We have been using a public Trello board to keep track of the ongoing tasks:
 
* https://trello.com/b/6Qi1pxVn/avocado-qemu
 
Meeting agenda, notes and meeting minutes are tracked at:
 
* https://public.etherpad-mozilla.org/p/AvocadoQEMU
 
== See Also ==
 
* [https://github.com/ehabkost/gdb-qemu gdb-qemu], a set of scripts that look for compatibility bugs by poking at QEMU internal data structures using GDB
 
The following sub-pages exist:
 
{{Special:PrefixIndex/Testing/}}

Latest revision as of 12:16, 12 July 2021

Continuous Integration

Continuous Integration for the project is a distributed affair spread across a number of public CI services. However our primary CI loop is run on GitLab with other services filling in gaps in it's coverage.

There is a strong preference for tests to be directly runable with our existing make infrastructure. The public services to run tests all offer free tier accounts for FLOSS developers so people can run their own tests with the same CI setup although generally these need public repositories.

System Focus Status
GitLab CI Majority of CI testing (builds x86 & cross), various check targets https://gitlab.com/qemu-project/qemu/badges/master/pipeline.svg [1]
Cirrus CI FreeBSD, MacOS and Windows MSYS2 compile and test https://api.cirrus-ci.com/github/qemu/qemu.svg [2]
Travis non-x86 hosts, mostly deprecated qemu.png?branch=master&file=qemu.png [3]
Coverity Static analysis https://scan.coverity.com/projects/378/badge.svg?flat=1&foo=qemu.svg [4]
Patchew Apply and test patches as they are sent on the mailing list. https://patchew.org/QEMU/badge.svg [5]
Documentation Build the RST portions of the doc/ subtree https://readthedocs.org/projects/qemu/badge/?version=latest&foo=qemu.svg [6]

Tests included in the QEMU source

QEMU includes a test suite comprising:

  • unit tests for library code
  • QTest-based tests, which inject predefined stimuli into the device emulation code.
  • qemu-iotests, a regression test suite for the block layer code.

Run make check-help for a full breakdown of the various sub-checks that can be run. We also have a documentation in the source tree.

make check

The unit tests and QTest-based can be run with "make check". Use "make check-help" to see a list of other available test targets and parameters (for example, you can use "make check SPEED=slow V=1" for a verbose, more thorough test run). These unit tests are used in our continuous integration systems, based on Travis and Patchew.

Currently make check includes the following:

  • check-qapi-schema
  • check-unit
  • check-qtest
  • check-decodetree
  • check-block

make check-tcg

This will attempt to build tests to be run under QEMU for all the enabled guest architectures. You can either supply cross compilers to configure or will attempt to fall-back to precanned docker images with them.

qemu-iotests

Main article: Testing/QemuIoTests

A subset of the qemu-iotests is run from the toplevel build directory with make check-block. A full version of the testsuite, taking around half an hour to run, is run with make check-block SPEED=thorough. For full control over the iotests, cd tests/qemu-iotests/ and run ./check --help for further information.

make docker

The build system supports a number of Docker build targets which allow the source tree to be built and tested on a number of different Linux distributions regardless of your host. See Testing/DockerBuild for more information.

device-crash-test script

The scripts/device-crash-test script can be used to run QEMU with multiple -machine and -device combinations, to look for obvious crashes machine or device code.

make check-acceptance

This make target runs the tests under tests/acceptance, which are higher level functional tests.

These tests are written using the Avocado Testing Framework (which will be installed automatically) in conjunction with a the avocado_qemu.Test class, implemented at tests/acceptance/avocado_qemu.

System emulation

Various machines are tested for Testing/Acceptance#Machines, see the test source to see which files are downloaded/used.

We also have a collection of links to disk images which can be used to test system emulation.

User mode emulation

Here are some links to executables that can be used to test Linux user mode emulation:

  • linux-user-busyboxes-0.1.tar.xz - Collection of static busybox binaries for almost all Linux target architectures that QEMU simulates. For quick smoke testing of Linux user mode emulation.

It is also possible to run the Linux Test Project's syscall test suite under the Linux user mode emulation.

Dynamic code analysis

This includes any test to detect memory leaks, reads of uninitialised memory, buffer overflows or other forms of illegal memory access, that needs QEMU to be run, not merely compiled.

Valgrind

Typically these kind of tests are done using Valgrind on a Linux host. Any of the disk images and executables listed above can be used in such tests.

# Simple i386 boot test (BIOS only) with Valgrind.
valgrind --leak-check=full --track-origins=yes --verbose qemu-system-i386

clang UBSan

The [clang undefined behavior sanitizer] can be used to warn about accidental uses of C undefined behavior when QEMU is run. To use it you first need to configure and build QEMU with a clang compiler with the right options:

mkdir build/clang
(cd build/clang && ../../configure --cc=clang --cxx=clang++ \
   '--extra-cflags=-fsanitize=undefined -fno-sanitize=shift-base')
make -C build/clang -j8

(The -fno-sanitize=shift-base is a workaround for [LLVM bug 25552] where it did not correctly suppress some shift-related warnings when -fwrapv was in use. If you're using a clang where that bug is fixed, likely 3.9 or better, you can drop it.)

Then when you run the resulting QEMU binaries messages will be printed when UB is invoked:

hw/core/loader.c:67:15: runtime error: null pointer passed as argument 1, which is declared to never be null

See the clang documentation for more information including how to produce stack backtraces on errors.

Static code analysis

There are a number of tools which analyse C code and try to detect typical errors. None of these tools is perfect, so using different tools with QEMU will detect more bugs. Be prepared to also get lots of false warnings!

ccc-analyzer (clang)

This is an example used on Debian. It needs package clang.

# Start from the root directory with QEMU code.
mkdir -f bin/debug/ccc-analyzer
cd bin/debug/ccc-analyzer
../../../configure --enable-debug --enable-trace-backend=stderr \
     --cc=/usr/share/clang/scan-build/ccc-analyzer --disable-docs
make


smatch

Here is a typical example using smatch (from git://repo.or.cz/smatch.git):

# Start from the root directory with QEMU code.
mkdir -f bin/debug/smatch
cd bin/debug/smatch
CHECK="smatch" ../../../configure --enable-debug --cc=cgcc --host-cc=cgcc
make

This example expects that smatch and cgcc are installed in your PATH (if not, you must add absolute paths to the example).

Coverity

Periodic scans of QEMU are done on the public Coverity Scan service (scan.coverity.com). You can request access on their website, and the administrator will grant it if you are an active participant in QEMU development.

Coverity is confused slightly by multiple definitions of functions with the same name. For this reason, Coverity scans are done as follows:

mkdir cov-int
./configure --audio-drv-list=oss,alsa,sdl,pa --disable-werror
make libqemustub.a
cov-build --dir cov-int make
tar cvf - cov-int | xz > cov-int.tar.xz

Notice that libqemustub.a is ignored by Coverity. This is because some stubs call abort() and this causes dead-code false positives. The file cov-int.tar.xz can then be uploaded to Coverity Scan's "Submit build" page. Customarily, the "project version" is set to the output of git describe HEAD and the "description/tag" is set to "commit XYZ" where XYZ is the full SHA1 hash of the commit.

Avocado and Avocado-VT

Avocado is a generic testing framework (used in the make check-acceptance tests).

Avocado-VT is the culmination of the old "virt-test" project (and previously known as KVM autotest) with a compatibility layer with to make it run under Avocado. Avocado-VT adds extensive support for Virtualization testing, including first level support for testing QEMU.

To get started with Avocado-VT please visit:

To learn more about Avocado please visit:


After installing it, you can use Avocado-VT tests with your own build of QEMU:

avocado run boot --vt-qemu-bin /path/to/qemu-system-x86_64

Testing related meetings

There's a regular meeting about QEMU testing automation, Avocado's role in that, CI efforts and related topics.

The meetings will take place every Tuesday from 6:00 AM to 7:00 AM, (GMT-05:00) Eastern Time - New York.

The meeting ID is 2282383352, and it can be used in any of the following ways to join the meeting:

We have been using a public Trello board to keep track of the ongoing tasks:

Meeting agenda, notes and meeting minutes are tracked at:

See Also

  • gdb-qemu, a set of scripts that look for compatibility bugs by poking at QEMU internal data structures using GDB

The following sub-pages exist: