Testing: Difference between revisions

From QEMU
(add description of testing with ccc-analyzer)
(16 intermediate revisions by 7 users not shown)
Line 1: Line 1:
== QEMU disk images ==
* [{{PagesStartingWith|Testing/}} All testing pages]


Here is a collection of disk images which can be used to test system emulation.
== Unit tests ==


{| class="wikitable" border="1"
QEMU includes a test suite comprising:
!  File
 
!  Comment
* unit tests for library code
|-
* [[Features/QTest|QTest]]-based tests, which inject predefined stimuli into the device emulation code.
[http://wiki.qemu.org/download/linux-0.2.img.bz2 linux-0.2.img.bz2] (8 MB)
* [[Testing/QemuIoTests|qemu-iotests]], a regression test suite for the block layer code.
|  Small Linux disk image containing a 2.6.20 Linux kernel, X11 and various utilities to test QEMU
 
|-
The first two are run with <tt>make check</tt>. These unit tests are used in [[Testing/ContinuousIntegration|our continuous integration]] systems, based on [[Testing/Travis|Travis]] and [[Testing/Patchew|Patchew]].
|  [http://odin.fdos.org/odin2005/odin1440.img odin1440.img]
 
|  FreeDOS floppy disk image from [http://odin.fdos.org/ ODIN] (Steve Nickolas)
qemu-iotests is run from the toplevel build directory with <tt>make check-block</tt>. A full version of the testsuite, taking around half an hour to run, is run with <tt>sh ../tests/check-block.sh</tt>.
|-
 
[http://nopid.free.fr/small.ffs.bz2 small.ffs.bz2]
== System emulation ==
|  Small NetBSD Image (thanx to Nicolas Ollinger)
 
|-
We have [[Testing/System Images|a collection of disk images]] which can be used to test system emulation.
[http://wiki.qemu.org/download/minix204.tar.bz2 minix204.tar.bz2]
|  Minix 2.0.4 (thanx to Túlio Almeida Pexoto)
|-
| [http://wiki.qemu.org/download/efi-bios.tar.bz2 efi-bios.tar.bz2]
|  EFI BIOS for QEMU (thanx to Tristan Gingold)
|-
|  [http://wiki.qemu.org/download/sparc-test-0.2.tar.gz sparc-test-0.2.tar.gz]
|  SPARC Linux 2.6 test kernel and initrd disk image
|-
[http://wiki.qemu.org/download/arm-test-0.2.tar.gz arm-test-0.2.tar.gz]
|  ARM Linux 2.6 test kernel and initrd disk image (thanx to Paul Brook)
|-
|  [http://wiki.qemu.org/download/mips-test-0.2.tar.gz mips-test-0.2.tar.gz]
|  MIPS Linux 2.6 test kernel and initrd disk image (thanx to Thiemo Seufer)
|-
|  [http://wiki.qemu.org/download/mipsel-test-0.2.tar.gz mipsel-test-0.2.tar.gz]
|  MIPS little endian Linux 2.6 test kernel and initrd disk image (thanx to Thiemo Seufer)
|-
|  [http://wiki.qemu.org/download/coldfire-test-0.1.tar.bz2 coldfire-test-0.1.tar.bz2]
|  Coldfire Linux 2.6 test kernel and initrd disk image (thanx to Paul Brook)
|-
|  [http://wiki.qemu.org/download/sh-test-0.2.tar.bz2 sh-test-0.2.tar.bz2]
|  SH4 Linux 2.6 test kernel and initrd disk image (thanx to Shin-ichiro KAWASAKI)
|-
[http://wiki.qemu.org/download/cris-axisdev88-img-linux2_6_33.tgz cris-axisdev88-img-linux2_6_33.tgz]
| CRIS AXIS Devboard88 Linux 2.6 test image with selftesting testsuite (Edgar E. Iglesias)
|-
|  [http://wiki.qemu.org/download/mb-s3adsp1800-linux-2_6_34.tgz mb-s3adsp1800-linux-2_6_34.tgz]
|  Microblaze S3ADSP1800 Linux 2.6 test image with selftesting testsuite (Edgar E. Iglesias)
|-
|  [http://wiki.qemu.org/download/ppc-virtexml507-linux-2_6_34.tgz ppc-virtexml507-linux-2_6_34.tgz]
|  PPC-440 Virtex-ML507 Linux 2.6 test image (Edgar E. Iglesias)
|-
|  [http://wiki.qemu.org/download/xtensa-dc232b_kernel_rootfs.tgz xtensa-dc232b_kernel_rootfs.tgz]
|  Xtensa Linux 2.6.29 test image (Max Filippov)
|}


== QEMU Linux user mode emulation tests ==
== User mode emulation ==


These executables can be used to test Linux user mode emulation.
These executables can be used to test Linux user mode emulation.
Line 63: Line 27:
|  [http://wiki.qemu.org/download/linux-user-test-0.3.tar.gz linux-user-test-0.3.tar.gz]
|  [http://wiki.qemu.org/download/linux-user-test-0.3.tar.gz linux-user-test-0.3.tar.gz]
|  Distribution of shared libraries and various shell executables for almost all Linux target architectures that QEMU simulates. It is used to make regression tests on the Linux user mode emulation.
|  Distribution of shared libraries and various shell executables for almost all Linux target architectures that QEMU simulates. It is used to make regression tests on the Linux user mode emulation.
|-
|  [https://kos.to/linux-user-busyboxes-0.1.tar.xz linux-user-busyboxes-0.1.tar.xz]
|  Collection of static busybox binaries for almost all Linux target architectures that QEMU simulates. For quick smoke testing of Linux user mode emulation.
|}
|}
It is also possible to [[Testing/LTP|run the Linux Test Project's syscall test suite under the Linux user mode emulation]].


== Dynamic code analysis ==
== Dynamic code analysis ==
Line 70: Line 39:
buffer overflows or other forms of illegal memory access.
buffer overflows or other forms of illegal memory access.


Typically these kind of tests are done using Valgrind on a Linux host.
Typically these kind of tests are done using [[Documentation/Debugging with Valgrind|Valgrind]] on a Linux host.
Any of the disk images and executables listed above can be used in such tests.
Any of the disk images and executables listed above can be used in such tests.


Line 92: Line 61:
       --cc=/usr/share/clang/scan-build/ccc-analyzer --disable-docs
       --cc=/usr/share/clang/scan-build/ccc-analyzer --disable-docs
  make
  make
At least on my Linux host (1 GiB RAM, 2 GiB swap), make hangs when
ccc-analyzer analyzes target-mips/translate.c: function decode_opc
is too complex for the analyzer and takes all memory. Killing the
clang process helps in this situation. It's needed 6 times because
there are 4 MIPS system emulations and 2 Linux MIPS user emulations.
I guess this is because target-mips/translate.c contains switches with
cases covering a very large range; assuming ccc-analyzer expands these
case ranges somehow, it probably blows up memory completely.


=== smatch ===
=== smatch ===
Line 105: Line 84:
This example expects that smatch and cgcc are installed in your PATH
This example expects that smatch and cgcc are installed in your PATH
(if not, you must add absolute paths to the example).
(if not, you must add absolute paths to the example).
=== Coverity ===
Periodic scans of QEMU are done on the public Coverity Scan service (scan.coverity.com).  You can request access on their website, and the administrator will grant it if you are an active participant in QEMU development.
Coverity is confused slightly by multiple definitions of functions with the same name. For this reason, Coverity scans are done as follows:
mkdir cov-int
./configure --audio-drv-list=oss,alsa,sdl,pa --disable-werror
make libqemustub.a
cov-build --dir cov-int make
tar cvf - cov-int | xz > cov-int.tar.xz
Notice that libqemustub.a is ignored by Coverity.  This is because some stubs call <tt>abort()</tt> and this causes dead-code false positives. The file cov-int.tar.xz can then be uploaded to [https://scan.coverity.com/projects/378/builds/new Coverity Scan's "Submit build" page]. Customarily, the "project version" is set to the output of <tt>git describe HEAD</tt> and the "description/tag" is set to "commit XYZ" where XYZ is the '''full''' SHA1 hash of the commit.
== Avocado and Avocado-VT ==
Avocado is a generic testing framework, while Avocado-VT adds support for Virtualization testing, including first level support for testing QEMU.
To get started with Avocado-VT please visit:
* http://avocado-vt.readthedocs.io
* https://github.com/avocado-framework/avocado-vt
To learn more about Avocado please visit:
* http://avocado-framework.readthedocs.io
* https://github.com/avocado-framework/avocado
After installing it, you can use Avocado-VT tests with your own build of QEMU:
avocado run boot --vt-qemu-bin /path/to/qemu-system-x86_64

Revision as of 08:16, 13 October 2016

Unit tests

QEMU includes a test suite comprising:

  • unit tests for library code
  • QTest-based tests, which inject predefined stimuli into the device emulation code.
  • qemu-iotests, a regression test suite for the block layer code.

The first two are run with make check. These unit tests are used in our continuous integration systems, based on Travis and Patchew.

qemu-iotests is run from the toplevel build directory with make check-block. A full version of the testsuite, taking around half an hour to run, is run with sh ../tests/check-block.sh.

System emulation

We have a collection of disk images which can be used to test system emulation.

User mode emulation

These executables can be used to test Linux user mode emulation.

File Comment
linux-user-test-0.3.tar.gz Distribution of shared libraries and various shell executables for almost all Linux target architectures that QEMU simulates. It is used to make regression tests on the Linux user mode emulation.
linux-user-busyboxes-0.1.tar.xz Collection of static busybox binaries for almost all Linux target architectures that QEMU simulates. For quick smoke testing of Linux user mode emulation.

It is also possible to run the Linux Test Project's syscall test suite under the Linux user mode emulation.

Dynamic code analysis

This includes any test to detect memory leaks, reads of uninitialised memory, buffer overflows or other forms of illegal memory access.

Typically these kind of tests are done using Valgrind on a Linux host. Any of the disk images and executables listed above can be used in such tests.

# Simple i386 boot test (BIOS only) with Valgrind.
valgrind --leak-check=full --track-origins=yes --verbose qemu-system-i386

Static code analysis

There are a number of tools which analyse C code and try to detect typical errors. None of these tools is perfect, so using different tools with QEMU will detect more bugs. Be prepared to also get lots of false warnings!

ccc-analyzer (clang)

This is an example used on Debian. It needs package clang.

# Start from the root directory with QEMU code.
mkdir -f bin/debug/ccc-analyzer
cd bin/debug/ccc-analyzer
../../../configure --enable-debug --enable-trace-backend=stderr \
     --cc=/usr/share/clang/scan-build/ccc-analyzer --disable-docs
make

At least on my Linux host (1 GiB RAM, 2 GiB swap), make hangs when ccc-analyzer analyzes target-mips/translate.c: function decode_opc is too complex for the analyzer and takes all memory. Killing the clang process helps in this situation. It's needed 6 times because there are 4 MIPS system emulations and 2 Linux MIPS user emulations.

I guess this is because target-mips/translate.c contains switches with cases covering a very large range; assuming ccc-analyzer expands these case ranges somehow, it probably blows up memory completely.

smatch

Here is a typical example using smatch (from git://repo.or.cz/smatch.git):

# Start from the root directory with QEMU code.
mkdir -f bin/debug/smatch
cd bin/debug/smatch
CHECK="smatch" ../../../configure --enable-debug --cc=cgcc --host-cc=cgcc
make

This example expects that smatch and cgcc are installed in your PATH (if not, you must add absolute paths to the example).

Coverity

Periodic scans of QEMU are done on the public Coverity Scan service (scan.coverity.com). You can request access on their website, and the administrator will grant it if you are an active participant in QEMU development.

Coverity is confused slightly by multiple definitions of functions with the same name. For this reason, Coverity scans are done as follows:

mkdir cov-int
./configure --audio-drv-list=oss,alsa,sdl,pa --disable-werror
make libqemustub.a
cov-build --dir cov-int make
tar cvf - cov-int | xz > cov-int.tar.xz

Notice that libqemustub.a is ignored by Coverity. This is because some stubs call abort() and this causes dead-code false positives. The file cov-int.tar.xz can then be uploaded to Coverity Scan's "Submit build" page. Customarily, the "project version" is set to the output of git describe HEAD and the "description/tag" is set to "commit XYZ" where XYZ is the full SHA1 hash of the commit.

Avocado and Avocado-VT

Avocado is a generic testing framework, while Avocado-VT adds support for Virtualization testing, including first level support for testing QEMU.

To get started with Avocado-VT please visit:

To learn more about Avocado please visit:


After installing it, you can use Avocado-VT tests with your own build of QEMU:

avocado run boot --vt-qemu-bin /path/to/qemu-system-x86_64