Testing/LTP: Difference between revisions
(Created page with '== Testing Linux usermode emulation with the Linux Test Project == You can use the [http://ltp.sourceforge.net/ Linux Test Project]'s syscall tests to test QEMU's linux-user mod…') |
|||
| Line 91: | Line 91: | ||
Note that the test suite often considers "syscall unimplemented" as a PASS condition. To find out whether QEMU is just missing syscalls completely you'll need to look for "qemu: Unsupported syscall:" lines in the output file. | Note that the test suite often considers "syscall unimplemented" as a PASS condition. To find out whether QEMU is just missing syscalls completely you'll need to look for "qemu: Unsupported syscall:" lines in the output file. | ||
You can investigate a failure by running a single test like this: | |||
./runltp -f /opt/ltp/runtest/syscalls -s accept4 | |||
(the <code>-s</code> option takes a pattern specifying tests to run). | |||
=== Current status === | === Current status === | ||
As of v1.4.0-rc1 the ARM QEMU ran 959 tests with 86 failures. If anybody would like to set up an automated system to run these tests nightly and produce pretty web pages of regressions/progressions that would be cool :-) | As of v1.4.0-rc1 the ARM QEMU ran 959 tests with 86 failures. If anybody would like to set up an automated system to run these tests nightly and produce pretty web pages of regressions/progressions that would be cool :-) | ||
Revision as of 13:56, 8 February 2013
Testing Linux usermode emulation with the Linux Test Project
You can use the Linux Test Project's syscall tests to test QEMU's linux-user mode support. These instructions are for how to do that using a Debian chroot running under an x86 Ubuntu or Debian host (I tested with Ubuntu Precise).
These instructions set up an armhf chroot; they should in theory work for any architecture supported by both Debian and QEMU.
Setting up the testsuite environment
First we need to set up the chroot. This takes a while but once you've done it you can reuse the chroot for multiple test runs. We build the LTP testsuite inside the chroot, which is slow but simpler than setting up a cross-build environment. The finished chroot including the compiled testsuite needs about 620MB of disk space. [This could almost certainly be trimmed down but disk is cheap and time is expensive :-)]
Download the LTP source tarball; I used the "January 2013 Stable" release: https://sourceforge.net/projects/ltp/files/LTP%20Source/ltp-20130109/ltp-full-20130109.bz2/download
As root in the host system:
mkdir /srv/chroot/ltp-armhf qemu-debootstrap --arch armhf wheezy /srv/chroot/ltp-armhf echo "deb http://ftp.us.debian.org/debian wheezy main" > /srv/chroot/ltp-armhf/etc/apt/sources.list cd /srv/chroot/ltp-armhf tar xvjf ltp-full-20130109.bz2 chroot /srv/chroot/ltp-armhf
Then in the chroot (the compile stage will take an hour or two):
apt-get update apt-get install build-essential cd ltp-full-20130109 ./configure && make && make install
This will install the tests in /opt/ltp/ inside the chroot.
Create an /opt/ltp/qemu.skiplist file inside the chroot with the following contents:
# skiplist for QEMU testing # This is a list of tests which hang completely under QEMU # or are otherwise badly behaved (as opposed to merely failing). # We should probably investigate them more closely at some point. # # Skip all the clone tests, QEMU threading support is known to be broken # and one of the clone tests seems to cause the LTP test harness # to bail out entirely. clone01 clone02 clone03 clone04 clone05 clone06 clone07 # Seems to hang fork13 # These tests get in a total mess with signals kill10 kill11 # This runs OK but thrashes the machine with lots of processes msgctl11 # These three seem to hang msgrcv03 nanosleep04 splice02 # these tests try to restart syslogd!?! syslog01 syslog02 syslog03 syslog04 syslog05 syslog06 syslog07 syslog08 syslog09 syslog10 syslog11 syslog12 # hangs waitpid02
Running tests
OK, now we're ready to actually do a test run! You'll need to build the QEMU to test as a static executable, for example:
./configure --target-list=arm-linux-user --static && make -j2
Then copy the arm-linux-user/qemu-arm binary into /srv/chroot/ltp-armhf/usr/bin/qemu-arm-static (you'll need to make sure you don't still have a shell open in the chroot or the copy will fail).
To run tests in the chroot:
[ -e /proc/cpuinfo ] || mount proc /proc -t proc cd /opt/ltp ./runltp -p -l "qemu-$(date +%FT%T).log" -o "qemu-$(date +%FT%T).out" -f /opt/ltp/runtest/syscalls -S /opt/ltp/qemu.skiplist
This will take an hour or so, and writes a human readable results summary to a file in /opt/ltp/results/, and the complete test output dump to a file in /opt/ltp/output/. You can track its progress by tailing the output file that is created in /opt/ltp/output/.
(The LTP test runner appends results to existing log files rather than overwriting them, which is why we make sure to include a date/timestamp in the filenames.)
Note that the test suite often considers "syscall unimplemented" as a PASS condition. To find out whether QEMU is just missing syscalls completely you'll need to look for "qemu: Unsupported syscall:" lines in the output file.
You can investigate a failure by running a single test like this:
./runltp -f /opt/ltp/runtest/syscalls -s accept4
(the -s option takes a pattern specifying tests to run).
Current status
As of v1.4.0-rc1 the ARM QEMU ran 959 tests with 86 failures. If anybody would like to set up an automated system to run these tests nightly and produce pretty web pages of regressions/progressions that would be cool :-)