Testing/LTP: Difference between revisions

From QEMU
 
(12 intermediate revisions by 3 users not shown)
Line 1: Line 1:
== Testing Linux usermode emulation with the Linux Test Project ==
== Testing Linux usermode emulation with the Linux Test Project ==


You can use the [http://ltp.sourceforge.net/ Linux Test Project]'s syscall tests to test QEMU's linux-user mode support. These instructions are for how to do that using a Debian chroot running under an x86 Ubuntu or Debian host (I tested with Ubuntu Precise).
You can use the [http://linux-test-project.github.io/ Linux Test Project]'s syscall tests to test QEMU's linux-user mode support. These instructions are for how to do that using a Debian chroot running under an x86 Ubuntu or Debian host (I tested with Ubuntu Precise).


These instructions set up an armhf chroot; they should in theory work for any architecture supported by both Debian and QEMU.
These instructions set up an armhf chroot; they should in theory work for any architecture supported by both Debian and QEMU.
Line 10: Line 10:
The finished chroot including the compiled testsuite needs about 620MB of disk space. [This could almost certainly be trimmed down but disk is cheap and time is expensive :-)]
The finished chroot including the compiled testsuite needs about 620MB of disk space. [This could almost certainly be trimmed down but disk is cheap and time is expensive :-)]


Download the LTP source tarball; I used the "January 2013 Stable" release:
Download the LTP source tarball; I used the tar.gz of tag 20150903 from https://github.com/linux-test-project/ltp/tags but really you should use a newer version, at least 20160920.
https://sourceforge.net/projects/ltp/files/LTP%20Source/ltp-20130109/ltp-full-20130109.bz2/download


As root in the host system:
As root in the host system:
   mkdir /srv/chroot/ltp-armhf
   mkdir /srv/chroot/ltp-armhf
   qemu-debootstrap --arch armhf wheezy /srv/chroot/ltp-armhf
   qemu-debootstrap --arch armhf --variant=buildd jessie /srv/chroot/ltp-armhf-20150903
   echo "deb http://ftp.us.debian.org/debian wheezy main" > /srv/chroot/ltp-armhf/etc/apt/sources.list
   echo "deb http://ftp.uk.debian.org/debian jessie main" > /srv/chroot/ltp-armhf-20150903/etc/apt/sources.list
   cd /srv/chroot/ltp-armhf
   cd /srv/chroot/ltp-armhf-20150903
   tar xvjf ltp-full-20130109.bz2
   tar xvjf ltp-full-20150903.bz2
   chroot /srv/chroot/ltp-armhf
   chroot /srv/chroot/ltp-armhf-20150903
 
Note: for older architectures which aren't supported by Debian any more and whose packages have moved into the archives you can use a command line like this:
  qemu-debootstrap --arch alpha --variant=buildd lenny /srv/chroot/alpha http://archive.debian.org/debian


Then in the chroot (the compile stage will take an hour or two):
Then in the chroot (the compile stage will take an hour or two):
  [ -e /proc/cpuinfo ] || mount proc /proc -t proc
  touch /etc/hosts
  cd ltp-full-20150903
   apt-get update
   apt-get update
   apt-get install build-essential
   apt-get install automake autoconf iproute2 net-tools sudo
  cd ltp-full-20130109
   make autotools && ./configure && make && make install
   ./configure && make && make install


This will install the tests in <code>/opt/ltp/</code> inside the chroot.
This will install the tests in <code>/opt/ltp/</code> inside the chroot.
(We touch /etc/hosts because the LTP getdtablesize test assumes that file exists. We install iproute2 and net-tools for the benefit of the sendmsg01 test. We install sudo for the utimensat tests.)


Create an <code>/opt/ltp/qemu.skiplist</code> file inside the chroot with the following contents:
Create an <code>/opt/ltp/qemu.skiplist</code> file inside the chroot with the following contents:
Line 34: Line 40:
# This is a list of tests which hang completely under QEMU
# This is a list of tests which hang completely under QEMU
# or are otherwise badly behaved (as opposed to merely failing).
# or are otherwise badly behaved (as opposed to merely failing).
# We should probably investigate them more closely at some point.
#
# Updated skiplist as of 2016-10-20
#  
#  
# Skip all the clone tests, QEMU threading support is known to be broken
# we don't implement clone flags correctly, so in clone2
# and one of the clone tests seems to cause the LTP test harness
# the child gets the wrong return value for getppid() and kills
# to bail out entirely.
# the test harness by accident
clone01
clone02
clone02
clone03
 
clone04
clone05
clone06
clone07
# Seems to hang
# Seems to hang
fork13
fork13
# These tests get in a total mess with signals
futex_wait03
kill10
 
kill11
# This runs OK but thrashes the machine with lots of processes
# This runs OK but thrashes the machine with lots of processes
msgctl11
msgctl11
# These three seem to hang
# these tests try to restart syslogd, which is a bad idea in a chroot
msgrcv03
nanosleep04
splice02
# these tests try to restart syslogd!?!
syslog01
syslog01
syslog02
syslog02
Line 70: Line 67:
syslog11
syslog11
syslog12
syslog12
# hangs
# This doesn't hang but it seems to get very confused and I think
waitpid02
# it ends up not unmounting a loopback device, which then makes a
# lot of later tests bail out (and the whole test framework complains
# that it can't remove its temp dir when it cleans up).
mmap16
</pre>
</pre>
WARNING: this skiplist might not be entirely sufficient. Best to keep an eye on what's running; if a qemu process seems to have got stuck running one test for a long time you can just kill it (and add the missing entry to the skiplist).


=== Running tests ===
=== Running tests ===
Line 79: Line 81:
     ./configure --target-list=arm-linux-user --static && make -j2
     ./configure --target-list=arm-linux-user --static && make -j2


Then copy the <code>arm-linux-user/qemu-arm</code> binary into <code>/srv/chroot/ltp-armhf/usr/bin/qemu-arm-static</code> (you'll need to make sure you don't still have a shell open in the chroot or the copy will fail).
Then copy the <code>arm-linux-user/qemu-arm</code> binary into <code>/srv/chroot/ltp-armhf-20150903/usr/bin/qemu-arm-static</code> (you'll need to make sure you don't still have a shell open in the chroot or the copy will fail).


To run tests in the chroot:
To run tests in the chroot:
Line 94: Line 96:
You can investigate a failure by running a single test like this:
You can investigate a failure by running a single test like this:
     ./runltp -f /opt/ltp/runtest/syscalls -s accept4
     ./runltp -f /opt/ltp/runtest/syscalls -s accept4
(the <code>-s</code> option takes a pattern specifying tests to run).
(the <code>-s</code> option takes a regex specifying tests to run).


=== Current status ===
=== Current status ===


As of v1.4.0-rc1 the ARM QEMU ran 959 tests with 86 failures. If anybody would like to set up an automated system to run these tests nightly and produce pretty web pages of regressions/progressions that would be cool :-)
As of v1.4.0-rc1 the ARM QEMU ran 959 tests with 86 failures. If anybody would like to set up an automated system to run these tests nightly and produce pretty web pages of regressions/progressions that would be cool :-)

Latest revision as of 18:07, 20 October 2016

Testing Linux usermode emulation with the Linux Test Project

You can use the Linux Test Project's syscall tests to test QEMU's linux-user mode support. These instructions are for how to do that using a Debian chroot running under an x86 Ubuntu or Debian host (I tested with Ubuntu Precise).

These instructions set up an armhf chroot; they should in theory work for any architecture supported by both Debian and QEMU.

Setting up the testsuite environment

First we need to set up the chroot. This takes a while but once you've done it you can reuse the chroot for multiple test runs. We build the LTP testsuite inside the chroot, which is slow but simpler than setting up a cross-build environment. The finished chroot including the compiled testsuite needs about 620MB of disk space. [This could almost certainly be trimmed down but disk is cheap and time is expensive :-)]

Download the LTP source tarball; I used the tar.gz of tag 20150903 from https://github.com/linux-test-project/ltp/tags but really you should use a newer version, at least 20160920.

As root in the host system:

  mkdir /srv/chroot/ltp-armhf
  qemu-debootstrap --arch armhf --variant=buildd jessie /srv/chroot/ltp-armhf-20150903
  echo "deb http://ftp.uk.debian.org/debian jessie main" > /srv/chroot/ltp-armhf-20150903/etc/apt/sources.list
  cd /srv/chroot/ltp-armhf-20150903
  tar xvjf ltp-full-20150903.bz2
  chroot /srv/chroot/ltp-armhf-20150903

Note: for older architectures which aren't supported by Debian any more and whose packages have moved into the archives you can use a command line like this:

  qemu-debootstrap --arch alpha --variant=buildd lenny /srv/chroot/alpha http://archive.debian.org/debian

Then in the chroot (the compile stage will take an hour or two):

  [ -e /proc/cpuinfo ] || mount proc /proc -t proc
  touch /etc/hosts
  cd ltp-full-20150903
  apt-get update
  apt-get install automake autoconf iproute2 net-tools sudo
  make autotools && ./configure && make && make install

This will install the tests in /opt/ltp/ inside the chroot.

(We touch /etc/hosts because the LTP getdtablesize test assumes that file exists. We install iproute2 and net-tools for the benefit of the sendmsg01 test. We install sudo for the utimensat tests.)

Create an /opt/ltp/qemu.skiplist file inside the chroot with the following contents:

# skiplist for QEMU testing
# This is a list of tests which hang completely under QEMU
# or are otherwise badly behaved (as opposed to merely failing).
#
# Updated skiplist as of 2016-10-20
# 
# we don't implement clone flags correctly, so in clone2
# the child gets the wrong return value for getppid() and kills
# the test harness by accident
clone02

# Seems to hang
fork13
futex_wait03

# This runs OK but thrashes the machine with lots of processes
msgctl11
# these tests try to restart syslogd, which is a bad idea in a chroot
syslog01
syslog02
syslog03
syslog04
syslog05
syslog06
syslog07
syslog08
syslog09
syslog10
syslog11
syslog12
# This doesn't hang but it seems to get very confused and I think
# it ends up not unmounting a loopback device, which then makes a
# lot of later tests bail out (and the whole test framework complains
# that it can't remove its temp dir when it cleans up).
mmap16

WARNING: this skiplist might not be entirely sufficient. Best to keep an eye on what's running; if a qemu process seems to have got stuck running one test for a long time you can just kill it (and add the missing entry to the skiplist).

Running tests

OK, now we're ready to actually do a test run! You'll need to build the QEMU to test as a static executable, for example:

   ./configure --target-list=arm-linux-user --static && make -j2

Then copy the arm-linux-user/qemu-arm binary into /srv/chroot/ltp-armhf-20150903/usr/bin/qemu-arm-static (you'll need to make sure you don't still have a shell open in the chroot or the copy will fail).

To run tests in the chroot:

   [ -e /proc/cpuinfo ] || mount proc /proc -t proc
   cd /opt/ltp
   ./runltp -p -l "qemu-$(date +%FT%T).log" -o "qemu-$(date +%FT%T).out" -f /opt/ltp/runtest/syscalls -S /opt/ltp/qemu.skiplist 

This will take an hour or so, and writes a human readable results summary to a file in /opt/ltp/results/, and the complete test output dump to a file in /opt/ltp/output/. You can track its progress by tailing the output file that is created in /opt/ltp/output/.

(The LTP test runner appends results to existing log files rather than overwriting them, which is why we make sure to include a date/timestamp in the filenames.)

Note that the test suite often considers "syscall unimplemented" as a PASS condition. To find out whether QEMU is just missing syscalls completely you'll need to look for "qemu: Unsupported syscall:" lines in the output file.

You can investigate a failure by running a single test like this:

   ./runltp -f /opt/ltp/runtest/syscalls -s accept4

(the -s option takes a regex specifying tests to run).

Current status

As of v1.4.0-rc1 the ARM QEMU ran 959 tests with 86 failures. If anybody would like to set up an automated system to run these tests nightly and produce pretty web pages of regressions/progressions that would be cool :-)