Requirements/GatingCI

From QEMU
Revision as of 17:05, 8 November 2019 by Pm215 (talk | contribs)

Problem Statement

A gating CI is a prerequisite to having a multi-maintainer model of merging. By having a common set of tests that are run prior to a merge you do not rely on who is currently doing merging duties having access to the current set of test machines.

Currently pre-merge testing is done via a set of tests done by ad-hoc shell scripts run on a set of machines using personal accounts of the overall maintainer. We want to replace this ad-hoc system with one which:

  • does not use any machines which aren't usable with generic project role accounts
  • uses a known and maintainable CI system (eg Gitlab) rather than hand-hacked scripts
  • can be handed over to another person to handle releases

Current Tests

Peter currently has access to a random menagerie of build and test machines which are accessed by his scripts.

The scripts are kept in:

https://git.linaro.org/people/peter.maydell/misc-scripts.git/tree

though they are best treated as a reference for what we currently do rather than used as a base for anything.

The set of machine I currently test on are:

  • an S390x box (this is provided to the project by IBM's Community Cloud so can be used for the new CI setup)
  • aarch32 (as a chroot on an aarch64 system)
  • aarch64
  • ppc64 (on the GCC compile farm)
  • OSX
  • Windows crossbuilds
  • NetBSD, FreeBSD and OpenBSD using the tests/vm VMs
  • x86-64 Linux with a variety of different build configs (see the 'remake-merge-builds' script for how these are set up)

I also have access to a SPARC box but am not currently testing with it as there are hangs which I did not have time to investigate.

Testing process:

  • I get an email which is a pull request, and I run the "apply-pullreq" script, which takes the GIT URL and tag/branch name to test.
  • apply-pullreq performs the merge into a 'staging' branch
  • apply-pullreq also performs some simple local tests:
    • does git verify-tag like the GPG signature?
    • are we trying to apply the pull before reopening the dev tree for a new release?
    • does the pull include commits with bad UTF8 or bogus qemu-devel email addresses?
    • submodule updates are only allowed if the --submodule-ok option was specifically passed
  • apply-pullreq then invokes parallel-buildtest to do the actual testing
  • parallel-buildtest is a trivial wrapper around GNU Parallel which invokes 'mergebuild' on each of the test machines
  • if all is OK then the user gets to do the 'git push' to push the staging branch to master

In almost all cases 'mergebuild' is simply "run 'make -C build' and then 'make -C build check'". The exceptions are:

  • the Windows crossbuilds don't try to run 'make check'
  • the x86-64 host runs the 'pull-buildtest' script, which:
    • does make/make check for multiple configs
    • includes one build from 'make clean' (almost everything else does an incremental build)
    • runs 'make check-tcg' on the all-linux-static config
    • runs a trivial set of 'ls' binaries for a bunch of linux-user guests (this is probably mostly redundant now we have check-tcg)

The parallel-buildtest script causes GNU parallel to print a series of lines with the logfiles containing the captured stdout/stderr from each machine. I run the output of those through the 'greplogs' script which looks for things that look like error messages. This is intended to capture warnings/errors which didn't manage to cause the make process to return failure for one reason or another.