Requirements/GatingCI

Problem Statement

A gating CI is a prerequisite to having a multi-maintainer model of merging. By having a common set of tests that are run prior to a merge you do not rely on who is currently doing merging duties having access to the current set of test machines.

Currently pre-merge testing is done via a set of tests done by ad-hoc shell scripts run on a set of machines using personal accounts of the overall maintainer. We want to replace this ad-hoc system with one which:

• does not use any machines which aren't usable with generic project role accounts
• uses a known and maintainable CI system (eg Gitlab) rather than hand-hacked scripts
• can be handed over to another person to handle releases

Current Tests

Peter currently has access to a random menagerie of build and test machines which are accessed by his scripts.

The scripts are kept in:

https://git.linaro.org/people/peter.maydell/misc-scripts.git/tree

though they are best treated as a reference for what we currently do rather than used as a base for anything.

The set of machine I currently test on are:

• an S390x box (this is provided to the project by IBM's Community Cloud so can be used for the new CI setup)
• aarch32 (as a chroot on an aarch64 system)
• aarch64
• ppc64 (on the GCC compile farm)
• OSX
• Windows crossbuilds
• NetBSD, FreeBSD and OpenBSD using the tests/vm VMs
• x86-64 Linux with a variety of different build configs (see the 'remake-merge-builds' script for how these are set up)

I also have access to a SPARC box but am not currently testing with it as there are hangs which I did not have time to investigate.

Testing process:

• I get an email which is a pull request, and I run the "apply-pullreq" script, which takes the GIT URL and tag/branch name to test.
• apply-pullreq performs the merge into a 'staging' branch
• apply-pullreq also performs some simple local tests:
  • does git verify-tag like the GPG signature?
  • are we trying to apply the pull before reopening the dev tree for a new release?
  • does the pull include commits with bad UTF8 or bogus qemu-devel email addresses?
  • submodule updates are only allowed if the --submodule-ok option was specifically passed
• apply-pullreq then invokes parallel-buildtest to do the actual testing
• parallel-buildtest is a trivial wrapper around GNU Parallel which invokes 'mergebuild' on each of the test machines
• if all is OK then the user gets to do the 'git push' to push the staging branch to master

In almost all cases 'mergebuild' is simply "run 'make -C build' and then 'make -C build check'". The exceptions are:

• the Windows crossbuilds don't try to run 'make check'
• the x86-64 host runs the 'pull-buildtest' script, which:
  • does make/make check for multiple configs
  • includes one build from 'make clean' (almost everything else does an incremental build)
  • runs 'make check-tcg' on the all-linux-static config
  • runs a trivial set of 'ls' binaries for a bunch of linux-user guests (this is probably mostly redundant now we have check-tcg)

The parallel-buildtest script causes GNU parallel to print a series of lines with the logfiles containing the captured stdout/stderr from each machine. I run the output of those through the 'greplogs' script which looks for things that look like error messages. This is intended to capture warnings/errors which didn't manage to cause the make process to return failure for one reason or another.