Problem Statement

A gating CI is a prerequisite to having a multi-maintainer model of merging. By having a common set of tests that are run prior to a merge you do not rely on who is currently doing merging duties having access to the current set of test machines.

Currently pre-merge testing is done via a set of tests done by ad-hoc shell scripts run on a set of machines using personal accounts of the overall maintainer. We want to replace this ad-hoc system with one which:

does not use any machines which aren't usable with generic project role accounts
uses a known and maintainable CI system (eg Gitlab) rather than hand-hacked scripts
can be handed over to another person to handle releases

Current Tests

Peter currently has access to a random menagerie of build and test machines which are accessed by his scripts.

The scripts are kept in:

https://git.linaro.org/people/peter.maydell/misc-scripts.git/tree

though they are best treated as a reference for what we currently do rather than used as a base for anything.

The set of machine I currently test on are:

an S390x box (this is provided to the project by IBM's Community Cloud so can be used for the new CI setup)
aarch32 (as a chroot on an aarch64 system)
aarch64
ppc64 (on the GCC compile farm)
OSX
Windows crossbuilds
NetBSD, FreeBSD and OpenBSD using the tests/vm VMs
x86-64 Linux with a variety of different build configs (see the 'remake-merge-builds' script for how these are set up)

I also have access to a SPARC box but am not currently testing with it as there are hangs which I did not have time to investigate.

Testing process:

I get an email which is a pull request, and I run the "apply-pullreq" script, which takes the GIT URL and tag/branch name to test.
apply-pullreq performs the merge into a 'staging' branch
apply-pullreq also performs some simple local tests:
- does git verify-tag like the GPG signature?
- are we trying to apply the pull before reopening the dev tree for a new release?
- does the pull include commits with bad UTF8 or bogus qemu-devel email addresses?
- submodule updates are only allowed if the --submodule-ok option was specifically passed
apply-pullreq then invokes parallel-buildtest to do the actual testing
parallel-buildtest is a trivial wrapper around GNU Parallel which invokes 'mergebuild' on each of the test machines
if all is OK then the user gets to do the 'git push' to push the staging branch to master

In almost all cases 'mergebuild' is simply "run 'make -C build' and then 'make -C build check'". The exceptions are:

the Windows crossbuilds don't try to run 'make check'
the x86-64 host runs the 'pull-buildtest' script, which:
- does make/make check for multiple configs
- includes one build from 'make clean' (almost everything else does an incremental build)
- runs 'make check-tcg' on the all-linux-static config
- runs a trivial set of 'ls' binaries for a bunch of linux-user guests (this is probably mostly redundant now we have check-tcg)

The parallel-buildtest script causes GNU parallel to print a series of lines with the logfiles containing the captured stdout/stderr from each machine. I run the output of those through the 'greplogs' script which looks for things that look like error messages. This is intended to capture warnings/errors which didn't manage to cause the make process to return failure for one reason or another.