GIT submodule usage

QEMU uses GIT submodules to reference code from a number of 3rd party source repositories.

For developers building QEMU from official releases, there is no need to understand git submodules as the 3rd party source is all bundled into the tar.xz when releases are created.

For developers building QEMU directly from GIT, however, 'make' will attempt to automatically populated the submodules with the requisite content. In general this should be transparent to developers, but there are some development build environments in which automatic submodule updates will not work. What follows in this page will explain the way submodules are handled and how to deal with unusual build environment requirements that may arise.

Automatic submodule checkout

The QEMU GIT repository records a changeset hash for each submodule it includes. When the build is performed various the submodules need to be checked out in the source directory with the matching changeset hash.

Some modules (e.g. ui/keycodemapdb) are compulsory for all builds, while other modules (e.g. dtc) are optional depending on build config & host system. When 'configure' runs it will determine which submodules are to be used and print a list of them in the summary output. For example

  $ ./configure
  ...snip...
  local state directory   /usr/local/var
  Manual directory  /usr/local/share/man
  ELF interp prefix /usr/gnemul/qemu-%M
  Source path       /home/berrange/src/virt/qemu
  GIT submodules    ui/keycodemapdb dtc capstone
  C compiler        cc
  Host C compiler   cc
  C++ compiler      c++
  Objective-C compiler clang
  ...snip...

When 'make' runs the first thing it will do is invoke

 $ $SRC_DIR/scripts/git-submodule.sh status LIST OF MODULES

this will check the .git-submodule-status file against the expected changeset hashes for each submodule. If any listed submodule is not checked out, or checked out with the wrong changeset hash, it will exit with non-zero status.

If 'make' learnt that the submodule checkout was out of date it will then run

 $ $SRC_DIR/scripts/git-submodule.sh update LIST OF MODULES

This will checkout each submodule in the source directory, and then return the status in the .git-submodule-status file.

'make' will now proceed with the rest of the QEMU build as normal, so this automatic checkout of submodules should be transparent to most developers.

Cleaning up submodules from source checkout

If wishing to test build from a completely pristine source directory state, it is important to be aware that a simple 'git clean' will NOT purge checked out submodules. Thus to fully clean a source directory requires

  $ git clean -f -x -d
  $ git submodule deinit --all -f

Updating submodules to latest upstream code

Each submodule used by QEMU has an associated upstream changeset hash associated with it. This ensures that for a given point in time in QEMU GIT history, there is always an exact corresponding point in time in each submodule's GIT history. The implication, however, is that when further commits are made to the submodules upstream master branch these will not be picked up by QEMU without explicit action by a QEMU maintainer. To illustrate how to update to the latest upstream code for a submodule, consider this example using the ui/keycodemapdb module.

  $ cd ui/keycodemapdb
  $ git fetch origin
  $ git checkout origin/master
  $ cd ../..
  $ git add ui/keycodemapdb
  $ git commit -s -m 'Update keycodemapdb to latest upstream because ...why...'

Building when automatic submodule update fails

This describes potential solutions when the automatic submodule update is unable to work transparently

Read only source checkouts

There may be times where a developer's GIT checkout of QEMU is read-only. For example, the checkout may be made on one machine, and then exported read-only over NFS to another machine where the builds are performed in a different directory. In such a scenario, 'make' on the build machine will be unable to automatically update the submodule checkout because ths source directory is readonly. To deal with this requires the developer to manually update the submodules before running 'make' on the build machine. The procedure to deal with this is as follows...

Run configure as normal, with whatever arguments are usually given:

  $ ./configure ...args...

In the summary text configure will list what submodules it wishes to have present:

  GIT submodules    ui/keycodemapdb dtc capstone

Now in the machine where the source directory is writable, we must manually checkout the submodules, passing this list of submodules, in the exact printed order:

 $ scripts/git-submodule.sh update ui/keycodemapdb dtc capstone

This will checkout each submodule, and populate the .git-submodule-status file to reflect it

It is now possible to go back to the build machine and run 'make' as normal.

Periodically will pulling in new code from GIT master upstream, the list of required submodule hashes will change. If this happens, 'make' on the build machine will try (and fail) to update the submodules. In this case, just re-run the 'git-submodule.sh update LIST OF MODULES' command again.

Builds on machines without direct network access

There may be times where a developer's build machine does not have direct internet access required to reach git.qemu.org to checkout GIT submodules. It may be possible to work around these network restrictions by running GIT via a proxy command, such as a 'tsocks'. In such a case, it is possible to instruct QEMU to use this proxy running to invoke GIT. For example, to use 'tsocks' with GIT a configure arg can be given:

  $ ./configure --with-git='tsocks git' ...other args...

Now when 'make' updates submodules it will automatically invoke 'tsocks git' instead of just 'git'. The rest of the submodule handling should be totally transparent at this point.

In more complicated scenarios it is possible to run the 'git-submodules.sh' script explicitly as described in the previous section.