Features/Meson: Difference between revisions

From QEMU
(Created page with "What's next? == Blockers for further conversions == ;Complete moving installation rules to meson.build :Marc-André and Paolo already have patches for this. You're welcome t...")
 
 
(140 intermediate revisions by 6 users not shown)
Line 1: Line 1:
What's next?
For the rationale behind the conversion and the development notes, see [[Features/Meson/Design]].


== Blockers for further conversions ==
== Summary of changes ==
;Source code changes:
* You cannot <tt>#include "config-target.h"</tt> and <tt>"config-devices.h"</tt>.  Instead, you <tt>#include CONFIG_TARGET</tt> and <tt>CONFIG_DEVICES</tt> (i.e. filenames are specified via macros; <tt>meson.build</tt> takes care of defining the macros).
* .c files (including .inc.c files) cannot be #included.  They have been renamed to .c.inc, for consistency with existing .rst.inc files.
* The name of files produced by decodetree is fixed (the C code for <tt>XYZ.decode</tt> ends up in <tt>decode-XYZ.c.inc</tt>).
* Subdirectories with trace events require a forwarding header file; see hw/scsi/trace.h for an example.


;Build layout changes:
* The Makefile is not recursive anymore.  You don't <tt>make aarch64-softmmu/all</tt> or <tt>make arm-linux-user/all</tt>, you <tt>make qemu-system-aarch64</tt> or <tt>make qemu-arm</tt>.
* Emulator binaries have moved to the root of the build directory.  For now, paths such as cris-softmmu/qemu-system-cris are preserved for backwards compatibility through symlinks (created by <tt>configure</tt>).  They are just a user convenience however.  You cannot "rm" them to force a rebuild, for example.
* Other binaries have moved to the directory where they reside in the source tree.  For example, "virtiofsd" has moved to "tools/virtiofsd/virtiofsd".
* Most binaries are built by default by <tt>make all</tt>, including several in contrib/ that weren't built before.  Currently, vhost-user-blk and rdmacm-mux are not because they fail on 32-bit and big-endian platforms respectively.
* Sphinx manuals are only built if their constituent files are changed, therefore "make sphinxdocs" will be plenty fast when you've only modified a file or two.
* Dependencies added to libqemuutil.a will propagate to all programs that link to it.  If util/ code has dependencies, it should be conditional on have_system or have_block if applicable (this was a bug in the Makefiles).
* For bisection: incremental builds across the conversion work fine in the forwards direction.  They probably don't work at all backwards across the conversion, so you'll need to throw away your build tree when bisection moves you backwards across the conversion.  This is mitigated by the fact that bisection usually starts at a release, and the patches have been applied on top of the 5.1.0 tag.
;New conventions:
* New configure options should also be added to <tt>meson_options.txt</tt> so that dependencies are tested in <tt>meson.build</tt> rather than <tt>configure</tt>.  How to do so is documented in docs/devel.
;Impact to developers
* If you were already using out-of-tree (VPATH) builds, nothing changes; an incremental pull of these changes should still build.
* If you are used to in-tree builds, you'll want to do <tt>make distclean</tt> prior to merging in these changes (if you forget, git will remind you about various trace.h files that still exist as built files in your in-tree build vs. checked-in files post-patch).  After that one-time clean, you can then proceed to do <tt>./configure && make</tt> as before, but things will now automatically create a subdirectory build/ on your behalf where the actual build is performed.
== CI holes ==
The following issues were _not_ found by GitLab CI:
* SDL 2.0.8 requires -Wno-undef
* s390x-softmmu is special for cross-compilation, because it is the only target to use the host C compiler.
* There is special code in configure to handle a C++ compiler that does not play well with the C compiler.
* Not all linux-user TCG tests are covered.
* Big endian platforms are not covered.
* ARM host platforms are not covered.
* 32-bit POSIX platforms are not covered.
* SystemTap is not covered.
* daxctl is not covered.
* --disable-tools --enable-system builds are not covered.
* Peter uses Ubuntu+MXE instead of Fedora-mingw to build cross compilers.  docker-test-mingw only works for Fedora and should be made more generic.  Also container images for Fedora-mingw should be prepared in the same way as the existing container images for Debian+MXE.
== Reported and unfixed bugs ==
* meson.build files are missing copyright/license header comments
== Known issues ==
* Meson does not like outputs of custom_target (e.g. tracetool and qapi-gen.py) to be in multiple directories.  Right now this is only a problem for the QAPI tests.
== Projects ==
;Write a linter for meson.build
:Visit the output of <tt>meson introspect --ast meson.build</tt> (including subdirectories) and warn for:
:* dependencies searched with a method other than "pkg-config" or "system"
:* dependencies lacking "static: enable_static"
:* variables not defined on all paths (Meson accepts undefined variables on the RHS of short-circuiting boolean operators)
<!--
== Done ==
;"Handle (faked-up) in-tree build
:'''Benefit''': simpler upgrade path
;Complete moving installation rules to meson.build
;Complete moving installation rules to meson.build
:Marc-André and Paolo already have patches for this. You're welcome to propose yourself for helping!
:'''Benefit:''' allows removing parts of Makefile, leaving it as just an orchestrator for submodules, Makefile.ninja and Makefile.mtest
:'''Benefit:''' allows removing parts of Makefile, leaving it as just an orchestrator for submodules, Makefile.ninja and Makefile.mtest
;Complete moving build rules to meson.build
;Complete moving tests/Makefile.include build rules to meson.build
:Move the build rules from tests/Makefile.include to tests/meson.build.  You can take inspiration from tests/qtest/Makefile.include
:'''Benefit:''' allows removing CFLAGS, LDFLAGS, LIBS and LIBS_SOFTMMU, as well as almost all of rules.mak.
:'''Benefit:''' allows removing CFLAGS, LDFLAGS, LIBS and LIBS_SOFTMMU, as well as most if not all of rules.mak.
;Remove pkg-config variables from QEMU_CFLAGS and LIBS
:All Meson executables should specify their dependencies explicitly, either directly or indirectly via declare_dependency.  Makefiles instead did not propagate dependencies correctly from static libraries, for example. Therefore, flags for dependencies had to be included in QEMU_CFLAGS and LIBS.
:'''Benefit:''' allows moving pkg-config dependencies to Meson.
;Relocatable install
:'''Benefit:''' on Windows only /usr/share/qemu is relocatable, everything else (icons etc.) is not.  POSIX is currently not relocatable and would get it for free.
;Convert submodule build rules to meson
:Our submodules are small and the rules to build them can easily be embedded in QEMU itself (just like we're already doing for tests/fp).  SLIRP could use subprojects, but that requires https://github.com/mesonbuild/meson/pull/7740.
:'''Benefit''': make QEMU build fully non-recursive
:'''Disadvantage''': QEMU duplicates build rules for foreign code.
;Include config-target.mak files in the source tree
:Remove the configuration loop in configure and just include the content of the .mak files directly in the tree, for example in a target/configs/ directory(There are some small complications because config-target.mak file currently includes the host disassembler symbols too, e.g. CONFIG_I386_DIS).
:'''Benefit''': remove large swaths of imperative code, remove *-linux-user and *-softmmu directories from the build tree.
;Get rid of ninjatool and just require Ninja
:'''Benefit''': 1000 lines of code go away, can use things such as <tt>ninja -t clean &lt;TARGET&gt;</tt>
:'''Disadvantage''': recursive build re-enters through the window
;Create configure help and command line parsing code from meson_options.txt.
:Use the output of "meson introspect --buildoptions" to initialize variables, parse command line options, print the help and create -D command line options for meson.  PoC [https://gist.github.com/bonzini/54571af56c1de4c3346f69b1483d0245 here] (requires developers to install "jq").
:'''Benefit''': remove code duplication, centralize command line handling
-->
 
== Links to Meson issues and pull requests ==
Meson changes that are pending or have not been put to use yet.
=== Pull requests ===
;[https://github.com/mesonbuild/meson/pull/6102 python: add automatic dependency tracking for scripts]
:Avoids listing sources for qapi-gen.py and tracetool.py


== Easy ==
=== Merged ===
;Move pkg-config tests to meson.build
;[https://github.com/mesonbuild/meson/pull/9400 allow custom targets in extract_objects and test] (0.61.0)
:Replace declare_dependency statements with dependency statements
;[https://github.com/mesonbuild/meson/pull/10419 fix custom_target whose input is extract_objects of generated sources] (0.63.0)
:'''Benefit''': allows removing *_CFLAGS and *_LIBS variables from config-host.mak.  Allows using dependency objects instead of CONFIG_* symbols when declaring sourcesets.
:Removes dependency of dbus-display on --enable-modules and allows modinfo to receive object files instead of source files. Needs new ninja though.
;Change simple config-host.mak symbols to Meson options
<!--
:Some symbols (e.g. CONFIG_PARALLELS) are only used placed in config-host.mak to be read from Meson. Instead pass them through -D... command-line options
[https://github.com/mesonbuild/meson/pull/10206 add_project_dependencies] (0.63.0)
:'''Benefit''': allows removing CONFIG_* variables from config-host.mak.
:Needed to move glib detection to Meson.
-->
;[https://github.com/mesonbuild/meson/pull/11198 Add declare_dependency(objects: ...)] (1.1.0)
:Enables a better fix for https://bugs.launchpad.net/qemu/+bug/1909256.


== Medium ==
<!--
;Include config-target.mak files in the tree
=== Other features===
:Remove the configuration loop in configure and just include the content of the .mak files directly in the tree, for example in a target/configs/ directory.  (There are some small complications because config-target.mak file currently includes the host disassembler symbols too, e.g. CONFIG_I386_DIS).
* prefer_static option can be used for --static. (0.63.0)
:'''Benefit''': remove large swaths of imperative code, remove *-linux-user and *-softmmu directories from the build tree.
-->
;Move other compile tests from configure to meson.build

Latest revision as of 17:57, 18 May 2023

For the rationale behind the conversion and the development notes, see Features/Meson/Design.

Summary of changes

Source code changes
  • You cannot #include "config-target.h" and "config-devices.h". Instead, you #include CONFIG_TARGET and CONFIG_DEVICES (i.e. filenames are specified via macros; meson.build takes care of defining the macros).
  • .c files (including .inc.c files) cannot be #included. They have been renamed to .c.inc, for consistency with existing .rst.inc files.
  • The name of files produced by decodetree is fixed (the C code for XYZ.decode ends up in decode-XYZ.c.inc).
  • Subdirectories with trace events require a forwarding header file; see hw/scsi/trace.h for an example.
Build layout changes
  • The Makefile is not recursive anymore. You don't make aarch64-softmmu/all or make arm-linux-user/all, you make qemu-system-aarch64 or make qemu-arm.
  • Emulator binaries have moved to the root of the build directory. For now, paths such as cris-softmmu/qemu-system-cris are preserved for backwards compatibility through symlinks (created by configure). They are just a user convenience however. You cannot "rm" them to force a rebuild, for example.
  • Other binaries have moved to the directory where they reside in the source tree. For example, "virtiofsd" has moved to "tools/virtiofsd/virtiofsd".
  • Most binaries are built by default by make all, including several in contrib/ that weren't built before. Currently, vhost-user-blk and rdmacm-mux are not because they fail on 32-bit and big-endian platforms respectively.
  • Sphinx manuals are only built if their constituent files are changed, therefore "make sphinxdocs" will be plenty fast when you've only modified a file or two.
  • Dependencies added to libqemuutil.a will propagate to all programs that link to it. If util/ code has dependencies, it should be conditional on have_system or have_block if applicable (this was a bug in the Makefiles).
  • For bisection: incremental builds across the conversion work fine in the forwards direction. They probably don't work at all backwards across the conversion, so you'll need to throw away your build tree when bisection moves you backwards across the conversion. This is mitigated by the fact that bisection usually starts at a release, and the patches have been applied on top of the 5.1.0 tag.
New conventions
  • New configure options should also be added to meson_options.txt so that dependencies are tested in meson.build rather than configure. How to do so is documented in docs/devel.
Impact to developers
  • If you were already using out-of-tree (VPATH) builds, nothing changes; an incremental pull of these changes should still build.
  • If you are used to in-tree builds, you'll want to do make distclean prior to merging in these changes (if you forget, git will remind you about various trace.h files that still exist as built files in your in-tree build vs. checked-in files post-patch). After that one-time clean, you can then proceed to do ./configure && make as before, but things will now automatically create a subdirectory build/ on your behalf where the actual build is performed.

CI holes

The following issues were _not_ found by GitLab CI:

  • SDL 2.0.8 requires -Wno-undef
  • s390x-softmmu is special for cross-compilation, because it is the only target to use the host C compiler.
  • There is special code in configure to handle a C++ compiler that does not play well with the C compiler.
  • Not all linux-user TCG tests are covered.
  • Big endian platforms are not covered.
  • ARM host platforms are not covered.
  • 32-bit POSIX platforms are not covered.
  • SystemTap is not covered.
  • daxctl is not covered.
  • --disable-tools --enable-system builds are not covered.
  • Peter uses Ubuntu+MXE instead of Fedora-mingw to build cross compilers. docker-test-mingw only works for Fedora and should be made more generic. Also container images for Fedora-mingw should be prepared in the same way as the existing container images for Debian+MXE.

Reported and unfixed bugs

  • meson.build files are missing copyright/license header comments

Known issues

  • Meson does not like outputs of custom_target (e.g. tracetool and qapi-gen.py) to be in multiple directories. Right now this is only a problem for the QAPI tests.

Projects

Write a linter for meson.build
Visit the output of meson introspect --ast meson.build (including subdirectories) and warn for:
  • dependencies searched with a method other than "pkg-config" or "system"
  • dependencies lacking "static: enable_static"
  • variables not defined on all paths (Meson accepts undefined variables on the RHS of short-circuiting boolean operators)


Links to Meson issues and pull requests

Meson changes that are pending or have not been put to use yet.

Pull requests

python: add automatic dependency tracking for scripts
Avoids listing sources for qapi-gen.py and tracetool.py

Merged

allow custom targets in extract_objects and test (0.61.0)
fix custom_target whose input is extract_objects of generated sources (0.63.0)
Removes dependency of dbus-display on --enable-modules and allows modinfo to receive object files instead of source files. Needs new ninja though.
Add declare_dependency(objects: ...) (1.1.0)
Enables a better fix for https://bugs.launchpad.net/qemu/+bug/1909256.