Features/Meson/Design: Difference between revisions

From QEMU
m (Add meson category)
 
(104 intermediate revisions by 2 users not shown)
Line 15: Line 15:
* it should be possible to modify meson.build without knowing QEMU specific details, and that should be ''already'' possible now at the beginning of the transition (to avoid creating technical debt).  This means keeping the magic confined in Makefile rules and external scripts, while having a pretty plain meson.build.
* it should be possible to modify meson.build without knowing QEMU specific details, and that should be ''already'' possible now at the beginning of the transition (to avoid creating technical debt).  This means keeping the magic confined in Makefile rules and external scripts, while having a pretty plain meson.build.


[https://patchew.org/QEMU/1560165301-39026-1-git-send-email-pbonzini@redhat.com/ A PoC was posted] in June 2019.  It includes a program called "ninjatool" that replaces ninja for the few cases where meson invokes it directly (<tt>ninja -t compdb</tt>, <tt>ninja -t clean</tt>) and that also is able to convert build.ninja files to Make syntax. Because of this, it is trivial to have make build products that depend on meson build products. Therefore bottom up is the natural direction for the conversion.
[https://patchew.org/QEMU/1560165301-39026-1-git-send-email-pbonzini@redhat.com/ The first proof of concept was posted] in June 2019.  It includes a program called "ninjatool" that replaces ninja for the few cases where meson invokes it directly (<tt>ninja -t compdb</tt>, <tt>ninja -t clean</tt>) and that also is able to convert build.ninja files to Make syntax. Because of this, it is trivial to have make build products that depend on meson build products. Therefore bottom up is the natural direction for the conversion.


This fits in a generic plan to drop Makefile magic in favor of build rule generators written in high-level languages. For example, in order to keep <tt>make check</tt>-like rules working, we could generate Makefile rules from the output of <tt>meson introspect --tests</tt>.  This, together with the use of ninja2make also gives us an escape mechanism in case we want to do something that Meson cannot (yet) do, or doesn't do the way we want it.
This fits in a generic plan to drop Makefile magic in favor of build rule generators written in high-level languages. For example, in order to keep <tt>make check</tt>-like rules working, we could generate Makefile rules from the output of <tt>meson introspect --tests</tt>.  This, together with the use of ninja2make also gives us an escape mechanism in case we want to do something that Meson cannot (yet) do, or doesn't do the way we want it.
Line 23: Line 23:
== Issues ==
== Issues ==


* [https://github.com/mesonbuild/meson/pull/5149  Support a NINJA environment variable] (fixed in 0.51.0)
=== Meson ===
* [https://github.com/mesonbuild/meson/pull/5031 Kconfig Module] (included in 0.51.0 but still named "unstable-kconfig", should be stabilized before QEMU switches to meson)
Included in 0.51.0:
* [https://github.com/mesonbuild/meson/pull/5028 new module "sourceset" to match source file lists against configuration data] (included in 0.51.0)
* [https://github.com/mesonbuild/meson/pull/5149  Support a NINJA environment variable]
* [https://github.com/mesonbuild/meson/pull/5119 Script found by find_program() does not end up in dependencies] (fixed in 0.51.0)
* [https://github.com/mesonbuild/meson/pull/5031 Kconfig Module] (still named "unstable-kconfig", should be stabilized before QEMU switches to meson)
* [https://github.com/mesonbuild/meson/pull/5409 do not include host environment in test/benchmark introspection data] (fixed in 0.51.0)
* [https://github.com/mesonbuild/meson/pull/5028 new module "sourceset" to match source file lists against configuration data]
* [https://github.com/mesonbuild/meson/pull/5119 Script found by find_program() does not end up in dependencies]
* [https://github.com/mesonbuild/meson/pull/5409 do not include host environment in test/benchmark introspection data]
Included in 0.52.0:
* [https://github.com/mesonbuild/meson/pull/5644 Ugly command lines for custom targets with <tt>capture</tt>].
* [https://github.com/mesonbuild/meson/pull/5653 vs_module_defs for modules]
* [https://github.com/mesonbuild/meson/pull/5703 Allow kconfig to read from built files]
* [https://github.com/mesonbuild/meson/pull/5720 all_dependencies method for sourceset]
* [https://github.com/mesonbuild/meson/pull/5895 configure_file(): Allow multiple inputs in command mode ]
* [https://github.com/mesonbuild/meson/pull/5944 mtest: TAP: ignore unhandled input lines]
* [https://github.com/mesonbuild/meson/pull/5959  environment: Support taking values from dict]
* [https://github.com/mesonbuild/meson/pull/5702 Add depfile to configure_file()]
Included in 0.53.0:
* [https://github.com/mesonbuild/meson/pull/6058 Add tags targets]
* [https://github.com/mesonbuild/meson/pull/6325 mintro: include test protocol in introspection data]
Included in 0.55.0:
* [https://github.com/mesonbuild/meson/pull/6599 Rename unstable-kconfig to keyval]
Nice to have:
* [https://github.com/mesonbuild/meson/issues/5935 install_man() could allow custom_target() argument]
* [https://github.com/mesonbuild/meson/pull/5967 ExternalProgram: do not log locally found programs]
* [https://github.com/mesonbuild/meson/pull/6102 python: add automatic dependency tracking for scripts]
 
 
Others:
* Target file name must not contain a path separator - makes it impossible to generate e.g. audio/trace.h.  Ideas:
* Target file name must not contain a path separator - makes it impossible to generate e.g. audio/trace.h.  Ideas:
** add something that behaves like "generator()" but generates custom_targets
** add something that behaves like "generator()" but generates custom_targets
** generate files like trace/trace-audio.h and add one-line forwarding header files so that we can keep the <tt>#include "trace.h"</tt> idiom.
** generate files like trace/trace-audio.h and add one-line forwarding header files so that we can keep the <tt>#include "trace.h"</tt> idiom.
* Meson handles generated headers more sanely, without requiring Makefile to depend on them. However we need to check that files are not rebuilt too often if we don't have timestamp files.  See also [https://github.com/mesonbuild/meson/pull/5573 here].
* Meson doesn't like .inc.c files, it thinks they are sources. Renamed to .c.inc.
* Meson doesn't want to rename files on installations. Currently renaming ".stp-installed" to ".stp" and ".stp" to ".stp-build". Alternative: ".stp" and "trace/*.stp" or "trace/stp-installed/*.stp"
 
=== QEMU ===
* Deprecate in-tree build?  Meson only supports out-of-tree.
 
== Coding style ==
 
* Static libraries should be named <tt>libfoo</tt>.
* Dependencies (defined with any of <tt>dependency</tt>, <tt>cc.find_library</tt> or <tt>declare_dependency</tt>) should not have the <tt>lib</tt> prefix.
* Sourcesets should be named <tt>foo_ss</tt>.
* Do not assign disablers to variables.
* Do not use <tt>dependency</tt> unless you are able to remove the corresponding variable from config-host.mak.
* Use <tt>files('aa', 'bb')</tt>, not <tt><nowiki>files('''aaa bbb'''.split())</nowiki></tt>


== Status ==
== Status ==


* PoC: https://patchew.org/QEMU/1560165301-39026-1-git-send-email-pbonzini@redhat.com/
* [https://github.com/bonzini/qemu/commits/meson-poc Next PoC] status (see above for pending Meson pull requests):
* Next PoC should:
** ''(done)'' converted libqemuutil.a, trace, QAPI
** move ninjatool generation to meson.build
** ''(done)'' converted some of contrib/
** convert libqemuutil.a, trace, QAPI, possibly qemu-ga
** ''(done, merged)'' make TCG tests independent of Makefile.target
* Further phases:
** ''(done, merged)'' commonize code for modules
** ship Meson as a submodule
** ''(done)'' consistent coding style
** convert emulators and other tools
** ''(done)'' converted qemu-ga
** convert tests
** ''(done)'' converted remaining contrib/ and tools
** convert configure tests - until then, meson.build can read in the contents of config-host.mak
** ''(done)'' support ui/audio/block module bulding
** ''(done)'' conversion of emulators
** ''(done)'' extract changes that are independent of the conversion to separate patches
** ''(done)'' move scripts/create_config invocation from rules.mak to meson
** ''(done)'' create windowed/console variants of Windows executables
** ''(done)'' Generate .stp files from Meson.
** ''(done)'' ship Meson as a submodule
** ''(done)'' use new Meson 0.53 feature: [https://github.com/mesonbuild/meson/commit/9b1a85747334e3e1c21e9a27f4aa79f7f44c1f2d Dictionary entry using string variable as key]
** ''(done)'' convert pc-bios/*/Makefile (done)
** ''(done)'' Makefile.ninja rebuild rules are trigger-happy
** ''(done)'' support Meson tests in <tt>make check</tt>
* Before merging:
** ''(done)'' Makefile.ninja rebuild rules are still trigger-happy + optimize ninjatool parser
** ''(done)'' fix warnings ("consider using the built-in..."); -Wall is harder
** ''(done)'' Move the configure script's summary to meson.build, using the new summary() function
(https://mesonbuild.com/Reference-manual.html#summary)
** ''(done)'' sparse support using compile_commands.json
** ''(done)'' extract trace file reorganizations (too ugly to commit before 5.0 though).
** ''(done)'' optimize ninja2make.
** ''(done)'' rebase to a newer version of QEMU (now at 7bd9d0a)
** ''(done)'' Check if "ninja" and "meson test" work, even though "make" is expected to stay around.
** ''(done)'' Check which of the remaining junk can be removed from rules.mak
** ''(done)'' [https://github.com/mesonbuild/meson/pull/7448 Stabilize keyval] in meson (to be released in upcoming 0.56)
** ''(in progress)'' Update build-system.txt
** Test other supported platforms
* After merging:
** convert tests (done, except tcg/vm/docker)
** convert dtc (https://www.spinics.net/lists/devicetree-compiler/msg02815.html)
** handle submodules from meson (switch to subproject() if possible)
** make docker, vm & check-tcg standalone
** convert configure tests
 
== Benchmarking ==
 
                                  before    after
----------------------------------------------------
../configure                    18s      43s
../configure && make -j18        169s      172s
make -j18 (do nothing)          4s        4s
make -j4 (do nothing)            6.5s      4s
touch ../configure && make -j18  52s      62s
touch ../configure && make -j4  71s      62s
make clean                      19s      2s
make -j18 clean                  3.5s      2s
----------------------------------------------------
ninja -j18 (do nothing)                    1s
ninja -t clean                            2s
ninja -j18                                96s
 
(Using ninja is not yet supported due to submodule handling in the Makefile).
 
[[Category:Completed_feature_pages]]
[[Category:Meson]]

Latest revision as of 06:46, 14 June 2024

Rules.mak has gotten insane, let's get rid of it!

More seriously, the current build system causes clutter in the toplevel Makefiles, and it makes simple tasks harder than they should be. For example, each executable in contrib/ has to touch three files: Makefile, Makefile.objs and contrib/*/Makefile.objs.

Meson has several advantages that directly matter for QEMU including the following:

  • build definitions in a very readable and user friendly DSL, which supports looping and conditions
  • ability to introspect the build definitions so that you can find out what is built without building it (the DSL is not Turing complete and most objects in it are immutable, so it cannot be abused that much :))
  • support for a non-recursive build from per-subdirectory input (similar to Makefile.objs)
  • strong checking of prerequisites when Meson is run, so that it's usually unnecessary to debug whatever files Meson generates
  • ease of distributing a full copy of Meson to support distros that ship an older version (no dependencies apart from Python 3.5).

The transition is designed around the following goals:

  • the build system should make it trivial to do trivial things; easy to do things that are a matter of cut-and-paste from something that already exist; possible to do everything else.
  • it should remain trivial to do things that used to be trivial, and most "make" invocations should be kept the same at least until everything is converted (and we can perhaps declare a flag day). It should not become harder to do things that are a matter of cut-and-paste.
  • it should be possible to modify meson.build without knowing QEMU specific details, and that should be already possible now at the beginning of the transition (to avoid creating technical debt). This means keeping the magic confined in Makefile rules and external scripts, while having a pretty plain meson.build.

The first proof of concept was posted in June 2019. It includes a program called "ninjatool" that replaces ninja for the few cases where meson invokes it directly (ninja -t compdb, ninja -t clean) and that also is able to convert build.ninja files to Make syntax. Because of this, it is trivial to have make build products that depend on meson build products. Therefore bottom up is the natural direction for the conversion.

This fits in a generic plan to drop Makefile magic in favor of build rule generators written in high-level languages. For example, in order to keep make check-like rules working, we could generate Makefile rules from the output of meson introspect --tests. This, together with the use of ninja2make also gives us an escape mechanism in case we want to do something that Meson cannot (yet) do, or doesn't do the way we want it.

The alternative of having two parallel build systems was rejected. This would introduce extra work for people not involved in the conversion, and the risk of bitrotting one of the two build systems is large.

Issues

Meson

Included in 0.51.0:

Included in 0.52.0:

Included in 0.53.0:

Included in 0.55.0:

Nice to have:


Others:

  • Target file name must not contain a path separator - makes it impossible to generate e.g. audio/trace.h. Ideas:
    • add something that behaves like "generator()" but generates custom_targets
    • generate files like trace/trace-audio.h and add one-line forwarding header files so that we can keep the #include "trace.h" idiom.
  • Meson doesn't like .inc.c files, it thinks they are sources. Renamed to .c.inc.
  • Meson doesn't want to rename files on installations. Currently renaming ".stp-installed" to ".stp" and ".stp" to ".stp-build". Alternative: ".stp" and "trace/*.stp" or "trace/stp-installed/*.stp"

QEMU

  • Deprecate in-tree build? Meson only supports out-of-tree.

Coding style

  • Static libraries should be named libfoo.
  • Dependencies (defined with any of dependency, cc.find_library or declare_dependency) should not have the lib prefix.
  • Sourcesets should be named foo_ss.
  • Do not assign disablers to variables.
  • Do not use dependency unless you are able to remove the corresponding variable from config-host.mak.
  • Use files('aa', 'bb'), not files('''aaa bbb'''.split())

Status

  • Next PoC status (see above for pending Meson pull requests):
    • (done) converted libqemuutil.a, trace, QAPI
    • (done) converted some of contrib/
    • (done, merged) make TCG tests independent of Makefile.target
    • (done, merged) commonize code for modules
    • (done) consistent coding style
    • (done) converted qemu-ga
    • (done) converted remaining contrib/ and tools
    • (done) support ui/audio/block module bulding
    • (done) conversion of emulators
    • (done) extract changes that are independent of the conversion to separate patches
    • (done) move scripts/create_config invocation from rules.mak to meson
    • (done) create windowed/console variants of Windows executables
    • (done) Generate .stp files from Meson.
    • (done) ship Meson as a submodule
    • (done) use new Meson 0.53 feature: Dictionary entry using string variable as key
    • (done) convert pc-bios/*/Makefile (done)
    • (done) Makefile.ninja rebuild rules are trigger-happy
    • (done) support Meson tests in make check
  • Before merging:
    • (done) Makefile.ninja rebuild rules are still trigger-happy + optimize ninjatool parser
    • (done) fix warnings ("consider using the built-in..."); -Wall is harder
    • (done) Move the configure script's summary to meson.build, using the new summary() function

(https://mesonbuild.com/Reference-manual.html#summary)

    • (done) sparse support using compile_commands.json
    • (done) extract trace file reorganizations (too ugly to commit before 5.0 though).
    • (done) optimize ninja2make.
    • (done) rebase to a newer version of QEMU (now at 7bd9d0a)
    • (done) Check if "ninja" and "meson test" work, even though "make" is expected to stay around.
    • (done) Check which of the remaining junk can be removed from rules.mak
    • (done) Stabilize keyval in meson (to be released in upcoming 0.56)
    • (in progress) Update build-system.txt
    • Test other supported platforms
  • After merging:

Benchmarking

                                 before    after
----------------------------------------------------
../configure                     18s       43s
../configure && make -j18        169s      172s
make -j18 (do nothing)           4s        4s
make -j4 (do nothing)            6.5s      4s
touch ../configure && make -j18  52s       62s
touch ../configure && make -j4   71s       62s
make clean                       19s       2s
make -j18 clean                  3.5s      2s
----------------------------------------------------
ninja -j18 (do nothing)                    1s
ninja -t clean                             2s
ninja -j18                                 96s

(Using ninja is not yet supported due to submodule handling in the Makefile).