Features/Documentation

QEMU's documentation needs some reorganization.

Ultimately there should be six manuals:

QEMU User-mode Emulation User's Guide (docs/user, Features/Documentation/user): currently in qemu-doc.texi
QEMU System Emulation User's Guide (docs/system, Features/Documentation/system): Category:User documentation; the largest part of qemu-doc.texi; also covers qemu-img, qemu-io; docs/ (not subdirectories)
QEMU System Emulation Management and Interoperability Guide (docs/interop, Features/Documentation/interop)
QEMU System Emulation Guest Hardware Specifications (docs/specs, Features/Documentation/specs)
QEMU Tools Guide (docs/tools): covers qemu-img, qemu-nbd, virtfs-proxy-helper and other "standalone tools" binaries
QEMU Developer's Guide (docs/devel, Features/Documentation/devel): Category:Developer documentation; docs/devel/; the implementation notes in qemu-doc.texi; tcg/README; doc comments in the source code; automatically generated docs for Python classes in qemu-iotests and scripts/qmp; not to be shipped to end users (unlike the other manuals)

Choices

Based on experience from the Linux kernel, QEMU's docs pipeline is going to be based on Sphinx (sample developer doc, sample user doc. Sphinx is extensible and it is easy to add new input formats and input directives.

Currently, QEMU documentation is written in a mix of Texinfo and text files roughly based on Markdown. Sphinx's native format is reStructuredText (rST). The plan is to (eventually) convert everything to rST format.

Currently, QEMU doc comments have never been actually used together with a actual documentation generator in mind, but they are roughly based on gtk-doc syntax. We will probably end up keeping a copy of the Linux kernel's kernel-doc script. kernel-doc's main supported backend is rST. There is a GSoC project to look at generating API documentation from our doc comments.

Pre-Sphinx status

(This concentrates on the documentation that's installed by 'make install', because anything else is for developers only, making it lower priority to clean up and more amenable to messing around with anyway.)

As of QEMU 4.0 (early 2019) we install:

in $DESTDIR/usr/local/share/doc/qemu:

qemu-doc.html
qemu-ga-ref.html
qemu-qmp-ref.html
qemu-doc.txt
qemu-ga-ref.txt
qemu-qmp-ref.txt
interop/  (the only Sphinx manual we currently ship)

in $DESTDIR/usr/local/share/man/:

man1/qemu.1
man1/qemu-img.1
man1/virtfs-proxy-helper.1
man1/qemu-trace-stap.1
man8/qemu-ga.8
man8/qemu-nbd.8
man7/qemu-block-drivers.7
man7/qemu-cpu-models.7
man7/qemu-ga-ref.7
man7/qemu-qmp-ref.7

(possibly I missed one or two files, because what we install depends on configure options and maybe my test build-n-install didn't build everything we might ship.)

This documentation is generated from a mix of:

hand-written texinfo:

qemu-doc.texi (a top level file with both content and @include directives)
qemu-deprecated.texi
qemu-ga.texi
qemu-img.texi
qemu-nbd.texi
qemu-option-trace.texi
qemu-tech.texi
docs/security.texi
docs/qemu-cpu-models.texi
docs/qemu-block-drivers.texi
docs/interop/qemu-ga-ref.texi
docs/interop/qemu-qmp-ref.texi
fsdev/virtfs-proxy-helper.texi
scripts/qemu-trace-stap.texi

texinfo sections extracted from *.hx files by hxtool
texinfo autogenerated from json by qapi-gen.py

(The .html and .txt files are generated from the texinfo directly; the manpages via texi2pod and pod2man, which effectively create the manpages from marked-up subsqections of the input texi.)

Intended final status

We eventually want to have:

in $DESTDIR/usr/local/share/doc/qemu:

interop/ sphinx manual
system/ sphinx manual (system emulation user's guide)
user/ sphinx manual (user-mode emulation user's guide)
specs/ sphinx manual (guest hardware etc specs)

in $DESTDIR/usr/local/share/man/:

the same set of manpages we have currently

not actually installed:

devel/ sphinx manual

Sphinx supports a "plain text" output format, so we could create a "one big text file" format output for each of our four installed manuals. However there doesn't seem to be much support for retaining a plain-text version of the documentation, so we should just ship the HTML version.

Transition plan

This is based on the plan proposed in this email: https://lists.gnu.org/archive/html/qemu-devel/2019-05/msg04932.html

0: API documentation generation

Work on generating API documentation can be done independently and is not addressed here, because that will just go into the devel/ manual, and doesn't interact at all with the user-facing manuals. Paolo sorted out integrating the kerneldoc Sphinx extension into our build process, and we use it in docs/devel/bitops.rst and docs/devel/memory.rst as an example/proof of concept; so the machinery is all there and tested.

1: qemu-ga-ref and qemu-qmp-ref (IN PROGRESS)

qemu-ga-ref and qemu-qmp-ref will become part of the interop/ manual. This requires:

adding rst-generation support to qapi-gen.py
converting the hand-written texinfo parts to rst
listing those new .rst files in docs/interop/index.rst
makefile runes to create manpages via the 'man' sphinx builder from docs/interop/qemu-ga-ref.rst and qemu-qmp-ref.rst
there is a tests/qapi-schema/doc-good.test.texi which I think is acting as a test of the texi doc generation; this should be converted to test the rst doc generation instead
delete all the old texinfo build machinery, install runes, etc

We could do this as its own patch series which would end up with a result that would be fine to ship, even if we don't get any of the other manuals converted for a particular QEMU release. It's the easiest part because these docs already have an obvious home in a Sphinx manual we're already shipping.

For the "create manpage" part, Sphinx supports a "manpage" builder, which will do what we need. It works by specifying a list of "this .rst becomes this manpage in this manpage-section", so we can for instance have the qemu-ga-ref and qemu-qmp-ref be subsections of the HTML interop/ manual but also pulled out into their own manpages. (It is also possible to make sections of a .rst file be output only for particular builders, but there seem to be some limitations on it -- notably it filters the output but it doesn't affect things like tables of contents for formats which have them. So far it looks like we can do what we want without it.)

There is a patch series on the list ("Convert QAPI doc comments to generate rST instead of texinfo") which handles the conversion of these autogenerated docs to rST.

2: Move developer docs into devel manual (DONE)

Any parts of the texi docs which are really "developer docs" should be moved into devel/

This is easy because we don't need to maintain a very coherent manual for devel/, we can just have it be a bucket of random documents for the moment. In particular:

the qemu-tech.texi "translator internals" section should be moved (I suspect it's a bit out of date so we should add a warning note at the top of the new file about that)
I think we should just delete the "QEMU compared to other emulators" section of qemu-tech.texi (and the accompanying bibliography of URLs) since it hasn't been updated in several years and it's not really documenting QEMU. This sort of general-information stuff would live better on the wiki if we care about it at all.
I don't think there's anything else that's really developer documentation, but I haven't looked too closely

A patchset doing this has been sent to the list and was applied in time for QEMU 4.1.

3: Convert things which are mostly standalone manpages (DONE)

Round about here things start to get trickier because we are looking at things which are part of the big fat qemu-doc.html. The easiest next pieces to pull out are the ones which are standalone manpages which we're also just using @include to stuff into qemu-doc.html:

qemu-ga.texi (DONE)
qemu-nbd.texi (DONE)
qemu-img.texi (DONE)
qemu-cpu-models.texi (DONE)
qemu-block-drivers.texi (DONE)
virtfs-proxy-helper.texi (DONE)
scripts/qemu-trace-stap.texi (DONE)

We can:

convert each of these to a .rst file in the right sphinx manual (probably mostly system/)
generate the manpage output using sphinx as we did for qemu-ga-ref etc
just drop the @include from qemu-doc.texi

qemu-img.texi here will be the tricky one as it uses hxtool to generate the texi for the command line option descriptions. The plan here is to have a Sphinx extension which handles slicing fragments out of the .hx files -- we use SRST/ERST directives that do the same thing for rST as the STEXI/ETEXI ones we have at the moment do for texinfo. (The extension is now in master.)

This will result in a slightly "bucket of distinct docs" feel to the system/ manual at this point, but these files are all basically mostly self-contained, so it wouldn't be too terrible to ship a QEMU in this state.

4: Create the tools manual (DONE)

The idea of having a docs/tools manual was a late one, so some existing docs got moved to docs/interop that we'd now like in tools. We need to create the tools manual and move these files to it. I've sent a patchset to the list to do this.

5: Convert everything else (DONE)

At this point what we have left is:

qemu-doc.texi itself
qemu-deprecated.texi
docs/security.texi
qemu-option-trace.texi
the leftovers from qemu-tech.texi
the stuff generated via hxtool from hmp-commands-info.hx, hmp-commands.hx and qemu-options.hx

and we pretty much need to:

convert all that over to rst in the system/ and user/ manuals (perhaps sometimes interop/)
generate the qemu.1 manpage
delete all the leftover machinery

We could do at least some of this in chunks, but if we had to release QEMU midway through this part of the transition we'd have a very weird setup where half the core-emulator-docs were in the old html file and the other half were in the new style manuals, so we should definitely try to avoid doing that.

6: Further cleanup

The steps above prioritizing getting away from having any Texinfo documentation. Once we've done that we can look at improvements to the manuals, such as:

Sorting the rST files in the docs/ top level directory into the appropriate manuals; this might mean splitting them up if they cover material that should really be in different manuals
Convert .txt files in docs/ to rST; again, consider whether the material needs to be split so parts go in one manual and parts in another
Improving the quality of indexing and tables of contexts: the old Texinfo manual has an index, but the rST ones don't currently. rST has facilities for indexes; we should look into using them.
Generally wordsmithing, rearranging and improving the documentation would likely help a lot.

7: Blue sky ideas

We should definitely not look at anything in this section before we've dealt with the basics above :-)

One thing I've wondered from time to time is whether it would be useful to have documentation for board models as doc comments in the source files that implement them. Our current documentation is very frequently badly out of date on what board models we support and what the functionality and limitations for each board are. Perhaps doing this as a doc comment would encourage us to at least have a placeholder amount of documentation for each board?