Features/tcg-multithread: Difference between revisions

From QEMU
 
(41 intermediate revisions by 8 users not shown)
Line 1: Line 1:
=MultiThreaded support in the TCG=
This is the feature that allows the Tiny Code Generator run one host-thread per guest thread or guest vCPU (in system emulation mode). It was first introduced in QEMU [[ChangeLog/2.9|2.9]] for Alpha and ARM. Work to enable full multi-threading support in additional system emulations is on going.
===Improving TCG performance===


==OverView==
==Overview==


Qemu can currently emulate a number of CPU’s in parallel, but it does so in a single thread. Given many modern hosts are multi-core, and many targets equally use multiple cores, a significant performance advantage can be realised by making use of multiple host threads to simulate multiple target cores.
QEMU's system emulation mode could always emulate multiple vCPUs but it scheduled them in a single thread and executed each one in tern in a round-robin fashion. To switch to a host-thread per vCPU a number of changes had to be made to the core code as well as explicit support in each guest architecture. The design decisions are documented in {{src|path=docs/devel/multi-thread-tcg.txt}}.


This is work in progress - we expect to publish results on this wiki page as progress is made.
There was a talk at KVM Forum 2015 ([https://www.youtube.com/watch?v=KnSW0WjWHZI video] [http://www.linux-kvm.org/images/c/cf/02x02-Alex_Benee-Towards_Multithreaded_TCG.pdf slides]) which is a little out of date but acts as a useful primer on the challenges involved.


==Plan==
==Controlling MTTCG==
The TCG today is close to being thread safe, but there is still some concern that there are remaining issues. We will address this by first focusing on user-level TCG threads as this seems a straightforward target. Subsequently the wider case of system level multi-threading will be looked at. The following is an ''currently incomplete'' list of issues to address:


===Global TCG State===
Once a MTTCG guest is supported there should be no need to enable it explicitly. The system emulation will enable it if the following conditions are met:


Currently there is no protection against two threads attempting to generate code at the same time into the translation buffer. This means you do see corrupted code generation from time to time in multi-threaded apps. There are a couple of approaches we could take from adding locking to the code generator so only one thread at a time could generate code to having separate translation buffers for each thread of execution.
* The guest architecture has defined TARGET_SUPPORTS_MTTCG
* The host architectures TCG_TARGET_DEFAULT_MO supports TCG_GUEST_DEFAULT_MO


There are also a number of global variables and assumptions in the various back-ends which will need to audited. I suspect these values will need to be wrapped up in a portable TCGContext.
When this is not the case you can force MTTCG by specifying:


===Signal Handling===
    $QEMU $OPTS --accel tcg,thread=multi


There are two types of signal we need to handle. Synchronous (e.g. SIGBUS, SIGSEG) and Asynchronous (e.g. SIGSTOP, SIGINT, SIGUSR). While any signal can be sent asynchronously most of the common synchronous ones occur when there is an error in the translated code. As such rectifying machine state is fairly well tested. For Asynchronus signals there are a plethora of edge cases to deal with especially around the handling of signals with respect to system calls. If they arrive during translated code there behaviour is fairly easy to handle however when in QEMU's own code care has to be taken that syscalls respond correctly to the EINTR.
although you are likely to get strange behaviour. If you suspect that guest emulation is incorrect you can revert to single threaded mode and re-run your test:


==How to get involved==
    $QEMU $OPTS --accel tcg,thread=single
Right now, there is a small dedicated team looking at this issue. Those are:
   
* Fred Konrad
==Incompatibilities==
* Mark Burton
* Pavel Dovgaluk


If you would like to be involved, please use the qemu-devel mail list.
MTTCG is not compatible with -icount and enabling icount will force a single threaded run.


We will run phone conference calls as appropriate to co-ordinate activity and we will feed back to the main Qemu mail lists as progress is made.
==Developer Details==


===Porting a guest architecture===


==Other Work==
Before MTTCG can be enabled for a guest the following changes must be made.


This is the most important section initially, and we welcome any, and all comments and other work.
* Correctly translate atomic/exclusive instructions (see tcg_gen_atomic_)
If you know of any patch sets that may be of value, PLEASE let us know via the qemu-devel mail list.
* Ensure the translation step correctly handles barrier instructions (tcg_gen_mb)
* Define TCG_GUEST_DEFAULT_MO
* Audit instructions that modify system state
** generally this means taking BQL (e.g. HELPER(set_cp_reg))
* Audit MMU management functions
** cputlb provides an API for various tlb_flush_FOO operations
** updates to the guests page tables need to be atomic (e.g. dirty bits)
* Audit power/reset sequences
** see for example {{src|path=target/arm/arm-powerctl.c}}


===Proof of concept implementations===
The work queue API async_[safe_]run_on_cpu provides a mechanism for one vCPU to queue work on another.
Below are all the proof of concept implementations we have found thus far. It is highly likely that some of these patch sets can help us to reach an up-streamable solution. At the very least these provide some evidence that there is a performance improvement to be had.
 
    http://dl.acm.org/citation.cfm?id=2259030&CFID=454906387&CFTOKEN=60579010
Once this work is done your final patch can update configure and enable TARGET_SUPPORTS_MTTCG
    https://github.com/podinx/PQEMU
 
    http://www.cs.nthu.edu.tw/~ychung/conference/ICPADS2011.pdf
===Testing===
 
Ideally you'll want a comprehensive set of tests to exercise the corner cases of system emulation behaviour. See [https://github.com/stsquad/kvm-unit-tests/tree/mttcg/current-tests-v7 Alex's kvm-unit-tests] for an example of how the ARM architecture is exercised.
 
==Further Work==
 
* Enabling strong-on-weak memory consistency (e.g. emulate x86 on an ARM host)
 
==People==
 
Now MTTCG is merged it is supported by the TCG maintainers. However the following people where involved:
 
* Fred Konrad (Original core MTTCG patch set)
* Alex Bennée (ARM testing, base enabling tree)
* Alvise Rigo (LL/SC work)
* Emilio Cota (QHT, cmpxchg atomics)
 
==Other Reading==
 
* Emilio's slides for his CGO17 paper - http://www.cs.columbia.edu/~cota/pubs/cota_cgo17-slides.pdf
* Cross-ISA Machine Emulation for Multicores - http://www.cs.columbia.edu/~cota/pubs/cota_cgo17.pdf DOI:10.1109/CGO.2017.7863741
* Cross-ISA Machine Instrumentation using Fast and Scalable Dynamic Binary Translation - https://dl.acm.org/doi/pdf/10.1145/3313808.3313811

Latest revision as of 10:14, 4 August 2021

This is the feature that allows the Tiny Code Generator run one host-thread per guest thread or guest vCPU (in system emulation mode). It was first introduced in QEMU 2.9 for Alpha and ARM. Work to enable full multi-threading support in additional system emulations is on going.

Overview

QEMU's system emulation mode could always emulate multiple vCPUs but it scheduled them in a single thread and executed each one in tern in a round-robin fashion. To switch to a host-thread per vCPU a number of changes had to be made to the core code as well as explicit support in each guest architecture. The design decisions are documented in docs/devel/multi-thread-tcg.txt.

There was a talk at KVM Forum 2015 (video slides) which is a little out of date but acts as a useful primer on the challenges involved.

Controlling MTTCG

Once a MTTCG guest is supported there should be no need to enable it explicitly. The system emulation will enable it if the following conditions are met:

  • The guest architecture has defined TARGET_SUPPORTS_MTTCG
  • The host architectures TCG_TARGET_DEFAULT_MO supports TCG_GUEST_DEFAULT_MO

When this is not the case you can force MTTCG by specifying: 

   $QEMU $OPTS --accel tcg,thread=multi

although you are likely to get strange behaviour. If you suspect that guest emulation is incorrect you can revert to single threaded mode and re-run your test: 

   $QEMU $OPTS --accel tcg,thread=single

Incompatibilities

MTTCG is not compatible with -icount and enabling icount will force a single threaded run.

Developer Details

Porting a guest architecture

Before MTTCG can be enabled for a guest the following changes must be made.

  • Correctly translate atomic/exclusive instructions (see tcg_gen_atomic_)
  • Ensure the translation step correctly handles barrier instructions (tcg_gen_mb)
  • Define TCG_GUEST_DEFAULT_MO
  • Audit instructions that modify system state
    • generally this means taking BQL (e.g. HELPER(set_cp_reg))
  • Audit MMU management functions
    • cputlb provides an API for various tlb_flush_FOO operations
    • updates to the guests page tables need to be atomic (e.g. dirty bits)
  • Audit power/reset sequences

The work queue API async_[safe_]run_on_cpu provides a mechanism for one vCPU to queue work on another.

Once this work is done your final patch can update configure and enable TARGET_SUPPORTS_MTTCG

Testing

Ideally you'll want a comprehensive set of tests to exercise the corner cases of system emulation behaviour. See Alex's kvm-unit-tests for an example of how the ARM architecture is exercised.

Further Work

  • Enabling strong-on-weak memory consistency (e.g. emulate x86 on an ARM host)

People

Now MTTCG is merged it is supported by the TCG maintainers. However the following people where involved:

  • Fred Konrad (Original core MTTCG patch set)
  • Alex Bennée (ARM testing, base enabling tree)
  • Alvise Rigo (LL/SC work)
  • Emilio Cota (QHT, cmpxchg atomics)

Other Reading

Retrieved from "https://wiki.qemu.org/index.php?title=Features/tcg-multithread&oldid=10560"