Features/COLO: Difference between revisions

From QEMU
No edit summary
(Add my contact info)
 
(30 intermediate revisions by 3 users not shown)
Line 10: Line 10:


= Feature authors =
= Feature authors =
* '''Name:''' Hailiang Zhang, Congyang Wen, Chen Zhang
* '''Name:''' Chen Zhang, Lukas Straub, Hailiang Zhang, Congyang Wen, Yong Wang, Guang Wang
* ''' Email:''' zhang.zhanghailiang@huawei.com, wency@cn.fujitsu.com, zhangchen.fnst@fujitsu.com
* ''' Email:''' zhangckid@gmail.com/chen.zhang@intel.com, lukasstraub2@web.de, zhang.zhanghailiang@huawei.com, wency@cn.fujitsu.com, wang.yongD@h3c.com, wang.guanga@h3c.com


= Architecture =
= Architecture =
Line 33: Line 33:
outbound packets until it has successfully synchronized the PVM state to the SVM.
outbound packets until it has successfully synchronized the PVM state to the SVM.


  Primary Node                                                           Secondary Node
  Primary Node                                                             Secondary Node
  +------------+  +-----------------------+      +------------------------+  +------------+
  +------------+  +-----------------------+      +------------------------+  +------------+
  |            |  |      HeartBeat      |<----->|       HeartBeat        |  |            |
  |            |  |      HeartBeat      +<----->+       HeartBeat        |  |            |
  | Primary VM |  +-----------|-----------+      +-----------|------------+  |Secondary VM|
  | Primary VM |  +-----------+-----------+      +-----------+------------+  |Secondary VM|
  |            |              |                              |              |            |
  |            |              |                              |              |            |
  |            |  +-----------|-----------+      +-----------|------------+  |            |
  |            |  +-----------|-----------+      +-----------|------------+  |            |
Line 43: Line 43:
  |            |  |      +--------+      |      |      +--------+        |  |            |
  |            |  |      +--------+      |      |      +--------+        |  |            |
  |            |  |  +---------------+  |      |  +---------------+    |  |            |
  |            |  |  +---------------+  |      |  +---------------+    |  |            |
  |            |  |  | VM Checkpoint |-------------->| VM Checkpoint |    |  |            |
  |            |  |  | VM Checkpoint +-------------->+ VM Checkpoint |    |  |            |
  |            |  |  +---------------+  |      |  +---------------+    |  |            |
  |            |  |  +---------------+  |      |  +---------------+    |  |            |
|            |  |                      |      |                        |  |            |
  |Requests<--------------------------\ /-----------------\ /--------------------->Requests|
  |Requests<---------------------------^------------------------------------------>Requests|
|            |  |                  ^ ^ |      |      | |              |  |            |
  |Responses----------------------\ /--|--------------\ /------------------------Responses|
  |Responses+---------------------\ /-|-|------------\ /-------------------------+Responses|
  |            |  |              | | | |      |   |  |                 |  |            |
  |            |  |              | | | | |      | | |  | |              |  |            |
  |            |  | +-----------+ | | | |      |   |  | +------------+ |  |            |
  |            |  | +-----------+ | | | | |      | | |  | | +----------+ |  |            |
  |            |  | | COLO disk | | | | |      |   |  | | COLO disk | |  |            |
  |            |  | | COLO disk | | | | | |      | | |  | | | COLO disk| |  |            |
  |            |  | |  Manager |-|-|--|--------------|--|->| Manager   | |  |            |
  |            |  | |  Manager +---------------------------->| Manager | |  |            |
  |            |  | +|----------+ | |  |  |       |   |  |  +-----------|+ |  |            |
  |            |  | ++----------+ v v | | |      |  | v  v | +---------++ |  |           |
  |            |  |  |            | |  |  |      |   |  |             |  |  |            |
|           |  |  |+-----------+-+-+-++|       | ++-+--+-+---------+ |  |  |            |
  +------------+ +--|------------|-|--|--+       +---|--|--------------|--+  +------------+
  |            |  | ||  COLO Proxy    ||      | |  COLO Proxy    | |  |  |            |
                    |            | |  |              |  |              |
  |            | |  || (compare packet ||      | |(adjust sequence | |  |  |            |
+-------------+    | +----------v-v--|--+      +---|--v-----------+ |   +-------------+
|            |  |  ||and mirror packet)||      | |    and ACK)    | |  |  |            |
  | VM Monitor |   | | COLO Proxy      |       |   COLO Proxy    |  |    | VM Monitor |
  |            |  |  |+------------+---^-+|      | +-----------------+ |  |  |            |
|            |    | |(compare packet)  |      | (adjust sequence)|  |    |            |
+------------+  +-----------------------+       +------------------------+  +------------+
  +-------------+   | +----------|----^--+      +------------------+  |    +-------------+
+------------+     |            |  |                                |     +------------+
                    |            |    |                                |
  | VM Monitor |     |             |   |                               |     | VM Monitor |
+------------------|------------|----|--+      +---------------------|------------------+
  +------------+     |            |  |                               |     +------------+
  |  Kernel        |           |   |  |      |  Kernel            |                  |
  +---------------------------------------+      +----------------------------------------+
  +------------------|------------|----|--+      +---------------------|------------------+
  |  Kernel        |             |   |  |      |  Kernel            |                  |
                     |           |   |                                |
  +---------------------------------------+      +----------------------------------------+
     +--------------v+  +--------v----|--+      +------------------+ +v-------------+
                     |             |   |                                |
     +--------------v+  +---------v---+--+      +------------------+ +v-------------+
     |  Storage    |  |External Network|      | External Network | |  Storage    |
     |  Storage    |  |External Network|      | External Network | |  Storage    |
     +---------------+  +----------------+      +------------------+ +--------------+
     +---------------+  +----------------+      +------------------+ +--------------+
= Components introduction =
= Components introduction =


* HeartBeat
* HeartBeat (Not yet implemented)
: Runs on both the primary and secondary nodes, to periodically check platform
: Runs on both the primary and secondary nodes, to periodically check platform
: availability. When the primary node suffers a hardware fail-stop failure,
: availability. When the primary node suffers a hardware fail-stop failure,
Line 160: Line 162:


*'''Note:
*'''Note:
: a. HeartBeat is not been realized, so you need to trigger failover process by using 'x-colo-lost-heartbeat' command.
: HeartBeat is not been realized, so you need to trigger failover process by using 'x-colo-lost-heartbeat' command.
: b. COLO proxy compents is work-in-process, it only support periodic checkpoint mode now, just as Micro-checkpointing.


= Current Status =
= Current Status =
Github:(Checkout to latest colo branch)
Have been merged in Qemu upstream (v4.0).
:[https://github.com/Pating/qemu COLO replication branch]
:[https://github.com/coloft/qemu COLO framework branch]
:[https://github.com/zhangckid/qemu colo-proxy branch]


= How to setup/test COLO =
= How to setup/test COLO =
For automatically managed colo: [[Features/COLO/Managed HOWTO]]


=== '''Test environment prepare''' ===
For manual use of colo via qmp commands: [[Features/COLO/Manual HOWTO]]
* Qemu colo
** Checkout the latest COLO branch from [https://github.com/zhangckid/qemu/tree/colo-v2.7-proxy-mode-compare-and-rewriter-aug16 COLO-full]
<pre>
# cd qemu
# ./configure --target-list=x86_64-softmmu --enable-colo --enable-gcrypt --enable-replication
# make -j
</pre>


* Set Up the Bridge and network environment
= Heartbeat Service =
** You must setup you network environment like above picture('''Network link topology''' Normal).
COLO support internal and external heartbeat service, they both use same qemu interface(qmp or monitor commands).
For internal service, we introduce new Qemu module named Advanced Watch Dog for this job.
Advanced Watch Dog is an universal monitoring module on VMM side, it can be used to detect network down(VMM to guest, VMM to VMM, VMM to another remote server) and do previously set operation.
If you want to use it, please add "--enable-awd" when configure qemu.
In primary node:
<pre>
<pre>
In master, setup a bridge br0, using command brctl, like:
-monitor tcp::4445,server,nowait
# ifconfig eth0 down
-chardev socket,id=h1,host=3.3.3.3,port=9009,server,nowait
# ifconfig eth0 0.0.0.0
-chardev socket,id=heartbeat0,host=3.3.3.3,port=4445
# brctl addbr br0
-object iothread,id=iothread1
# brctl addif br0 eth0
-object advanced-watchdog,id=heart1,server=on,awd_node=h1,notification_node=heartbeat0,
# ifconfig br0 192.168.0.33 netmask 255.255.255.0
opt_script=colo_primary_opt_script,iothread=iothread1,pulse_interval=1000,timeout=5000
# ifconfig eth0 up
In slave, setup br0 like master side
</pre>
</pre>


* Qemu-ifup/Qemu-ifdown
colo_primary_opt_script:
** We need a script to bring up the TAP interface.
** a qemu-ifdown script is needed to reset you networking configuration which is configured by qemu-ifup script
:You can find this info from [http://en.wikibooks.org/wiki/QEMU/Networking http://en.wikibooks.org/wiki/QEMU/Networking].
<pre>
<pre>
NOTE: Don't forget to change this script file permission to be executable
x_colo_lost_heartbeat
 
Master:
root@master# cat /etc/qemu-ifup
#!/bin/sh
switch=br0
if [ -n "$1" ]; then
        ip link set $1 up
        brctl addif ${switch} $1
fi
root@master# cat /etc/qemu-ifdown
!/bin/sh
switch=br0
if [ -n "$1" ]; then
        brctl delif ${switch} $1
fi
Slave:
like Master side
</pre>
</pre>


=== '''Test steps''' ===
In secondary node:
'''Note: Here the primary side host ip is 9.61.1.8, secondary side host ip is 9.61.1.7.'''
*(1) Startup qemu
*''Primary side:''
<pre>
# x86_64-softmmu/qemu-system-x86_64 -enable-kvm -boot c -m 2048 -smp 2 -qmp stdio -vnc :7 -name primary -cpu qemu64,+kvmclock -device piix3-usb-uhci \
  -drive if=virtio,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,\
          children.0.file.filename=/mnt/sdd/pure_IMG/linux/redhat/rhel_6.5_64_2U_ide,children.0.driver=raw -S
  -netdev tap,id=hn0,vhost=off,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
  -device e1000,id=e0,netdev=hn0,mac=52:a4:00:12:78:66 \
  -chardev socket,id=mirror0,host=9.61.1.8,port=9003,server,nowait -chardev socket,id=compare1,host=9.61.1.8,port=9004,server,nowait \
  -chardev socket,id=compare0,host=9.61.1.8,port=9001,server,nowait -chardev socket,id=compare0-0,host=9.61.1.8,port=9001 \
  -chardev socket,id=compare_out,host=9.61.1.8,port=9005,server,nowait \
  -chardev socket,id=compare_out0,host=9.61.1.8,port=9005 \
  -object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0 \
  -object filter-redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out -object filter-redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0 \
  -object colo-compare,id=comp0,primary_in=compare0-0,secondary_in=compare1,outdev=compare_out0
</pre>


*'' Secondary side:''
<pre>
<pre>
# qemu-img create -f qcow2 /mnt/ramfs/active_disk.img 10G
-monitor tcp::4445,server,nowait
 
-chardev socket,id=h1,host=3.3.3.3,port=9009,reconnect=1
# qemu-img create -f qcow2 /mnt/ramfs/hidden_disk.img 10G
-chardev socket,id=heart1,host=3.3.3.8,port=4445
 
-object iothread,id=iothread1
# x86_64-softmmu/qemu-system-x86_64 -boot c -m 2048 -smp 2 -qmp stdio -vnc :7 -name secondary -enable-kvm -cpu qemu64,+kvmclock -device piix3-usb-uhci \
-object advanced-watchdog,id=heart1,server=off,awd_node=h1,notification_node=he
  -drive if=none,id=colo-disk0,file.filename=/mnt/sdd/pure_IMG/linux/redhat/rhel_6.5_64_2U_ide,driver=raw,node-name=node0 \
art1,opt_script=colo_secondary_opt_script,iothread=iothread1,timeout=10000
  -drive if=virtio,id=active-disk0,driver=replication,mode=secondary,file.driver=qcow2,top-id=active-disk0, \
        file.file.filename=/mnt/ramfs/active_disk.img,file.backing.driver=qcow2,\
        file.backing.file.filename=/mnt/ramfs/hidden_disk.img,\
        file.backing.backing=colo-disk0  \
  -netdev tap,id=hn0,vhost=off,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \
  -device e1000,netdev=hn0,mac=52:a4:00:12:78:66 -chardev socket,id=red0,host=9.61.1.8,port=9003 \
  -chardev socket,id=red1,host=9.61.1.8,port=9004 \
  -object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0 \
  -object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1 \
  -object filter-rewriter,id=rew0,netdev=hn0,queue=all -incoming tcp:0:8888
 
</pre>
</pre>


*(2) On Secondary VM's QEMU monitor, issue command''
colo_secondary_opt_script:
<pre>
<pre>
{'execute':'qmp_capabilities'}
nbd_server_stop
{ 'execute': 'nbd-server-start',
x_colo_lost_heartbeat
  'arguments': {'addr': {'type': 'inet', 'data': {'host': '9.61.1.7', 'port': '8889'} } }
}
{'execute': 'nbd-server-add', 'arguments': {'device': 'colo-disk0', 'writable': true } }
</pre>
</pre>
  '''''Note:'''''
This parts of code still under review.
  ''a. The qmp command nbd-server-start and nbd-server-add must be run before running the qmp command migrate on primary QEMU''
https://github.com/zhangckid/qemu/tree/colo-with-awd19dec1
  ''b. Active disk, hidden disk and nbd target's length should be the same.''
  ''c. It is better to put active disk and hidden disk in ramdisk.''


*(3) On Primary VM's QEMU monitor, issue command:''
For external service, user can write service and policy itself, it detect network down, just need notify COLO service.
<pre>
{'execute':'qmp_capabilities'}
{ 'execute': 'human-monitor-command',
  'arguments': {'command-line': 'drive_add -n buddy driver=replication,mode=primary,file.driver=nbd,file.host=9.61.1.7,file.port=8889,file.export=colo-disk0,node-name=node0'}}
{ 'execute':'x-blockdev-change', 'arguments':{'parent': 'colo-disk0', 'node': 'node0' } }
{ 'execute': 'migrate-set-capabilities',
      'arguments': {'capabilities': [ {'capability': 'x-colo', 'state': true } ] } }
{ 'execute': 'migrate', 'arguments': {'uri': 'tcp:xx.xx.xx.xx:8888' } }
</pre>
  '''''Note:'''''
  ''a. There should be only one NBD Client for each primary disk.''
  ''b. The qmp command line must be run after running qmp command line in secondary qemu''


*(5) Done
= Supporting documentation =
** You will see two runing VMs, whenever you make changes to PVM, SVM will be synced.
The idea is presented in Xen summit 2012, 2013 and [https://static.sched.com/hosted_files/xensummit19/d0/Application%20Agnostic%20High%20Availability%20Solution%20On%20Hypervisor%20Level%20-%20V1.4.pdf 2019]
 
and [http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 academia paper in SOCC 2013].
*(6) Failover test
**You can kill Primary VM and run 'x_colo_lost_heartbeat' in Secondary VM's monitor at the same time, then SVM will failover and client will not detect this change.
**Before issuing '{ "execute": "x-colo-lost-heartbeat" }' command, we have to issue block related command to stop block replication.
 
Primary: Remove the nbd child from the quorum:
  { 'execute': 'x-blockdev-change', 'arguments': {'parent': 'colo-disk0', 'child': 'children.1'}}
  { 'execute': 'human-monitor-command','arguments': {'command-line': 'drive_del blk-buddy0'}}
  Note: there is no qmp command to remove the blockdev now
 
Secondary: The primary host is down, so we should do the following thing:
  { 'execute': 'nbd-server-stop' }


= Supporting documentation =
It is also presented in  
The idea is presented in Xen summit 2012, and 2013,
[https://www.linux-kvm.org/images/1/1d/Kvm-forum-2013-COLO.pdf KVM forum 2013], [http://www.linux-kvm.org/images/0/01/01x07-Hongyang_Yang-Status_update_on_KVM-COLO.pdf KVM forum 2015] and [http://www.linux-kvm.org/images/a/af/03x08B-Hailang_Zhang-Status_Update_on_KVM-COLO_FT.pdf KVM forum2016]
and [http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 academia paper in SOCC 2013]. It's also presented in  
[http://www.linux-kvm.org/wiki/images/1/1d/Kvm-forum-2013-COLO.pdf KVM forum 2013],[http://www.linux-kvm.org/images/0/01/01x07-Hongyang_Yang-Status_update_on_KVM-COLO.pdf KVM forum 2015],[http://www.linux-kvm.org/images/a/af/03x08B-Hailang_Zhang-Status_Update_on_KVM-COLO_FT.pdf KVM forum2016]<br>


= Future works =
= Future works =

Latest revision as of 18:38, 30 January 2023

Feature Name

COarse-grained LOck-stepping Virtual Machines for Non-stop Service

Background

Virtual machine (VM) replication is a well known technique for providing application-agnostic software-implemented hardware fault tolerance "non-stop service". COLO is a high availability solution. Both primary VM (PVM) and secondary VM (SVM) run in parallel. They receive the same request from client, and generate response in parallel too. If the response packets from PVM and SVM are identical, they are released immediately. Otherwise, a VM checkpoint (on demand) is conducted.


Feature authors

  • Name: Chen Zhang, Lukas Straub, Hailiang Zhang, Congyang Wen, Yong Wang, Guang Wang
  • Email: zhangckid@gmail.com/chen.zhang@intel.com, lukasstraub2@web.de, zhang.zhanghailiang@huawei.com, wency@cn.fujitsu.com, wang.yongD@h3c.com, wang.guanga@h3c.com

Architecture

The architecture of COLO is shown in the bellow diagram. It consists of a pair of networked physical nodes: The primary node running the PVM, and the secondary node running the SVM to maintain a valid replica of the PVM. PVM and SVM execute in parallel and generate output of response packets for client requests according to the application semantics.

The incoming packets from the client or external network are received by the primary node, and then forwarded to the secondary node, so that Both the PVM and the SVM are stimulated with the same requests.

COLO receives the outbound packets from both the PVM and SVM and compares them before allowing the output to be sent to clients.

The SVM is qualified as a valid replica of the PVM, as long as it generates identical responses to all client requests. Once the differences in the outputs are detected between the PVM and SVM, COLO withholds transmission of the outbound packets until it has successfully synchronized the PVM state to the SVM. 

 Primary Node                                                              Secondary Node
+------------+  +-----------------------+       +------------------------+  +------------+
|            |  |       HeartBeat       +<----->+       HeartBeat        |  |            |
| Primary VM |  +-----------+-----------+       +-----------+------------+  |Secondary VM|
|            |              |                               |               |            |
|            |  +-----------|-----------+       +-----------|------------+  |            |
|            |  |QEMU   +---v----+      |       |QEMU  +----v---+        |  |            |
|            |  |       |Failover|      |       |      |Failover|        |  |            |
|            |  |       +--------+      |       |      +--------+        |  |            |
|            |  |   +---------------+   |       |   +---------------+    |  |            |
|            |  |   | VM Checkpoint +-------------->+ VM Checkpoint |    |  |            |
|            |  |   +---------------+   |       |   +---------------+    |  |            |
|Requests<--------------------------\ /-----------------\ /--------------------->Requests|
|            |  |                   ^ ^ |       |       | |              |  |            |
|Responses+---------------------\ /-|-|------------\ /-------------------------+Responses|
|            |  |               | | | | |       |  | |  | |              |  |            |
|            |  | +-----------+ | | | | |       |  | |  | | +----------+ |  |            |
|            |  | | COLO disk | | | | | |       |  | |  | | | COLO disk| |  |            |
|            |  | |   Manager +---------------------------->| Manager  | |  |            |
|            |  | ++----------+ v v | | |       |  | v  v | +---------++ |  |            |
|            |  |  |+-----------+-+-+-++|       | ++-+--+-+---------+ |  |  |            |
|            |  |  ||   COLO Proxy     ||       | |   COLO Proxy    | |  |  |            |
|            |  |  || (compare packet  ||       | |(adjust sequence | |  |  |            |
|            |  |  ||and mirror packet)||       | |    and ACK)     | |  |  |            |
|            |  |  |+------------+---^-+|       | +-----------------+ |  |  |            |
+------------+  +-----------------------+       +------------------------+  +------------+
+------------+     |             |   |                                |     +------------+
| VM Monitor |     |             |   |                                |     | VM Monitor |
+------------+     |             |   |                                |     +------------+
+---------------------------------------+       +----------------------------------------+
|   Kernel         |             |   |  |       |   Kernel            |                  |
+---------------------------------------+       +----------------------------------------+
                   |             |   |                                |
    +--------------v+  +---------v---+--+       +------------------+ +v-------------+
    |   Storage     |  |External Network|       | External Network | |   Storage    |
    +---------------+  +----------------+       +------------------+ +--------------+

Components introduction

  • HeartBeat (Not yet implemented)
Runs on both the primary and secondary nodes, to periodically check platform
availability. When the primary node suffers a hardware fail-stop failure,
the heartbeat stops responding, the secondary node will trigger a failover
as soon as it determines the absence.
When primary VM writes data into image, the colo disk manger captures this data
and send it to secondary VM’s which makes sure the context of secondary VM's image is consentient with
the context of primary VM 's image.
The following is the image of block replication workflow:
       +----------------------+            +------------------------+
       |Primary Write Requests|            |Secondary Write Requests|
       +----------------------+            +------------------------+
                 |                                       |
                 |                                      (4)
                 |                                       V
                 |                              /-------------\
                 |      Copy and Forward        |             |
                 |---------(1)----------+       | Disk Buffer |
                 |                      |       |             |
                 |                     (3)      \-------------/
                 |                 speculative      ^
                 |                write through    (2)
                 |                      |           |
                 V                      V           |
          +--------------+           +----------------+
          | Primary Disk |           | Secondary Disk |
          +--------------+           +----------------+

   1) Primary write requests will be copied and forwarded to Secondary
      QEMU.
   2) Before Primary write requests are written to Secondary disk, the
      original sector content will be read from Secondary disk and
      buffered in the Disk buffer, but it will not overwrite the existing
      sector content (it could be from either "Secondary Write Requests" or
      previous COW of "Primary Write Requests") in the Disk buffer.
   3) Primary write requests will be written to Secondary disk.
   4) Secondary write requests will be buffered in the Disk buffer and it
      will overwrite the existing sector content in the buffer.
  • COLO framework: COLO Checkpoint/Failover Controller
Modifications of save/restore flow to realize continuous migration, to make sure the state of VM in Secondary side
always be consistent with VM in Primary side.
  • COLO Proxy:
Delivers packets to Primary and Seconday, and then compare the reponse from both side. Then decide whether to
start a checkpoint according to some rules.
 Primary qemu                                                           Secondary qemu
+--------------------------------------------------------------+       +----------------------------------------------------------------+
| +----------------------------------------------------------+ |       |  +-----------------------------------------------------------+ |
| |                                                          | |       |  |                                                           | |
| |                        guest                             | |       |  |                        guest                              | |
| |                                                          | |       |  |                                                           | |
| +-------^--------------------------+-----------------------+ |       |  +---------------------+--------+----------------------------+ |
|         |                          |                         |       |                        ^        |                              |
|         |                          |                         |       |                        |        |                              |
|         |  +------------------------------------------------------+  |                        |        |                              |
|netfilter|  |                       |                         |    |  |   netfilter            |        |                              |
| +----------+ +----------------------------+                  |    |  |  +-----------------------------------------------------------+ |
| |       |  |                       |      |        out       |    |  |  |                     |        |  filter excute order       | |
| |       |  |          +-----------------------------+        |    |  |  |                     |        | +------------------->      | |
| |       |  |          |            |      |         |        |    |  |  |                     |        |   TCP                      | |
| | +-----+--+-+  +-----v----+ +-----v----+ |pri +----+----+sec|    |  |  | +------------+  +---+----+---v+rewriter++  +------------+ | |
| | |          |  |          | |          | |in  |         |in |    |  |  | |            |  |        |              |  |            | | |
| | |  filter  |  |  filter  | |  filter  +------>  colo   <------+ +-------->  filter   +--> adjust |   adjust     +-->   filter   | | |
| | |  mirror  |  |redirector| |redirector| |    | compare |   |  |    |  | | redirector |  | ack    |   seq        |  | redirector | | |
| | |          |  |          | |          | |    |         |   |  |    |  | |            |  |        |              |  |            | | |
| | +----^-----+  +----+-----+ +----------+ |    +---------+   |  |    |  | +------------+  +--------+--------------+  +---+--------+ | |
| |      |   tx        |   rx           rx  |                  |  |    |  |            tx                        all       |  rx      | |
| |      |             |                    |                  |  |    |  +-----------------------------------------------------------+ |
| |      |             +--------------+     |                  |  |    |                                                   |            |
| |      |   filter excute order      |     |                  |  |    |                                                   |            |
| |      |  +---------------->        |     |                  |  +--------------------------------------------------------+            |
| +-----------------------------------------+                  |       |                                                                |
|        |                            |                        |       |                                                                |
+--------------------------------------------------------------+       +----------------------------------------------------------------+
         |guest receive               | guest send
         |                            |
+--------+----------------------------v------------------------+
|                                                              |                          NOTE: filter direction is rx/tx/all
|                         tap                                  |                          rx:receive packets sent to the netdev
|                                                              |                          tx:receive packets sent by the netdev
+--------------------------------------------------------------+
In COLO-compare, we do packet comparing job.
Packets coming from the primary char indev will be sent to outdev.
Packets coming from the secondary char dev will be dropped after comparing.
colo-comapre need two input chardev and one output chardev: primary_in=chardev1-id (source: primary send packet) secondary_in=chardev2-id (source: secondary send packet) outdev=chardev3-id
  • Note:
HeartBeat is not been realized, so you need to trigger failover process by using 'x-colo-lost-heartbeat' command.

Current Status

Have been merged in Qemu upstream (v4.0).

How to setup/test COLO

For automatically managed colo: Features/COLO/Managed HOWTO

For manual use of colo via qmp commands: Features/COLO/Manual HOWTO

Heartbeat Service

COLO support internal and external heartbeat service, they both use same qemu interface(qmp or monitor commands). For internal service, we introduce new Qemu module named Advanced Watch Dog for this job. Advanced Watch Dog is an universal monitoring module on VMM side, it can be used to detect network down(VMM to guest, VMM to VMM, VMM to another remote server) and do previously set operation. If you want to use it, please add "--enable-awd" when configure qemu. In primary node: 

-monitor tcp::4445,server,nowait
-chardev socket,id=h1,host=3.3.3.3,port=9009,server,nowait
-chardev socket,id=heartbeat0,host=3.3.3.3,port=4445
-object iothread,id=iothread1
-object advanced-watchdog,id=heart1,server=on,awd_node=h1,notification_node=heartbeat0,
opt_script=colo_primary_opt_script,iothread=iothread1,pulse_interval=1000,timeout=5000

colo_primary_opt_script: 

x_colo_lost_heartbeat

In secondary node: 

-monitor tcp::4445,server,nowait
-chardev socket,id=h1,host=3.3.3.3,port=9009,reconnect=1
-chardev socket,id=heart1,host=3.3.3.8,port=4445
-object iothread,id=iothread1
-object advanced-watchdog,id=heart1,server=off,awd_node=h1,notification_node=he
art1,opt_script=colo_secondary_opt_script,iothread=iothread1,timeout=10000

colo_secondary_opt_script: 

nbd_server_stop
x_colo_lost_heartbeat

This parts of code still under review. https://github.com/zhangckid/qemu/tree/colo-with-awd19dec1

For external service, user can write service and policy itself, it detect network down, just need notify COLO service.

Supporting documentation

The idea is presented in Xen summit 2012, 2013 and 2019 and academia paper in SOCC 2013.

It is also presented in KVM forum 2013, KVM forum 2015 and KVM forum2016

Future works

1. Support continuously VM replication.
2. Support shared storage.
3. Develop the heartbeat part.
4. Reduce checkpoint VM’s downtime while do checkpoint.

Links

Retrieved from "https://wiki.qemu.org/index.php?title=Features/COLO&oldid=11091"