Features/COLO: Difference between revisions

From QEMU
No edit summary
Line 172: Line 172:
:[https://github.com/zhangckid/qemu colo-proxy branch]
:[https://github.com/zhangckid/qemu colo-proxy branch]


= Failover command =
We provide a qmp command:
<pre>x-colo-lost-heartbeat</pre>
This command will tell COLO that heartbeat is lost. COLO will do some action accordingly. External heartbeat modules can use this qmp command to communicate with COLO. Users can choose whatever heartbeat implementation they want.
Failover rule:<br>
TODO:


= How to setup/test COLO =
= How to setup/test COLO =
=== '''Hardware requirements''' ===
There is at least one directly connected nic to forward the network requests from client to secondary vm.
The directly connected nic must not be used by any other purpose. you should have directly connected nic for
each guest nic. If you don't have enouth directly connected nic, you can use vlan.
=== '''Network link topology''' ===
<pre>
=================================normal ======================================
                                +--------+
                                |client  |
        master                +----+---+                    slave
-------------------------+          |            + -------------------------+
  PVM                  |          +            |                          |
+-------+        +----[eth0]-----[switch]-----[eth0]---------+              |
|guest  |    +---+-+    |                        |      +---+-+            |
|    [tap0]--+ br0 |    |                        |      | br0 |            |
|      |    +-----+  [eth1]-----[forward]----[eth1]--+  +-----+    SVM    |
+-------+                |                        |    |            +-------+|
                        |                        |    |  +-----+  | guest ||
                      [eth2]---[checkpoint]---[eth2]  +--+br1  |-[tap0]    ||
                        |                        |      +-----+  |      ||
                        |                        |                +-------+|
-------------------------+                        +--------------------------+
e.g.
master:
br0: 192.168.0.33
eth1: 192.168.1.33
eth2: 192.168.2.33
slave:
br0: 192.168.0.88
br1: no ip address
eth1: 192.168.1.88
eth2: 192.168.2.88
NOTE: br0 is setup by adminitrator but br1 is setup by colo now.
</pre>
<pre>
===========================after failover=====================================
                                +--------+
                                |client  |
    master (dead)              +----+---+                slave (alive)
-------------------------+          |            ---------------------------+
  PVM                    |          +            |                          |
+-------+        +----[eth0]-----[switch]-----[eth0]-------+                |
|guest  |    +---+-+    |                        |    +---+-+              |
|    [tap0]--+ br0 |    |                        |    | br0 +--+          |
|      |    +-----+  [eth1]-----[forward]----[eth1]  +-----+  |    SVM  |
+-------+                |                        |              |  +-------+|
                        |                        |    +-----+  |  | guest ||
                      [eth2]---[checkpoint]---[eth2]  |br1  |  +[tap0]    ||
                        |                        |    +-----+    |      ||
                        |                        |                +-------+|
-------------------------+                        +--------------------------+
</pre>


=== '''Test environment prepare''' ===
=== '''Test environment prepare''' ===
* Prepare host kernel
:colo-proxy kernel module need cooperate with linux kernel.
:You should patch [https://github.com/coloft/colo-proxy/blob/master/patch4kernel/0001-colo-patch-for-kernel.patch colo-patch-for-kernel.patch] into kernel codes,
:then compile kernel and intall the new kernel (Recommend [https://www.kernel.org/ kernel-3.18.10])
* Proxy module
** proxy module is used for network packets compare.
<pre>
# git clone https://github.com/coloft/colo-proxy.git
# cd ./colo-proxy
# make
# make install
</pre>
* Modified [https://github.com/coloft/iptables.git iptables]
** We have add a new rule to iptables command.
<pre>
# git clone https://github.com/coloft/iptables.git
# cd ./iptables
# git checkout colo
# ./autogen.sh && ./configure
# make && make install
</pre>
*Modified arptables
** Please get the latest [http://git.netfilter.org/arptables arptables] and then compile and install
* Qemu colo
* Qemu colo
** Checkout the latest colo branch from [https://github.com/coloft/qemu/tree/colo-v1.5-basic colo-v1.5-basic] or
** Checkout the latest COLO branch from [https://github.com/zhangckid/qemu/tree/colo-v2.7-proxy-mode-compare-and-rewriter-aug16 COLO-full]
***[https://github.com/coloft/qemu/tree/colo-v1.5-developing colo-v1.5-developing(More features)]
<pre>
<pre>
# cd qemu
# cd qemu
# ./configure --target-list=x86_64-softmmu --enable-colo
# ./configure --target-list=x86_64-softmmu --enable-colo --enable-gcrypt --enable-replication
# make
# make -j
</pre>
</pre>


Line 312: Line 223:


=== '''Test steps''' ===
=== '''Test steps''' ===
('''''Note: We apply two scripts to help completing step (1) ~ step (2), [https://github.com/coloft/colo-test/blob/master/primary-colo.sh primary-colo.sh] [https://github.com/coloft/colo-test/blob/master/secondary-colo.sh secondary-colo.sh])'''''
*(1) Load modeule
<pre>
# modprobe xt_PMYCOLO (For slave side, modprobe xt_SECCOLO)
# modprobe nf_conntrack_colo (Other colo module will be automatically loaded by
script colo-proxy-script.sh)
# modprobe xt_mark
# modprobe kvm-intel
</pre>


*(2) Startup qemu
*(1) Startup qemu


*''Master side:''
*''Primary side:''
<pre>
<pre>
# x86_64-softmmu/qemu-system-x86_64 -machine pc-i440fx-2.3,accel=kvm,usb=off \
# x86_64-softmmu/qemu-system-x86_64 -enable-kvm -boot c -m 2048 -smp 2 -qmp stdio -vnc :7 -name primary -cpu qemu64,+kvmclock -device piix3-usb-uhci -drive if=virtio,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,children.0.file.filename=/mnt/sdd/pure_IMG/linux/redhat/rhel_6.5_64_2U_ide,children.0.driver=raw -S -netdev tap,id=hn0,vhost=off,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown -device e1000,id=e0,netdev=hn0,mac=52:a4:00:12:78:66 -chardev socket,id=mirror0,host=9.61.1.8,port=9003,server,nowait -chardev socket,id=compare1,host=9.61.1.8,port=9004,server,nowait -chardev socket,id=compare0,host=9.61.1.8,port=9001,server,nowait -chardev socket,id=compare0-0,host=9.61.1.8,port=9001 -chardev socket,id=compare_out,host=9.61.1.8,port=9005,server,nowait -chardev socket,id=compare_out0,host=9.61.1.8,port=9005 -object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0 -object filter-redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out -object filter-r!
-netdev tap,id=hn0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown,\
edirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0 -object colo-compare,id=comp0,primary_in=compare0-0,secondary_in=compare1,outdev=compare_out0
colo_script=./scripts/colo-proxy-script.sh,forward_nic=eth1 -device virtio-net-pci,id=net-pci0,netdev=hn0 \
-boot c -drive if=virtio,id=disk1,driver=quorum,read-pattern=fifo,cache=none,aio=native,\
children.0.file.filename=/mnt/sdb/pure_IMG/redhat/redhat-7.0.img,children.0.driver=raw \
-vnc :7 -m 2048 -smp 2 -device piix3-usb-uhci -device usb-tablet -monitor stdio -S
</pre>
</pre>



Revision as of 07:03, 1 September 2016

Feature Name

COarse-grained LOck-stepping Virtual Machines for Non-stop Service

Background

Virtual machine (VM) replication is a well known technique for providing application-agnostic software-implemented hardware fault tolerance "non-stop service". COLO is a high availability solution. Both primary VM (PVM) and secondary VM (SVM) run in parallel. They receive the same request from client, and generate response in parallel too. If the response packets from PVM and SVM are identical, they are released immediately. Otherwise, a VM checkpoint (on demand) is conducted.
The idea is presented in Xen summit 2012, and 2013, and academia paper in SOCC 2013. It's also presented in KVM forum 2013,KVM forum 2015,KVM forum2016

Feature authors

  • Name: Hailiang Zhang, Congyang Wen, Chen Zhang
  • Email: zhang.zhanghailiang@huawei.com, wency@cn.fujitsu.com, zhangchen.fnst@fujitsu.com

Architecture

The architecture of COLO is shown in the bellow diagram. It consists of a pair of networked physical nodes: The primary node running the PVM, and the secondary node running the SVM to maintain a valid replica of the PVM. PVM and SVM execute in parallel and generate output of response packets for client requests according to the application semantics.

The incoming packets from the client or external network are received by the primary node, and then forwarded to the secondary node, so that Both the PVM and the SVM are stimulated with the same requests.

COLO receives the outbound packets from both the PVM and SVM and compares them before allowing the output to be sent to clients.

The SVM is qualified as a valid replica of the PVM, as long as it generates identical responses to all client requests. Once the differences in the outputs are detected between the PVM and SVM, COLO withholds transmission of the outbound packets until it has successfully synchronized the PVM state to the SVM.

  Primary Node                                                            Secondary Node
+------------+  +-----------------------+       +------------------------+  +------------+
|            |  |       HeartBeat       |<----->|       HeartBeat        |  |            |
| Primary VM |  +-----------|-----------+       +-----------|------------+  |Secondary VM|
|            |              |                               |               |            |
|            |  +-----------|-----------+       +-----------|------------+  |            |
|            |  |QEMU   +---v----+      |       |QEMU  +----v---+        |  |            |
|            |  |       |Failover|      |       |      |Failover|        |  |            |
|            |  |       +--------+      |       |      +--------+        |  |            |
|            |  |   +---------------+   |       |   +---------------+    |  |            |
|            |  |   | VM Checkpoint |-------------->| VM Checkpoint |    |  |            |
|            |  |   +---------------+   |       |   +---------------+    |  |            |
|            |  |                       |       |                        |  |            |
|Requests<---------------------------^------------------------------------------>Requests|
|Responses----------------------\ /--|--------------\  /------------------------Responses|
|            |  |               | |  |  |       |   |  |                 |  |            |
|            |  | +-----------+ | |  |  |       |   |  |  +------------+ |  |            |
|            |  | | COLO disk | | |  |  |       |   |  |  | COLO disk  | |  |            |
|            |  | |   Manager |-|-|--|--------------|--|->| Manager    | |  |            |
|            |  | +|----------+ | |  |  |       |   |  |  +-----------|+ |  |            |
|            |  |  |            | |  |  |       |   |  |              |  |  |            |
+------------+  +--|------------|-|--|--+       +---|--|--------------|--+  +------------+
                   |            | |  |              |  |              |
+-------------+    | +----------v-v--|--+       +---|--v-----------+  |    +-------------+
|  VM Monitor |    | |  COLO Proxy      |       |    COLO Proxy    |  |    | VM Monitor  |
|             |    | |(compare packet)  |       | (adjust sequence)|  |    |             |
+-------------+    | +----------|----^--+       +------------------+  |    +-------------+
                   |            |    |                                |
+------------------|------------|----|--+       +---------------------|------------------+
|   Kernel         |            |    |  |       |   Kernel            |                  |
+------------------|------------|----|--+       +---------------------|------------------+
                   |            |    |                                |
    +--------------v+  +--------v----|--+       +------------------+ +v-------------+
    |   Storage     |  |External Network|       | External Network | |   Storage    |
    +---------------+  +----------------+       +------------------+ +--------------+

Components introduction

  • HeartBeat
Runs on both the primary and secondary nodes, to periodically check platform
availability. When the primary node suffers a hardware fail-stop failure,
the heartbeat stops responding, the secondary node will trigger a failover
as soon as it determines the absence.
When primary VM writes data into image, the colo disk manger captures this data
and send it to secondary VM’s which makes sure the context of secondary VM's image is consentient with
the context of primary VM 's image.
The following is the image of block replication workflow:
       +----------------------+            +------------------------+
       |Primary Write Requests|            |Secondary Write Requests|
       +----------------------+            +------------------------+
                 |                                       |
                 |                                      (4)
                 |                                       V
                 |                              /-------------\
                 |      Copy and Forward        |             |
                 |---------(1)----------+       | Disk Buffer |
                 |                      |       |             |
                 |                     (3)      \-------------/
                 |                 speculative      ^
                 |                write through    (2)
                 |                      |           |
                 V                      V           |
          +--------------+           +----------------+
          | Primary Disk |           | Secondary Disk |
          +--------------+           +----------------+
   1) Primary write requests will be copied and forwarded to Secondary
      QEMU.
   2) Before Primary write requests are written to Secondary disk, the
      original sector content will be read from Secondary disk and
      buffered in the Disk buffer, but it will not overwrite the existing
      sector content (it could be from either "Secondary Write Requests" or
      previous COW of "Primary Write Requests") in the Disk buffer.
   3) Primary write requests will be written to Secondary disk.
   4) Secondary write requests will be buffered in the Disk buffer and it
      will overwrite the existing sector content in the buffer.
  • COLO framework: COLO Checkpoint/Failover Controller
Modifications of save/restore flow to realize continuous migration, to make sure the state of VM in Secondary side
always be consistent with VM in Primary side.
  • COLO Proxy:
Delivers packets to Primary and Seconday, and then compare the reponse from both side. Then decide whether to
start a checkpoint according to some rules.
 Primary qemu                                                           Secondary qemu
+--------------------------------------------------------------+       +----------------------------------------------------------------+
| +----------------------------------------------------------+ |       |  +-----------------------------------------------------------+ |
| |                                                          | |       |  |                                                           | |
| |                        guest                             | |       |  |                        guest                              | |
| |                                                          | |       |  |                                                           | |
| +-------^--------------------------+-----------------------+ |       |  +---------------------+--------+----------------------------+ |
|         |                          |                         |       |                        ^        |                              |
|         |                          |                         |       |                        |        |                              |
|         |  +------------------------------------------------------+  |                        |        |                              |
|netfilter|  |                       |                         |    |  |   netfilter            |        |                              |
| +----------+ +----------------------------+                  |    |  |  +-----------------------------------------------------------+ |
| |       |  |                       |      |        out       |    |  |  |                     |        |  filter excute order       | |
| |       |  |          +-----------------------------+        |    |  |  |                     |        | +------------------->      | |
| |       |  |          |            |      |         |        |    |  |  |                     |        |   TCP                      | |
| | +-----+--+-+  +-----v----+ +-----v----+ |pri +----+----+sec|    |  |  | +------------+  +---+----+---v+rewriter++  +------------+ | |
| | |          |  |          | |          | |in  |         |in |    |  |  | |            |  |        |              |  |            | | |
| | |  filter  |  |  filter  | |  filter  +------>  colo   <------+ +-------->  filter   +--> adjust |   adjust     +-->   filter   | | |
| | |  mirror  |  |redirector| |redirector| |    | compare |   |  |    |  | | redirector |  | ack    |   seq        |  | redirector | | |
| | |          |  |          | |          | |    |         |   |  |    |  | |            |  |        |              |  |            | | |
| | +----^-----+  +----+-----+ +----------+ |    +---------+   |  |    |  | +------------+  +--------+--------------+  +---+--------+ | |
| |      |   tx        |   rx           rx  |                  |  |    |  |            tx                        all       |  rx      | |
| |      |             |                    |                  |  |    |  +-----------------------------------------------------------+ |
| |      |             +--------------+     |                  |  |    |                                                   |            |
| |      |   filter excute order      |     |                  |  |    |                                                   |            |
| |      |  +---------------->        |     |                  |  +--------------------------------------------------------+            |
| +-----------------------------------------+                  |       |                                                                |
|        |                            |                        |       |                                                                |
+--------------------------------------------------------------+       +----------------------------------------------------------------+
         |guest receive               | guest send
         |                            |
+--------+----------------------------v------------------------+
|                                                              |                          NOTE: filter direction is rx/tx/all
|                         tap                                  |                          rx:receive packets sent to the netdev
|                                                              |                          tx:receive packets sent by the netdev
+--------------------------------------------------------------+
In COLO-compare, we do packet comparing job.
Packets coming from the primary char indev will be sent to outdev.
Packets coming from the secondary char dev will be dropped after comparing.
colo-comapre need two input chardev and one output chardev: primary_in=chardev1-id (source: primary send packet) secondary_in=chardev2-id (source: secondary send packet) outdev=chardev3-id
  • Note:
a. HeartBeat is not been realized, so you need to trigger failover process by using 'x-colo-lost-heartbeat' command.
b. COLO proxy compents is work-in-process, it only support periodic checkpoint mode now, just as Micro-checkpointing.

Current Status

Github:(Checkout to latest colo branch)

COLO replication branch
COLO framework branch
colo-proxy branch


How to setup/test COLO

Test environment prepare

  • Qemu colo
    • Checkout the latest COLO branch from COLO-full
# cd qemu
# ./configure --target-list=x86_64-softmmu --enable-colo --enable-gcrypt --enable-replication
# make -j
  • Set Up the Bridge and network environment
    • You must setup you network environment like above picture(Network link topology Normal).
In master, setup a bridge br0, using command brctl, like:
# ifconfig eth0 down
# ifconfig eth0 0.0.0.0
# brctl addbr br0
# brctl addif br0 eth0
# ifconfig br0 192.168.0.33 netmask 255.255.255.0
# ifconfig eth0 up
In slave, setup br0 like master side
  • Qemu-ifup/Qemu-ifdown
    • We need a script to bring up the TAP interface.
    • a qemu-ifdown script is needed to reset you networking configuration which is configured by qemu-ifup script
You can find this info from http://en.wikibooks.org/wiki/QEMU/Networking.
NOTE: Don't forget to change this script file permission to be executable

Master:
root@master# cat /etc/qemu-ifup
#!/bin/sh
switch=br0
if [ -n "$1" ]; then
         ip link set $1 up
         brctl addif ${switch} $1
fi
root@master# cat /etc/qemu-ifdown
!/bin/sh
switch=br0
if [ -n "$1" ]; then
        brctl delif ${switch} $1
fi
Slave:
like Master side

Test steps

  • (1) Startup qemu
  • Primary side:
# x86_64-softmmu/qemu-system-x86_64 -enable-kvm -boot c -m 2048 -smp 2 -qmp stdio -vnc :7 -name primary -cpu qemu64,+kvmclock -device piix3-usb-uhci -drive if=virtio,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,children.0.file.filename=/mnt/sdd/pure_IMG/linux/redhat/rhel_6.5_64_2U_ide,children.0.driver=raw -S -netdev tap,id=hn0,vhost=off,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown -device e1000,id=e0,netdev=hn0,mac=52:a4:00:12:78:66 -chardev socket,id=mirror0,host=9.61.1.8,port=9003,server,nowait -chardev socket,id=compare1,host=9.61.1.8,port=9004,server,nowait -chardev socket,id=compare0,host=9.61.1.8,port=9001,server,nowait -chardev socket,id=compare0-0,host=9.61.1.8,port=9001 -chardev socket,id=compare_out,host=9.61.1.8,port=9005,server,nowait -chardev socket,id=compare_out0,host=9.61.1.8,port=9005 -object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0 -object filter-redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out -object filter-r!
edirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0 -object colo-compare,id=comp0,primary_in=compare0-0,secondary_in=compare1,outdev=compare_out0
  • Slave side:
# qemu-img create -f qcow2 /mnt/ramfs/active_disk.img 10G

# qemu-img create -f qcow2 /mnt/ramfs/hidden_disk.img 10G

# x86_64-softmmu/qemu-system-x86_64 -machine pc-i440fx-2.3,accel=kvm,usb=off \
-netdev tap,id=hn0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown,\
colo_script=./scripts/colo-proxy-script.sh,forward_nic=eth1 -device virtio-net-pci,id=net-pci0,netdev=hn0 \
-drive if=none,driver=raw,file=/mnt/sdb/pure_IMG/redhat/redhat-7.0.img,id=colo1,cache=none,aio=native \
-drive if=virtio,driver=replication,mode=secondary,throttling.bps-total-max=70000000,\
file.file.filename=/mnt/ramfs/active_disk.img,file.driver=qcow2,\
file.backing.file.filename=/mnt/ramfs/hidden_disk.img,\
file.backing.driver=qcow2,\
file.backing.backing.backing_reference=colo1,\
file.backing.allow-write-backing-file=on \
-vnc :7 -m 2048 -smp 2 -device piix3-usb-uhci -device usb-tablet -monitor stdio -incoming tcp:0:8888
 Note:
 1. The export name in secondary QEMU command line is the secondary disk's id. In the expampe, it is colo1.
 2. The export name for primary disk is not in the command line. It is in child_add's argument. Please see below.
 3. The export name for the same disk must be the same
 4. The qmp command nbd-server-start and nbd-server-add must be run before running the qmp command migrate on primary QEMU
 5. Don't use nbd-server-start's other options
 6. Active disk, hidden disk and nbd target's length should be the same.
 7. It is better to put active disk and hidden disk in ramdisk.
 8. It is all a single argument to -drive, and you should ignore the leading whitespace.
  • (3) On Secondary VM's QEMU monitor, issue command (This command must be run before command in step (4))
(qemu) nbd_server_start 192.168.2.88:8889
(qemu) nbd_server_add -w colo1
  • (4) On Primary VM's QEMU monitor, issue command:
(qemu) child_add disk1 child.driver=replication,child.mode=primary,child.file.host=192.168.2.88,child.file.port=8889,child.file.export=colo1,child.file.driver=nbd,child.ignore-errors=on
(qemu) migrate_set_capability colo on
(qemu) migrate tcp:192.168.2.88:8888
 Note:
 1. There should be only one NBD Client.
 2. host is the secondary physical machine's hostname or IP
 3. Each disk must have its own export name.
  • (5) Done
    • You will see two runing VMs, whenever you make changes to PVM, SVM will be synced.
  • (6) Failover test
    • You can kill SVM (PVM) and run 'colo_lost_heartbeat' in PVM's (SVM's) monitor at the same time, then PVM (SVM) will failover and client will not feel this change.
  • For Questions/Issues, please contact: Zhang Hailiang <zhang.zhanghailiang@huawei.com>;Yang Hongyang <yanghy@cn.fujitsu.com>; Wen Congyang <wency@cn.fujitsu.com>

Links