Features/COLO

Summary

COLO-COarse Grain LOck Stepping
Paper: academia paper in SOCC 2013

Virtual machine (VM) replication is a well known technique for providing application-agnostic software-implemented hardware fault tolerance "non-stop service". COLO is a high availability solution. Both primary VM (PVM) and secondary VM (SVM) run in parallel. They receive the same request from client, and generate response in parallel too. If the response packets from PVM and SVM are identical, they are released immediately. Otherwise, a VM checkpoint (on demand) is conducted. The idea is presented in Xen summit 2012, and 2013, and academia paper in SOCC 2013. It's also presented in KVM forum 2013: Kvm-forum-2013-COLO.pdf

There's also several RFC proposal posted to QEMU devel maillist:

http://lists.nongnu.org/archive/html/qemu-devel/2014-06/msg05567.html

http://lists.nongnu.org/archive/html/qemu-devel/2014-09/msg04459.html

Wiki: http://wiki.qemu.org/Features/COLO
Github:

https://github.com/coloft/qemuCheckout to latest colo branch

https://github.com/macrosheep/qemu

https://github.com/gao-feng/colo-proxy.git

* Copyright (c) 2015 HUAWEI TECHNOLOGIES CO.,LTD.
* Copyright (c) 2015 FUJITSU LIMITED
* Copyright (c) 2015 Intel Corporation

Components

• COLO Manager:
  • COLO Controller - Modifications of save/restore flow (Refer to MacroCheckpointing).
  • COLO Disk Manager - When primary VM writes data into image, the colo disk manger captures this data
    

and send it to secondary VM’s which makes sure the context of secondary VM's image is consentient with

the ontext of primary VM 's image.

• COLO Proxy

We need an module to compare the packets returned by Primary VM and Secondary VM

and decide whether to start a checkpoint according to some rules. It is a linux kernel module

for host.

Current Status

• COLO Manager:
  • COLO Controller/Frame (View on Github Checkout to latest colo branch, RFC patch has been posted(v3))
  • COLO Disk Manager (RFC patch has been posted)
• COLO Proxy(View on Github)

Failover & Heartbeat

We provide a qmp command:

colo_lost_heartbeat

This command will tell COLO that heartbeat is lost. COLO will do some action accordingly.

External heartbeat modules can use this qmp command to communicatewith COLO. Users can choose whatever heartbeat implementation they want.

Failover rule:
TODO

Block replication

This is the COLO Disk Manager implementation. Please refer to http://wiki.qemu.org/Features/BlockReplication

Installation

• Hardware requriements

There is at least one directly connected nic to forward the network requests from client to secondary vm. The directly connected nic must not be used by any other purpose. If your guest has more than one nic, you should have directly connected nic for each guest nic. If you don't have enouth directly connected nic, you can use vlan.

• Checkout the latest colo branch on Github.

/*for example, the latest brach is colo_v0.99, the version may rapidly grows, please make sure you are using the latest branch*/
# git clone https://github.com/macrosheep/qemu
# git checkout colo_v0.99

• configure and make:

# ./configure --target-list=x86_64-softmmu --enable-colo
# make

• Install COLO agent module:
  • Download COLO agent, and compile it, it is the kernel module, so copy the module to the directory /lib/modules/<version>/updates/, and run depmod.

# git clone https://github.com/wencongyang/colo-agent
# cd colo-agent; make
# cp *.ko /lib/modules/<version>/updates/
# depmod

Running COLO

• just like QEMU's normal migration, run 2 QEMU VM:
  • Add the following net parameters to start qemu for nic replication on both Primary and Secondary QEMU:

-netdev tap,id=hn0,colo_script=./qemu-colo/network-colo,colo_nicname=eth0,vhost=on -device virtio-net-pci,id=net-pci0,netdev=hn0
/* colo_nicname is the directly connected nic which will foward request to secondary host. */

• • Start Primary VM
  • Start Secondary VM with option:

-incoming tcp:[IP]:[PORT]

• on Primary VM's QEMU monitor, run following command:

   migrate_set_capability colo on
   migrate tcp:[IP]:[PORT]

• Some hints from Dr. David Alan Gilbert(dgilbert#redhat.com) for network replication:

  1) Each host has a physical interface (em4)
  2) and a bridge (brpair) that has em4 connected to it
     Note I've explicitly set the bridge address on each host to
     the MAC of the physical interface using
     ip link set dev brpair address 
  3) Each host has a Vlan alias em4.8 created with 
     ip link add link em4 name em4.8 type vlan id 8
     ip link set dev em4.8 up

     it's the em4.8 that carries the colo network comparison traffic.

  4) Each host has a tap device:
     ip tuntap add dev pairtap mode tap
     ip link set dev pairtap address 52:54:00:61:0f:46 up   # note not actually guests mac
     brctl addif brpair pairtap    # Not on secondary

  5) ensure that colo-tc is in your path, and start the qemu's
     
      try/bin/qemu-system-x86_64 -machine pc-i440fx-2.2,accel=kvm -m 4096 -smp 4 -nographic -drive id=image,file=/home/vms/f20.qcow2,cache=none -netdev tap,id=hn0,ifname=pairtap,colo_script=./qemu/network-colo,colo_nicname=em4.8,script=no,downscript=no -device virtio-net-pci,id=net-pci0,mac=52:54:00:61:0f:45,netdev=hn0

     (on secondary add the -incoming tcp::4444)

  6) then from primary ctrl-a c to get prompt and do
     migrate_set_speed 100G
     migrate_set_capability colo on
     migrate -d tcp:otherhost:4444

• For Questions/Issues, please contact: Yang Hongyang <yanghy#cn.fujitsu.com>; Wen Congyang <wency#cn.fujitsu.com>

Links

• COLO on Xen