Features/COLO/Manual HOWTO

Overview

This explains how to manually use qemu colo via qmp commands. For automatically managed colo, see Features/COLO/Managed HOWTO

Test environment prepare

• Qemu colo
  • Current COLO have been merged by upstream, so you just need use the official Qemu repo: https://github.com/qemu/qemu

# cd qemu
# ./configure --target-list=x86_64-softmmu
# make -j4

• Set Up the Bridge and network environment

In Primary, setup a bridge br0, using command brctl, like:
# ifconfig eth0 down
# ifconfig eth0 0.0.0.0
# brctl addbr br0
# brctl addif br0 eth0
# ifconfig br0 192.168.0.33 netmask 255.255.255.0
# ifconfig eth0 up
# echo "allow br0" > /etc/qemu/bridge.conf
In Secondary, setup br0 like Primary side

Test steps

Note: Here we are running both instances on the same Machine for testing, change the IP Addresses if you want to run it on two Hosts

• (1) Startup qemu
• Primary side:

# imagefolder="/mnt/vms/colo-test"

# cp --reflink=auto $imagefolder/primary.qcow2 $imagefolder/primary-copy.qcow2

# qemu-system-x86_64 -enable-kvm -cpu qemu64,+kvmclock -m 2048 -qmp stdio \
   -vnc :0 -device piix3-usb-uhci -device usb-tablet -name primary \
   -netdev tap,id=hn0,vhost=off,helper=/usr/lib/qemu/qemu-bridge-helper \
   -device rtl8139,id=e0,netdev=hn0 \
   -chardev socket,id=mirror0,host=127.0.0.1,port=9003,server,nowait \
   -chardev socket,id=compare1,host=127.0.0.1,port=9004,server,wait \
   -chardev socket,id=compare0,host=127.0.0.1,port=9001,server,nowait \
   -chardev socket,id=compare0-0,host=127.0.0.1,port=9001 \
   -chardev socket,id=compare_out,host=127.0.0.1,port=9005,server,nowait \
   -chardev socket,id=compare_out0,host=127.0.0.1,port=9005 \
   -object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0 \
   -object filter-redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out \
   -object filter-redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0 \
   -object iothread,id=iothread1 \
   -object colo-compare,id=comp0,primary_in=compare0-0,secondary_in=compare1,\
outdev=compare_out0,iothread=iothread1 \
   -drive if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,\
children.0.file.filename=$imagefolder/primary.qcow2,children.0.driver=qcow2 -S

• Secondary side:

# imagefolder="/mnt/vms/colo-test"

# qemu-img create -f qcow2 $imagefolder/secondary-active.qcow2 10G

# qemu-img create -f qcow2 $imagefolder/secondary-hidden.qcow2 10G

# qemu-system-x86_64 -enable-kvm -cpu qemu64,+kvmclock -m 2048 -qmp stdio \
   -vnc :1 -device piix3-usb-uhci -device usb-tablet -name secondary \
   -netdev tap,id=hn0,vhost=off,helper=/usr/lib/qemu/qemu-bridge-helper \
   -device rtl8139,id=e0,netdev=hn0 \
   -chardev socket,id=red0,host=127.0.0.1,port=9003,reconnect=1 \
   -chardev socket,id=red1,host=127.0.0.1,port=9004,reconnect=1 \
   -object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0 \
   -object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1 \
   -object filter-rewriter,id=rew0,netdev=hn0,queue=all \
   -drive if=none,id=colo-disk0,file.filename=$imagefolder/primary-copy.qcow2,driver=qcow2,\
node-name=node0 \
   -drive if=ide,id=active-disk0,driver=replication,mode=secondary,file.driver=qcow2,\
top-id=active-disk0,file.file.filename=$imagefolder/secondary-active.qcow2,\
file.backing.driver=qcow2,file.backing.file.filename=$imagefolder/secondary-hidden.qcow2,\
file.backing.backing=colo-disk0 \
   -incoming tcp:0:9998

• (2) On Secondary VM's QEMU monitor, issue command

{'execute':'qmp_capabilities'}
{'execute': 'nbd-server-start', 'arguments': {'addr': {'type': 'inet', 'data': {'host': '127.0.0.1', 'port': '9999'} } } }
{'execute': 'nbd-server-add', 'arguments': {'device': 'colo-disk0', 'writable': true } }
{'execute': 'trace-event-set-state', 'arguments': {'name': 'colo*', 'enable': true} }

 Note:
 a. The qmp command nbd-server-start and nbd-server-add must be run before running the qmp command migrate on primary QEMU
 b. Active disk, hidden disk and nbd target's length should be the same.
 c. It is better to put active disk and hidden disk in ramdisk.

• (3) On Primary VM's QEMU monitor, issue command:

{'execute':'qmp_capabilities'}
{'execute': 'human-monitor-command', 'arguments': {'command-line': 'drive_add -n buddy driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.1,file.port=9999,file.export=colo-disk0,node-name=node0'}}
{'execute': 'x-blockdev-change', 'arguments':{'parent': 'colo-disk0', 'node': 'node0' } }
{'execute': 'migrate-set-capabilities', 'arguments': {'capabilities': [ {'capability': 'x-colo', 'state': true } ] } }
{'execute': 'migrate', 'arguments': {'uri': 'tcp:127.0.0.1:9998' } }

 Note:
 a. There should be only one NBD Client for each primary disk.
 b. The qmp command line must be run after running qmp command line in secondary qemu

• (4) Done
  • You will see two runing VMs, whenever you make changes to PVM, SVM will be synced.

• (5) Failover test
  • You can kill one of the VMs and Failover on the surviving VM:

Primary: Remove the nbd child from the quorum, remove the filters and notify COLO to failover:

{'execute': 'x-blockdev-change', 'arguments': {'parent': 'colo-disk0', 'child': 'children.1'}}
{'execute': 'object-del', 'arguments':{ 'id': 'iothread1' }}
{'execute': 'object-del', 'arguments':{ 'id': 'comp0' }}
{'execute': 'object-del', 'arguments':{ 'id': 'm0' }}
{'execute': 'object-del', 'arguments':{ 'id': 'redire0' }}
{'execute': 'object-del', 'arguments':{ 'id': 'redire1' }}
{'execute': 'chardev-remove', 'arguments':{ 'id': 'compare0' }}
{'execute': 'chardev-remove', 'arguments':{ 'id': 'compare0-0' }}
{'execute': 'chardev-remove', 'arguments':{ 'id': 'compare1' }}
{'execute': 'chardev-remove', 'arguments':{ 'id': 'compare_out' }}
{'execute': 'chardev-remove', 'arguments':{ 'id': 'compare_out0' }}
{'execute': 'chardev-remove', 'arguments':{ 'id': 'mirror0' }}
{'execute': 'x-colo-lost-heartbeat' }

Secondary: The primary host is down, so we should do the following thing:

{'execute': 'nbd-server-stop'}
{'execute': 'x-colo-lost-heartbeat'}