Introduction

Postcopy recovery allows the postcopy migration to be interrupted, and also the migration can be resumed at any time when the migration channel is ready again (by either fixing up the broken channel, or provide a new channel).

Below is an example of how to use the postcopy recovery feature using QMP commands.

Example Commands

QMP handshakes

This needs to be run on both sides. It initialize the QMP channels.

 {"execute": "qmp_capabilities", "arguments": { "enable": [ "oob" ] } }
 {"return": {}}

Out-of-band messages are required on destination host for postcopy recovery otherwise we may face potential lockups on the destination node.

Enable postcopy

This needs to be run on both sides. It enables postcopy on both sides of the VMs.

 {"execute": "migrate-set-capabilities", "arguments": {"capabilities": [{"state": true, "capability": "postcopy-ram"}]}}
 {"return": {}}

Start precopy migration

This needs to be run on source side only. It starts the general precopy migration.

 {"execute": "migrate", "arguments": {"uri": "unix:/tmp/migration-test-12K6jV/migsocket"}}
 {"return": {}}

Start postcopy migration

This needs to be run on source side only. It switches the current precopy migration to postcopy migration so that the destination VM can start to run without migrating all the pages.

 {"execute": "migrate-start-postcopy"}
 {"return": {}}
 {"timestamp": {"seconds": 1559730429, "microseconds": 585575}, "event": "STOP"}
 {"timestamp": {"seconds": 1559730429, "microseconds": 613592}, "event": "RESUME"}

Break the migration channel

You can try to unplug the wire of the migration channel to emulate an interrupt of migration. Or you can use the "migrate-pause" command to emulate from teh software layer. The command only needs to be run on source side only.

 {"execute": "migrate-pause"}
 {"return": {}}

Check migration status

This can be run on both sides. After the migration was interrupted, you should see that both sides of VM went into "postcopy-paused" state.

 {"execute": "query-migrate"}
 {"return": {"status": "postcopy-paused", "socket-address": [{"path": "/tmp/migration-test-12K6jV/migsocket", "type": "unix"}]}}

Resume the interrupted postcopy migration

This should only be run on destination side. It rebuilds a migration channel. When using this command, you can either use the previous listening port if the network recovered. Or, we can simply use a new migration channel that can continue the migration. Here a new channel is used.

Note that here we used "exec-oob" rather than "execute" to queue an out-of-band message, so that the command will be executed in the isolated iothread channel.

 {"exec-oob": "migrate-recover", "arguments": {"uri": "unix:/tmp/migration-test-12K6jV/migsocket-recover"}, "id": "recover-cmd"}
 {"timestamp": {"seconds": 1559730429, "microseconds": 620072}, "event": "MIGRATION", "data": {"status": "setup"}}
 {"return": {}, "id": "recover-cmd"}

Then to resume the migration, we need to run this on source side. Note that we should use the new channel rather than the old one if it's not the same:

 {"execute": "migrate", "arguments": {"resume": true, "uri": "unix:/tmp/migration-test-12K6jV/migsocket-recover"}}
 {"return": {}}

Wait until migration completes

Check on both sides that the migration can be completed normally.

 {"execute": "query-migrate"}
 {"return": {"postcopy-blocktime": 89, "status": "completed", "postcopy-vcpu-blocktime": [90]}}

Note that the postcopy migration can be interrupted by many times, we can resume the migration using the same steps described above until the migration completes.

TBD

When postcopy recover triggers, we could lost the current postcopy queue, e.g., even after the recovery completes, the blocked threads on destination will still hang until the source accidentally send these pages. What we can do in the future is we always maintain a list of pages that the destination threads are requesting. We should remove the entries on the list when the page arrived. Then with such a list, after postcopy recovery happens, we can re-send these pages to source to make sure all the starved threads on destination will get the missing pages asap.