Atomic Snapshots of Multiple Devices

The snapshot_blkdev/blockdev-snapshot-sync command in QEMU 1.0 performs snapshots one device at a time, even if a guest has multiple devices. This can be troublesome in the instance of a snapshot failure. Should a snapshot fail, qemu will revert back to the original backing store but will still leave the guest in an overall inconsistent state, with some devices snapshotted and some not.

For instance, let us assume there are three devices in a guest: virtio0, virtio1, and virtio2. If we successfully perform a snapshot on virtio0 and virtio1, yet virtio2 fails, we could be in an inconsistent state. While we will have reverted virtio2 back to the previous backing store, virtio0 and virtio1 will have already successfully gone through the snapshot.

The only solution here is to stop the machine completely while the snapshots are performed. But ideally there would be a mechanism to allow all devices to have a snapshot taken as one atomic unit, so that for the snapshot to be successfully performed, all devices must have success.

QEMU 1.1 implements a "transaction" QMP command that operates on multiple block devices atomically. The transaction command receives one or more "transactionable" QMP commands and their arguments; the only transactionable command for now is blockdev-snapshot-sync. Execution of the commands is then split into two phases, a prepare phase and a commit/rollback phase. Should any command fail the prepare phase, the transaction immediately proceeds to roll back the completed prepare phases. If all commands are prepared successfully they are committed; the commit phase cannot fail, so that atomicity is achieved.

The transaction command is implemented using QAPI unions (discriminated records). Given the schema for a transactionable command, such as the following:

{ 'command': 'blockdev-snapshot-sync',
  'data': { 'device': 'str', 'snapshot-file': 'str', '*format': 'str' } }

a corresponding type is created and added to a union:

{ 'type': 'BlockdevSnapshot',
  'data': { 'device': 'str', 'snapshot-file': 'str', '*format': 'str' } }

{ 'union': 'BlockdevAction',
  'data': { 'blockdev-snapshot-sync': 'BlockdevSnapshot', /* ... */ } }

The transaction command then takes an array of actions:

{ 'command': 'transaction',
  'data': { 'actions': [ 'BlockdevAction' ] } }

Here is a sample execution of the command to snapshot two disks:

{ "execute": "transaction", "arguments":
  {'actions': [
    { 'type': 'blockdev-snapshot-sync', 'data' :
      { 'device': 'virtio0', 'snapshot-file': 'hd0-snap.qcow2' } },
    { 'type': 'blockdev-snapshot-sync', 'data' :
      { 'device': 'virtio1', 'snapshot-file': 'hd1-snap.qcow2' } } ] } }

Application to live block copy

Another feature that is new in QEMU 1.1 is live block device streaming. This feature lets guest retrieve data from a backing file while the guest is running; it enables quick provisioning of new virtual machines using shared remote storage, and lets the guest transition incrementally to fast local storage.

Streaming a block device to another location is also useful if management needs to migrate a guest's storage, for example in case of impending disk failures. However, in this context block streaming's fundamental deficiency is that the copy operation is performed while the virtual machine is already using the new storage; it is not possible to abort it and fall back to the old storage.

Luckily, storage migration is a simple extension of streaming. The block layer needs to be instructed to mirror writes to both the old and the new storage while streaming is in effect. Then, management can switch to the new storage at an arbitrary point after streaming is completed.

Unlike snapshotting, neither the start of block streaming, nor the "release" of old storage need to be done atomically across multiple devices. However, if the old storage has to be snapshotted at the time mirroring is started, then these two operations have to be done atomically.

Leaving aside for a moment the release operation, there are two possible implementation choices for an atomic snapshot+mirror operation. One is to specify both the snapshot destination and the mirror target, as in the following hypothetical QAPI schema:

{ 'command': drive-mirror',
  'data': { 'device': 'str', 'target': 'str', '*target-format': 'str',
            '*snapshot-file': 'str', '*snapshot-format': 'str' } }

This interface is simple to implement but it has two disadvantages. First, the interface is complicated. libvirt and oVirt right now need to do the above snapshot+mirror process because they want to copy storage outside QEMU; however, the additional arguments are there for everyone, even for people that can use block device streaming to do the copy. Second, the implementation must ensure a complete rollback of the snapshot operation in case mirroring fails. This is relatively complex to do; in fact, up to QEMU 1.0 blockdev-snapshot-sync couldn't even rollback correctly a single snapshot.

The latter requirement suggests plugging the drive-mirror command in the transaction command. The snapshot and mirror operations can be simply placed in the same transaction, which guarantees their atomicity. The schema then becomes:

{ 'command': drive-mirror',
  'data': { 'device': 'str', 'target': 'str', '*format': 'str' } }
{ 'type': 'BlockdevMirror',
  'data': { 'device': 'str', 'target': 'str', '*format': 'str' } }
{ 'union': 'BlockdevAction',
  'data': { 'blockdev-snapshot-sync': 'BlockdevSnapshot',
            'drive-mirror': 'BlockdevMirror' } }

and a sample execution of the command is as follows:

{ "execute": "transaction", "arguments":
  {'actions': [
    { 'type': 'blockdev-snapshot-sync', 'data' :
      { 'device': 'ide0-hd0', 'snapshot-file': 'base.qcow2' } },
    { 'type': 'drive-mirror', 'data' :
      { 'device': 'ide0-hd0', 'target': 'mirror.qcow2' } } ] } }

Switching the device to the new storage at the end of the copy operation is handled with another QMP command, drive-reopen. This command is not transactionable, so it is not included in BlockdevAction:

{ 'command': 'drive-reopen',
  'data': { 'device': 'str', 'new-image-file': 'str', '*format': 'str' } }

Image creation modes

Compared to the above definitions, QEMU 1.1 also introduces a mode argument to the blockdev-snapshot-sync and drive-mirror commands. The argument applies both to standalone command and to transactions. Its type is the NewImageMode enum:

{ 'enum': 'NewImageMode'
  'data': [ 'existing', 'absolute-paths', 'no-backing-file' ] }

The argument controls how QEMU creates the new image file:

existing directs QEMU to look for an existing image. The image must be on disk and should have the same contents as the disk that is currently attached to the virtual machine.
absolute-paths directs QEMU to create an image whose backing file is the current image. The current image is identified by an absolute path in the new image.
no-backing-file directs QEMU to create an image with no backing file at all. This is useful when the mirror target is a raw file, for example.

In the future, it is planned to have another mode, relative-paths. It will also create an image whose backing file is the current image, but the current image will be identified by a relative path in the new image.

Image creation occurs in the prepare phase and uses the mode argument; however, the new backing file chain is composed in the commit phase with no regard to the mode. This matters when the same disk is included twice in a transaction, as in the following example:

{ "execute": "transaction", "arguments":
  {'actions': [
    { 'type': 'blockdev-snapshot-sync', 'data' :
      { 'device': 'virtio0', 'snapshot-file': 'hd0-snap0.qcow2' } },
    { 'type': 'blockdev-snapshot-sync', 'data' :
      { 'device': 'virtio0', 'snapshot-file': 'hd0-snap1.qcow2' } } ] } }

Assuming virtio0 is associated to hd0-base.qcow2, the backing file chain at the end of the transaction will be hd0-base.qcow2 <- hd0-snap0.qcow2 <- hd0-snap1.qcow2. However, the hd0-snap1.qcow2 image file will point to hd0-base.qcow2. This is useful when doing a combined snapshot+mirror operation:

{ "execute": "transaction", "arguments":
  {'actions': [
    { 'type': 'blockdev-snapshot-sync', 'data' :
      { 'device': 'virtio0', 'snapshot-file': 'src/hd0-snap.qcow2' } },
    { 'type': 'drive-mirror', 'data' :
      { 'device': 'virtio0', 'target': 'dest/hd0-snap.qcow2' } } ] } }

Here, assume the backing storage is shared/hd0-base.qcow2. Mirroring will write to src/hd0-snap.qcow2 and dest/hd0-snap.qcow2 as expected, and dest/hd0-snap.qcow2 will point to the original storage. As soon as block streaming completes, management can switch the device to dest/hd0-snap.qcow2. src/hd0-snap.qcow2 is not part of the backing file chain anymore, and can be deleted.

Credits

Jeff Cody implemented atomic snapshots
Marcelo Tosatti implemented mirroring
Federico Simoncelli provided the first implementation of the mirroring commands
Eric Blake and Kevin Wolf participated to the discussions and suggested using discriminated unions
Paolo Bonzini provided a transaction-capable implementation of mirroring