Features/RDMALiveMigration: Difference between revisions
No edit summary |
No edit summary |
||
| Line 10: | Line 10: | ||
== design == | == design == | ||
# In order to provide maximum cross-device compatibility, we use the '''librdmacm''' library, which abstracts out the RDMA capabilities of each individual type of RDMA device, including infiniband, iWARP, as well as RoCE. This patch has been tested on both RoCE and infiniband devices from Mellanox. | |||
# A new file named "migration-rdma.c" contains the core code required to perform librdmacm connection establishment and the transfer of actual RDMA contents. | |||
# Files "arch_init.c" and "savevm.c" have been modified to transfer the VM's memory in the standard live migration path using RMDA memory instead of using TCP. | |||
# Currently, the XBZRLE capability and the detection of zero pages (dup_page()) significantly slow down the empircal throughput observed when RDMA is activated, so the code path skips these capabilities when RDMA is enabled. Hopefully, we can stop doing this in the future and come up with a way to preserve these capabilities simultaneously with the use of RDMA. | |||
== performance == | |||
[[File:Perf.png]] | [[File:Perf.png]] | ||
Revision as of 21:58, 11 February 2013
summary
Live migration using RDMA instead of TCP.
contact
- Name: Michael Hines
- Email: mrhines@us.ibm.com
description
Uses standard OFED software stack, which supports both RoCE and Infiniband.
design
- In order to provide maximum cross-device compatibility, we use the librdmacm library, which abstracts out the RDMA capabilities of each individual type of RDMA device, including infiniband, iWARP, as well as RoCE. This patch has been tested on both RoCE and infiniband devices from Mellanox.
- A new file named "migration-rdma.c" contains the core code required to perform librdmacm connection establishment and the transfer of actual RDMA contents.
- Files "arch_init.c" and "savevm.c" have been modified to transfer the VM's memory in the standard live migration path using RMDA memory instead of using TCP.
- Currently, the XBZRLE capability and the detection of zero pages (dup_page()) significantly slow down the empircal throughput observed when RDMA is activated, so the code path skips these capabilities when RDMA is enabled. Hopefully, we can stop doing this in the future and come up with a way to preserve these capabilities simultaneously with the use of RDMA.
performance
TODO
- Integrate with XOR-based run-length encoding (if possible)
- Stop skipping the zero-pages check