Features/FVD/Engineering

FVD was designed and engineered rigorously. Following a systematic methodology, testing and debugging facilities were built into FVD from day one to automatically catch even the most elusive bugs.

• On the observation of the lack of a method for testing race conditions (I/O order and callback order) in QEMU block drivers, an event-driven disk simulator (block/blksim.c) was developed along with FVD. This disk simulator allows the tester to exercise various race conditions, by fully controlling the I/O order, callback order, and return code of every I/O operation.

• On the observation of the limited code coverage of manual testing, a fully automated testing tool (qemu-io-auto.c) was developed along with FVD. This tool can, e.g., simulate 1,000 threads concurrently submitting overlapping disk I/O requests to QEMU block drivers, use blksim to inject I/O errors and race conditions, and automatically verify the correctness of I/O results. This tool can run unattended to exercise an unlimited number of randomized test cases. Once it finds a bug, the bug is precisely repeatable with the help of blksim, even if it is a rare race condition bug. This makes debugging much easier. By now, this automated tool has been running continuously for months to stress test FVD.

• The FVD code automatically checks potential memory corruption and resource leaks, including memory leak, unreleased locks, etc. These subtle issues may not immediately affect the correctness of read or write operations, but can cause problems in the long run.