Google Summer of Code 2019

From QEMU
Jump to: navigation, search

Introduction

QEMU is applying to Google Summer of Code 2019. This page contains our ideas list and information for students and mentors. Google Summer of Code is an open source internship program for university students offering 12-week, full-time, paid remote work from May to August!

Applicants: You are welcome to think about project ideas and familiarize yourself with QEMU, but please don't invest too much time at this early stage. Google will announce participating organizations on February 26.

Are you also eligible for Outreachy_2019_MayAugust internships? If you are eligible we recommend applying for the same project ideas through both GSoC and Outreachy.

Application Process

After contacting the mentor to discuss the project idea you should fill out the application form at [1]. The form asks for a problem description and outline of how you intend to implement a solution. You will need to do some background research (looking at source code, browsing relevant specifications, etc) in order to form an idea of how to tackle the project. The form asks for an initial 12-week project schedule which you should create by breaking down the project into tasks and estimating how long they will take. The schedule can be adjusted during the summer so don't worry about getting everything right ahead of time.

Candidates may be invited to an IRC interview with the mentor. The interview consists of a 30 minute C coding exercise, followed by technical discussion and a chance to ask questions you have about the project idea, QEMU, and GSoC. The coding exercise is designed to show fluency in C programming.

Here is a C coding exercise we have used in previous years when interviewing students: 2014 coding exercise

Try it and see if you are comfortable enough writing C. We cannot answer questions about the previous coding exercise but hopefully it should be self-explanatory.

If you find the exercise challenging, think about applying to other organizations where you have a stronger technical background and will be more competitive compared with other candidates.

Key Dates

From the timeline

  • March 25, 2019 - April 9, 2019 Student Applications
  • May 6, 2019 - Student Projects Announced

Find Us

  • IRC (GSoC specific): #qemu-gsoc on irc.oftc.net
  • IRC (development):
    • QEMU: #qemu on irc.oftc.net
    • KVM: #kvm on chat.freenode.net

Please contact the mentor for the project idea you are interested in. IRC is usually the quickest way to get an answer.

For general questions about QEMU in GSoC, please contact the following people:

Project Ideas

This is the listing of suggested project ideas. Students are free to suggest their own projects, see #How to propose a custom project idea below.

Device emulation

I2C Passthrough

Summary: Implement I2C device passthrough on Linux hosts so that a single board computer like a Raspberry Pi can be used to develop applications for microcontrollers like the micro:bit.

QEMU emulates I2C devices in software but currently cannot pass through real I2C devices from the host to the guest. It would be useful to access real I2C devices from inside the guest, for example for developers writing and testing software under QEMU on their computer. This would be used like -device i2c-passthrough,device=/dev/i2c-N,hostaddr=0x48,address=0x49 (another possibility is to implement Linux I2C device support as a character device, like -chardev linux-i2c,address=0x48,device=/dev/i2c-N,id=i2c-chardev -device i2c-passthrough,chardev=i2c-chardev,address=0x49.

A very basic implementation of I2C passthrough can be very simple (150-200 lines of code perhaps) using the read/write interface to /dev/i2c-N. However, read/write is very limited and cannot drive all I2C devices because it does not support repeated start conditions. The current I2C API in QEMU is also very limited. Therefore there are many possible extensions:

  • implementing I2C passthrough as an SMBusDevice instead of I2CSlave. SMBusDevice provides a higher-level interface than I2CSlave, with support for repeated start conditions and block transfers. (Question: how would the SMBusDevice API translate to /dev/i2c-N ioctls)?
  • improving SMBusDevice support in QEMU: currently the SMBusDevice->I2CSlave adaptor in hw/i2c/smbus_slave.c only implements a subset of SMBus, in particular it can only read a single byte at a time from the device. Your task could be to improve the QEMU I2CBus API so that, at least for some I2C bus implementations, more functionality of SMBusDevice is accessible via I2CBus.
  • Currently I2C is entirely synchronous. It would be nice to support asynchronous communication between the I2C master and the chardev, where the i2c-passthrough can hold down the I2C clock until linux-i2c has data ready. On the emulation side, this would require clock stretching support in i2c-passthrough.

If you are interested in device emulation especially, you could also implement an I2C adapter from scratch (e.g. the micro:bit TWI controller emulation on the nRF51 system-on-chip) if there's time.

This project will allow you to learn about the I2C bus and how to write device emulation code in QEMU. You will enjoy it if you like working with physical hardware.

Links:

Details:

  • Skill level: advanced (the project is not very difficult, but it is larger than usual)
  • Language: C
  • Mentor: Paolo Bonzini <pbonzini@redhat.com> ("bonzini" on IRC), Stefan Hajnoczi <stefanha@redhat.com> ("stefanha" on IRC)
  • Special requirements: you should already own already a relatively powerful single board computer (for example a Raspberry Pi 3 or a Le Potato) and one or more I2C devices (for example real-time clocks, temperature sensors, Wii Nunchuks) and basic tools for connecting them such as a soldering iron and 0.1" jumper cables and header pins. It is important that you have these before the coding period begins; if you don't, take into account shipping and customs time, and/or talk to the mentors.

PVRDMA Live Migration

Status: Sukrit Bhatnagar is working on this project for GSoC.

Summary: Add live migration support for the PVRDMA device.

The PVRDMA device allows Virtual Machines to use RoCE (RDMA over Converged Ethernet) without assigning a physical device (or a virtual function in SR-IOV case). The big wins are we don't need to pin the VM memory in RAM and the device can support Live Migration. This project addresses the latter point.

While the PVRDMA device can be used in a hybrid environment where the nodes can be a bare-metal machine or a VM, this project aims to enable Live Migration only if all the nodes are VMs.

The above assumption allows a relatively easy approach by creating a QEMU protocol for broadcasting/receiving notifications during Live Migration. Since RoCE uses Ethernet as the data link layer and QEMU already supports Live Migration for emulated Ethernet devices, the project will concentrate on passing the device state from source to destination and on the protocol mentioned above.


Links:

Details:

  • Skill level: intermediate
  • Language: C
  • Mentor: Marcel Apfelbaum <marcel.apfelbaum@gmail.com>, Yuval Shaia <yuval.shaia@oracle.com>

TCG (just-in-time compiler)

Measure Tiny Code Generation Quality

Status: Vanderson M. do Rosario is working on this project for GSoC.

Summary: Improve the diagnostics tools to asses code quality

Improving the code generation of the TCG backend is a hard task that involves reading through large amounts of text looking for anomalies in the generated code. It would be nice to have tools to more readily extract and parse code generation information. This would include:

Modifying code generator, dumping additional data

  • which are hot blocks (frequently run, hence more important performance wise)
  • export block JIT information for perf tool (the later version)

Tweaking -d op,out_asm output

  • how many fills/spills in a block (where register contents are moved due to register pressure)
  • number of host instructions for each guest instruction (JIT profiling has a basic version of this)
  • elide or beautify common blocks like softmmu access macros (which are always the same)

Modifying the HMP

  • support interactive exploration of translation state (system emulation)

QEMU currently only works on translating simple basic blocks with one or two exit paths. This work could be a pre-cursor to supporting Internships/ProjectIdeas/Multi-exit Hot Blocks in the future.

Links:


Details:

  • Skill level: intermediate or advanced, understanding of code generation (compilers/JITs)
  • Language: C, Assembly (x86 or preferred host)
  • Mentor: Alex Bennée <alex.bennee@linaro.org> (stsquad on #qemu IRC)
  • Suggested by: Alex Bennée

AVX

Status: Jan Bobek is working on this project for GSoC.

Summary: Support for AVX within TCG

QEMU's TCG just-in-time compiler translates target CPU instructions into host CPU instructions so that programs written for other CPU architectures can be run on any host. Modern CPUs features vector processing instruction sets, sometimes called Single Instruction Multiple Data (SIMD) instructions, for performing the same operation on multiple elements of data in just one instruction. Intel's SSE and AVX instruction set extensions were introduced for x86 CPUs for this purpose.

The target/i386 front end has support for TCG emulation of SSE4.1, but does not have support for later vector extensions such as AVX. Your task is to implement and test AVX instructions that are currently missing in QEMU.

Links:

Details:

  • Skill level: intermediate to advanced
  • Language: C
  • Mentor: Richard Henderson <richard.henderson@linaro.org> (rth on #qemu IRC)
  • Suggested by: Nick Renieris

Storage

io_uring AIO engine

Status: Aarushi Mehta is working on the project for Outreachy. Project status is here.

Summary: Add io_uring support to QEMU for high-performance disk I/O on Linux

The io_uring interface supersedes the Linux AIO API for asynchronous I/O. The core functionality of asynchronous I/O APIs is I/O request submission (reads/writes/flushes) and completion processing at a later point in time. Unlike a blocking read(2)/write(2) syscall, this allows the application threads to perform other activity while one or more I/O requests are in flight.

QEMU currently supports two asynchronous I/O engines: aio=threads (a thread-pool that invokes preadv(2)/pwritev(2)/fdatasync(2)) and aio=native (Linux AIO). This project will add io_uring support, which should achieve better performance than Linux AIO. This is because io_uring offers several optimizations:

  • Memory buffers can be registered (pinned) ahead of time to avoid pinning on each request
  • File descriptors can be held long-term to avoid the need to acquire them on each request
  • A single system call can both submit and complete I/O requests or polling mode can be used to avoid system calls altogether

This project consists of the following tasks:

  • Understanding the io_uring userspace ABI
  • Extending block/file-posix.c to use io_uring
  • Benchmarking io_uring against Linux AIO and aio=threads
  • Adding polling mode support for completions (similar to existing Linux AIO polling code in QEMU)
  • Stretch goal: Adding polling mode support for submissions
  • Stretch goal: Adding a fast path when QEMU block layer features are not in use
  • Stretch goal: Use IORING_OP_POLL_ADD so unify QEMU's polling and blocking event loop code paths

Links:

Details:

  • Skill level: intermediate
  • Language: C
  • Mentor: Julia Suvorova <jusual@mail.ru> ("jusual" on IRC), Stefan Hajnoczi <stefanha@redhat.com> ("stefanha" on IRC)

vhost-user-blk device backend

Summary: Implement a vhost-user-blk device backend inside QEMU so guests can efficiently access shared disk images.

QEMU can connect virtio-blk disks to external processes that act as vhost-user-blk device backends. This makes it possible for QEMU guests to access disks managed by SPDK or other software-defined storage appliances.

QEMU itself does not offer a vhost-user-blk device backend although the QEMU block layer has a number of features that make QEMU desirable as a software-defined storage appliance in its own right. For example, multiple VMs could safely access a shared qcow2 disk image file with one of the QEMUs acting as the vhost-user-blk device backend. Today this is can be worked around using QEMU's NBD support, but its performance will always be lower since it is a network protocol.

The goal is to add a vhost-user-blk device backend to QEMU so that disks can be exported to other processes. The following steps are necessary:

  • Understand libvhost-user, QEMU's library for implementing vhost-user device backends
  • Add a QEMU monitor command for instantiating vhost-user-blk device backends given a blockdev and a UNIX domain socket
  • Add a QEMU monitor command for shutting down and removing vhost-user-blk device backends
  • Implement a vhost-user-blk device backend using libvhost-user (see the vhost-user-blk.c stand-alone example below)
  • Extend QEMU's vhost-user tests to take advantage of your vhost-user-blk device backend

Links:

Details:

  • Skill level: intermediate
  • Language: C
  • Mentor: Kevin Wolf <kwolf@redhat.com> ("kwolf" on IRC), Stefan Hajnoczi <stefanha@redhat.com> ("stefanha" on IRC)

Testing & documentation

qgraph support for device tree discovery

Summary: Enable tests for the qgraph driver framework to discover devices using a device tree, instead of having to specify the devices on the machine manually.

qgraph, developed during GSoC 2018 (merging in progress), implements an introspectable description of QEMU's supported machine types and a set of drivers that tests can use to start a guest and interact with its devices. Currently, PCI is the only discoverable bus implemented in qgraph; any non-PCI device must be listed manually in the machine type. The purpose of this project is to add support to qgraph for reading a device tree and selecting tests based on the OpenFirmware device names found in the device tree. The device tree can be provided by the virtual machine, or it can be read from an external file.

If the task turns out to be too short to fill the whole internship, the remaining time can be filled by writing new tests for devices that are usually exposed as part of a device tree.

Links:

  • KVM Forum 2018 presentation on qtest: video
  • last version of the qgraph framework: patchew

Details:

  • Skill level: intermediate
  • Language: C
  • Mentor: Paolo Bonzini <pbonzini@redhat.com>, Laurent Vivier <lvivier@redhat.com>
  • Suggested by: Paolo Bonzini <pbonzini@redhat.com> (IRC nick bonzini)

virtio-net oss-fuzz support

Status: Alexander Oleinik is working on this project for GSoC.

Summary: Integrate oss-fuzz into QEMU so that the virtio-blk device can be fuzz tested.

oss-fuzz offers a fuzz testing service to open source projects. This means random inputs are continuously tested against the program in order to find crashes and other bugs. Fuzz testing complements hand-written test suites by exploring the input space of a program and therefore the code paths that may be taken.

The goal of this project is to experiment with integrating oss-fuzz into the virtio-net device that Qemu emulates for guest networking. virtio-net-pci is a PCI device connected to the virtual machine's PCI bus and has a configuration space that can be programmed by the guest. The device itself is specified by the VIRTIO specification which describes the functionality of the device. Bugs could potentially exist at both the PCI and VIRTIO levels, so it's important to fuzz both of them. Care should be taken to pick a design that could be generalized for all virtio devices, eg. virtio-blk.

Fuzzing emulated devices involves accessing their hardware registers randomly to make the device respond. QEMU has a device testing interface called "qtest" that accepts read/write and other commands over a socket and is ideal for writing device-level tests. You may find that oss-fuzz works better integrated directly into the QEMU program instead of as a separate qtest program, so you can consider adding a new command-line option to QEMU for running in oss-fuzz mode.

This project involves learning about VIRTIO and PCI devices, as well as figuring out how to integrate oss-fuzz into QEMU so that it can effective explore the code paths in virtio-net device emulation code. You will enjoy this project if you want to learn how device emulation works and are interested in fuzzers.

The project will primarily be in three phases. The first phase involves understanding the ecosytsem - Qemu, qtest, oss-fuzz and llvm etc. The second phase involves utilizing the qtest framework or utilizing the functionality to integrate in Qemu to fuzz the virtio-net device registers. THe third phase will involve running our fuzzing framework, analyzing results and identifying bugs.

Links:

Details:

  • Skill level: intermediate
  • Language: C
  • Mentor: Bandan Das <bsd@redhat.com> ("bsd" on #qemu IRC), Paolo Bonzini <pbonzini@redhat.com> ("bonzini" on #qemu IRC), Stefan Hajnoczi <stefanha@redhat.com> ("stefanha" on #qemu IRC)

Memory hotplug test

Summary: Create memory hotplug testing infrastructure.

Several QEMU targets support memory hotplug feature, which allows to (un)provision extra memory without requiring virtual machine reboot or restart for changes to take the effect. QEMU however lacks regression tests for the feature. Goal of the project is refactor QEMU's "-m" option legacy parsing into machine object properties, which should provide ability to inspect memory settings using QMP interface and implement test cases on top of that.

Project is to be split in to following parts:

  • Refactor "-m mem,slots,maxmem" into MachineClass properties and replace related global variables where they are still used with properties. Make "set_memory_options()" use new properties leaving only CLI parsing there. Goal is to make "set_memory_options()" function a thin wrapper that takes care of CLI parsing and makes CLI option '-m' an alias to new -machine mem/mem-slots/maxmem options.
  • Create a set of basic tests for 'make check' to verify that -m/-machine CLI parsing works as expected and based on that a set memory hot-add tests using QMP introspection to verify the expected behaviour.
  • If time allows create guest-host memory hotplug ABI tests for x86/sPAPR/s390x targets.

Links:

  • $(QEMU sources)/docs/memory-hotplug.txt

Details:

  • Skill level: intermediate
  • Language: C
  • Mentor: Igor Mammedov <imammedo@redhat.com> ("imammedo" on IRC), David Hildenbrand <david@redhat.com> ("dhildenb" on IRC)

API documentation generation

Status: Gabriel Barreto is working on this project for GSoC.

Summary: Generation of API documentation from doc comments

QEMU currently has many functions documented using the GTK-Doc syntax, but there is no mechanism to actually generate API documentation from these doc comments. We need build rules that generate API documentation from C and Python source code.

Tasks:

  • Picking a documentation generation tool and syntax (unclear if we should stay with GTK-Doc)
  • Fixing or converting existing doc comments to the chosen syntax
  • Writing build rules to generate the documentation
  • Extra tasks, if time allows:
    • Improving clarity or formatting of existing doc comments
    • Converting existing ad-hoc comments in the code to doc comment syntax
    • Add doc comments to existing APIs that are undocumented

Links:

Details:

  • Skill level: beginner
  • Language: C, Python, GNU Make
  • Mentor: Eduardo Habkost <ehabkost@redhat.com> ("ehabkost" on IRC)
  • Suggested by: Eduardo Habkost

Guest ABI automated testing

Summary: Automated test of Guest ABI and compatibility

QEMU tries to provide a stable guest ABI on versioned machine-types. Despite investing lots of effort keeping compatibility, we have no automated testing to detect common mistakes that break guest ABI. It should be possible to write automated test cases that will compare a virtual machine to a previously stored dump of guest ABI information.

A guest ABI dump may include, for example:

  • Data returned by CPUID instruction (or equivalent)
  • Physical memory and I/O port maps
  • Device addresses (PCI, USB, etc.)
  • Device IDs and other guest-visible device fields
  • Value of QOM properties that affect device behavior

This might require improving or adding new QMP commands to provide information to be validated by the automated test cases. Some test cases may use a custom kernel image for collecting guest-visible data, or extending the qtest protocol.


Links:

Details:

  • Skill level: intermediate
  • Language: C, Python
  • Mentor: Eduardo Habkost <ehabkost@redhat.com> ("ehabkost" on IRC)
  • Suggested by: Eduardo Habkost

Nested SVM test improvements

Summary: Implement tests for AMD SVM nested virtualization

KVM supports both AMD SVM and Intel VMX technologies for hardware-assisted virtualization on x86 and while similar in concept, these technologies have significant differences between them. When we want to test nested virtualization we need to make the test familiar with these differences (as the test, in fact, is a small hypervisor running on top of KVM). Currently, we have two frameworks which test nested virtualization: kvm-unit-tests and kvm selftests. SVM support in kvm-unit-tests lags behind VMX and kvm selftest framework doesn't support it at all. The project will have the following parts:

  • Implementing SVM support in kvm selftest framework, writing a generic 'nesting' test utilizing vmx/svm depending on the host's hardware.
  • Improving SVM support in kvm-unit-tests making it match (where possible) VMX and adding SVM-specific features.

The applicant must have access to modern physical AMD hardware (Opteron 62xx/63xx, Epyc) to be able to accomplish the task.

Links:

Details:

  • Skill level: intermediate or advanced
  • Language: C
  • Mentor: Vitaly Kuznetsov <vkuznets@redhat.com>, Paolo Bonzini <pbonzini@redhat.com> ("bonzini" on IRC)

Jailhouse

Configuration checker for Jailhouse Hypervisor

Summary: Validate Jailhouse configurations against a set of consistency rules

The Jailhouse hypervisor uses low-level configuration files in order to describe the partitioning of a system. These files are currently not checked for consistency and are rather easy to get wrong. Not all errors that the system designer may make can be identified, but at least a good part of them can.

The goal of this task is to enhance the existing Python-based tooling around Jailhouse to take a set of configurations (system+root-cell configuration and all at simultaneously active non-root cell configs), run a predefined list of checks against them and report any findings. This could look like that:

  # jailhouse config check ROOT.cell NON-ROOT-A.cell NON-ROOT-B.cell ...
  Error: MSI-X region of PCI device 00:01.2 directly mapped into NON-ROOT-A

The input to the checker shall be binary config files for which Jailhouse already has a parsing module that translate them into Python objects.

Rules that should at least be validated are:

  • memory region overlaps
  • invalid pass-through of critical resources (MSI-X, irq controllers, PCI config ports etc.)
  • inconsistencies between root and non-root configs (e.g duplicate assignments of resources, missing root-cell access to loadable memory regions of non-root cells etc.)
  • fully zero-initialized entries in configuration array (indicates missing elements)
  • invalid PCI capability or shared-memory region indices

The rules will be further detailed as the project starts. A bonus task can be the definition of additional rules, based on the analysis of the configuration format and its semantics.

Links:

Details:

  • Skill level: intermediate
  • Language: Python, C
  • Mentor: Jan Kiszka <jan.kiszka@web.de>

AMD Interrupt Remapping Support for Jailhouse hypervisor

Summary: Supplement existing AMD IOMMU code with interrupt remapping support.

Jailhouse currently supports IOMMU on AMD-based x86 systems, however it does so for memory transfers only. In order for the isolation to be complete, the analogous translation should be applied to interrupt messages as well. This technique is commonly known as interrupt remapping and it is provided by AMD IOMMU.

The goal of this project is to implement the missing bits in AMD IOMMU interrupt remapping support to bring it on par with Intel interrupt remapping support Jailhouse already has. This involves writing code for the hypervisor core and arch-specific parts as well as adding relevant options to the config file generator.

In order to succeed with this project, you must have a sufficiently recent (2015 or newer) AMD computer (PC or laptop) which you can use as a testbed. You should also check that:

  • The APU is Kaveri-based or newer
  • There is an IOMMU option in the UEFI setup tool and it's enabled
  • A recent Linux distribution reports IOMMU support in dmesg
  • And you can pass-through a PCI device such as a network or sound card to a guest running in the virt-manager.

You should also check that the mainboard has a serial port header as this helps debugging a lot.

Links:

Details:

  • Skill level: Advanced
  • Language: C
  • Mentor: Jan Kiszka <jan.kiszka@web.de>, Valentine Sinitsyn <valentine.sinitsyn@gmail.com>

Project idea template

=== TITLE ===
 
 '''Summary:''' Short description of the project
 
 Detailed description of the project.
 
 '''Links:'''
 * Wiki links to relevant material
 * External links to mailing lists or web sites
 
 '''Details:'''
 * Skill level: beginner or intermediate or advanced
 * Language: C
 * Mentor: Email address and IRC nick
 * Suggested by: Person who suggested the idea

How to propose a custom project idea

Applicants are welcome to propose their own project ideas. The process is as follows:

  1. Email your project idea to qemu-devel@nongnu.org. CC Stefan Hajnoczi <stefanha@gmail.com> and regular QEMU contributors who you think might be interested in mentoring.
  2. If a mentor is willing to take on the project idea, work with them to fill out the "Project idea template" above and email Stefan Hajnoczi <stefanha@gmail.com>.
  3. Stefan will add the project idea to the wiki.

Note that other candidates can apply for newly added project ideas. This ensures that custom project ideas are fair and open.

How to get familiar with our software

See what people are developing and talking about on the mailing lists:

Grab the source code or browse it:

Build QEMU and run it: QEMU on Linux Hosts

Important links

Information for mentors

Mentors are responsible for keeping in touch with their student and assessing the student's progress. GSoC has a mid-term evaluation and a final evaluation where both the mentor and student assess each other.

The mentor typically gives advice, reviews the student's code, and has regular communication with the student to ensure progress is being made.

Being a mentor is a significant time commitment, plan for 5 hours per week. Make sure you can make this commitment because backing out during the summer will affect the student's experience.

The mentor chooses their student by reviewing student application forms and conducting IRC interviews with candidates. Depending on the number of candidates, this can be time-consuming in itself. Choosing the right student is critical so that both the mentor and the student can have a successful experience.