Outreachy 2015 MayAugust: Difference between revisions

From QEMU
(Created page with '= Introduction = QEMU is seeking funding to participate as a mentoring organization in [https://www.gnome.org/outreachy/ Outreachy 2015 May-August]. This page contains our ideas…')
 
No edit summary
Line 42: Line 42:


This is the listing of suggested project ideas. Candidates are free to suggest their own projects by emailing qemu-devel@nongnu.org and (optionally) CCing potential mentors.
This is the listing of suggested project ideas. Candidates are free to suggest their own projects by emailing qemu-devel@nongnu.org and (optionally) CCing potential mentors.
== QEMU/KVM projects ==
=== VT-d interrupt emulation with KVM support ===
'''Summary:''' Extend the existing VT-d model with support for interrupt remapping, including KVM irqchip support
QEMU gained basic Intel IOMMU support (VT-d) during last year's GSoC. DMA remapping (DMAR) works already, but interrupt remapping is still missing. There are some hacks that pleases Linux as well as the Jailhouse hypervisor (both shall serve as test cases here), but they are not upstream, lack error handling and only work if the KVM in-kernel irqchip is disabled.
This project shall overcome the limitations. It can start with addressing the open issues in ''-no-kvm-irqchip'' mode and get the changes upstream. Then it shall look into extending KVM to support interrupt remapping emulation as well. The primary challenge here is to ensure the interrupts of the in-kernel IOAPIC are properly remapped according to the user space VT-d model and its programming by the guest. This likely involves designing a new KVM user space interface, getting design and implementation reviewed and accepted by the KVM community and then make use of it in QEMU. Other in-kernel IRQ delivery paths will need a look as well as far as they are IOMMU compatible (virtio is not yet, thus can be skipped; vfio may be considered).
Possibly not much code needs to be written for this, but intensive interaction with KVM maintainers will be required because sensitive code will have to be touched. So this will be a unique chance to get deeply involved in the QEMU/KVM development process and work on both kernel as well as user space components.
'''Links:'''
* [http://thread.gmane.org/gmane.comp.emulators.qemu/291304 VT-d emulation patches]
* [http://git.kiszka.org/?p=qemu.git;a=shortlog;h=refs/heads/queues/vtd-intremap VT-d interrupt remapping hacks]
* [http://www.intel.com/content/www/us/en/intelligent-systems/intel-technology/vt-directed-io-spec.html VT-d specification]
* [https://github.com/siemens/jailhouse Jailhouse hypervisor], specifically its [https://github.com/siemens/jailhouse/blob/master/hypervisor/arch/x86/vtd.c VT-d module]
'''Details:'''
* Skill level: advanced
* Language: C
* Mentor: Jan Kiszka <jan.kiszka@web.de>
* Suggested by: Jan Kiszka
=== Nested Virtualization hardening ===
'''Summary:''' Extend kvm-unit-tests with security and sanity checks
With Nested Virtualization, guest operating systems can behave as hypervisors and run their own guests utilizing host hardware extensions. In this context, the hypervisor that runs on baremetal is L0, the hypervisor/guest that L0 runs is L1 and so forth. A major challenge in improving the stability of this feature in KVM is to test various configurations with different hypervisors coupled with various guests. However, a more practical approach is to write unit tests that mimic the nested hypervisor's basic behavior. We already have a number of unit tests as part of the kvm-unit-tests framework. [0][1]
The scope of this project is to write unit tests for conditions that potentially let the level 2 guest kill or hang the level 1 guest or perhaps less importantly, the level 2 guest itself. First, we could visit known cases which have caused crashes/hangs in the past, and investigate whether we could write unit tests so we don't inadvertently introduce the same problems again when adding newer features and/or bug fixes. For example, a bug was introduced a while back that incorrectly injected L1's interrupts to L2 [2]. We could write a unit test that attempts to inject a few interrupts to L1(with help from L0) and verify that L1 indeed received all the interrupts. There are also cases[3] where emulate_invalid_guest_state=0 avoids hangs; which probably means that we could write tests that force certain paths during instruction emulation. This will probably also help with checks for BUG() at various places (and whether we should consider removing them with something less sinister :)) Another idea is to run nested unit tests with various values for kvm_intel module parameters - there have been cases where new commits break only if a certain feature is enabled/disabled.
Here's a tentative list of key hardening areas -
  1. Writing unit tests for missing VMX features tests; eg- invvpid,
    load/store, posted interrupts. [1]
  2. Unit tests for emulated instructions[4] - can we write a test that
    can hit a BUG() during instruction emulation ?
  3. A framework that can enable/disable various module parameters and run
    1. and 2.
  4. Unit tests for interrupt handling paths - can we write a test
    that times an interrupt injection such that results in an incorrect
      injection.
  5. Anything else that comes up as we start tackling these areas.
We should primarily focus on 1, 2 and 3 and get to the others if time permits.
'''Links:'''
* [https://www.youtube.com/watch?v=ryMMDWdKbS4 Improvements in Nested Virtualization, Devconf ]
* [https://git.kernel.org/cgit/virt/kvm/kvm-unit-tests.git kvm-unit-tests <nowiki>[0]</nowiki> ]
* [https://git.kernel.org/cgit/virt/kvm/kvm-unit-tests.git/tree/x86/vmx_tests.c vmx tests in kvm-unit-tests <nowiki>[1]</nowiki>]
* [https://lkml.org/lkml/2014/7/2/60 Nested VMX Interrupt handling bug <nowiki>[2]</nowiki>]
* [https://bugzilla.redhat.com/show_bug.cgi?id=998713 Nested VMX bug report with instruction emulation <nowiki>[3]</nowiki>]
* [https://git.kernel.org/cgit/virt/kvm/kvm.git/tree/arch/x86/kvm/emulate.c KVM instruction emulation source file <nowiki>[4]</nowiki>]
'''Details:'''
* Skill level: advanced
* Language: C
* Mentor: Bandan Das <bsd@redhat.com>
* Suggested by: Bandan Das


== QEMU projects ==
== QEMU projects ==
=== Integrate IDE ATAPI and SCSI CD-ROM emulation ===
'''Summary:''' Unify IDE ATAPI CD-ROM emulation with SCSI CD-ROM emulation to reduce code duplication and squash bugs
Currently the IDE ATAPI and SCSI CD-ROM emulated devices are two
distinct implementations, and have different bugs/features.
This leads to the situation that things which work when using the IDE
emulation don't work when using the SCSI emulation and vice versa.
So this project is for implementing a virtual ATA-to-SCSI bridge
in QEMU, use this for emulating an IDE ATAPI drive, and merging the
missing features from the IDE implementation into the SCSI one.
This project will allow you to expore IDE and SCSI storage protocols and learn how CD-ROM drives are operated.
'''Links:'''
* [https://en.wikipedia.org/wiki/ATA_Packet_Interface ATA Packet Interface on Wikipedia]
* [https://en.wikipedia.org/wiki/Scsi SCSI on Wikipedia]
'''Details:'''
* Skill level: intermediate
* Language: C
* Mentor: Hannes Reinecke <hare@suse.de>, John Snow <jsnow@redhat.com> (jsnow on QEMU IRC)
* Suggested by: Hannes Reinecke <hare@suse.de>
=== AMD IOMMU emulation ===
'''Summary:''' Rework existing AMD IOMMU emulation patches according to new IOMMU model of QEMU
There was already a proposal to add AMD IOMMU emulation in 2011, but QEMU was lacking a proper layer to integrate this effort smoothly. This is now possible like the successful integration of basic VT-d emulation demonstrated during last year's GSoC. So this task is a little bit simpler: study the old patches as well as the Intel version, port the old ones over the new interfaces, add ACPI tables (missing in the old version), ensure that the resulting IOMMU model works according to the AMD specification - and that guests like Linux/KVM can use it correctly. The model should also support interrupt remapping, but only for the user-space irqchip mode (kernel interface support is part of a different project proposal).
'''Links:'''
* [http://thread.gmane.org/gmane.comp.emulators.kvm.devel/73400 AMD IOMMU patches]
* [http://thread.gmane.org/gmane.comp.emulators.qemu/291304 VT-d emulation patches]
* [http://developer.amd.com/wordpress/media/2012/10/488821.pdf AMD IOMMU specification]
'''Details:'''
* Skill level: intermediate to advanced
* Language: C
* Mentor: Jan Kiszka <jan.kiszka@web.de>
* Suggested by: Jan Kiszka
=== Implement support for Mac OS 9 in QEMU ===
'''Summary:'''
QEMU has gone a long way in emulating a Macintosh. But we can still improve. Adding support for Mac OS 9 would be a great imporovement. This would allow everyone who misses their older applications to be reacquainted with them. It would also expand QEMU's abilities.
'''Links:'''
* Classic API - https://developer.apple.com/legacy/library/documentation/mac/pdf/MacintoshToolboxEssentials.pdf
* Discussion on implementing Mac OS 9 support - http://www.emaculation.com/forum/viewtopic.php?t=7047
* History of the Mac OS - http://www.operating-system.org/betriebssystem/_english/bs-macos.htm
* Technical info - http://www.emaculation.com/forum/viewtopic.php?f=3&t=8152
'''Details:'''
* Skill level: Advanced
* Language: C
* Mentor: Alexander Graf <agraf@suse.de>
* Suggested by: John Arbuckle <programmingkidx@gmail.com>


== Libvirt projects ==
== Libvirt projects ==
=== Introducing job control to the storage driver ===
'''Summary:''' Currently, libvirt support job cancellation and progress reporting on domains.
That is, if there's a long running job on a domain, e.g. migration, libvirt
reports how much data has already been transferred to the destination and how
much still needs to be transferred. However, libvirt lacks such information
reporting in storage area, to which libvirt developers refer to as the storage
driver. The aim is to report progress on several storage tasks, like volume
wiping, file allocation an others.
* Skill level: intermediate
=== Enhancing libvirt-designer ===
'''Summary:''' The project is in its very early stage of life. The [https://www.redhat.com/archives/libvirt-announce/2012-October/msg00003.html libvirt-designer] tries to ease generation of libvirt XML with coworking with libosinfo project. Contact me and we can find something suitable.
* Skill level: beginer
=== Making virsh more bash like ===
'''Summary:''' If you have ever used virsh, you certainly reached the point where you stuggle with its user friendliness. Or unfriendliness I should rather say. Virsh is missing a lot of bash functionality that users consider natural: from automatic completion of object names, through redirecting command outputs through piping commands together. The aim would be to make these functions available in virsh and thus make user experience better.
* Skill level: Advanced
=== Abstracting device address allocation ===
'''Summary:''' There are many types of addresses that devices can have in libvirt's XML description.  Not all address types are properly assigned and checked for duplicates.  The goal of this is to have an abstract data structure that would handle assigning all kinds of addresses, handle duplicates, etc.
* Skill level: beginner
=== Admin interface APIs ===
'''Summary:''' We are currently working on an administrative interface for the libvirt daemon.  This interface will be able to probe and change daemon settings live.  Since this is going to be a new interface, there needs to be new APIs implemented.  These APIs may do various interesting things strarting from showing the current logging filters up to force-disconnecting clients with certain properties.  It is up to you what you would like to add, but of course this needs to be discussed and agreed on with the community.
* Skill level: intermediate
=== Your own idea ===
=== Your own idea ===


Line 64: Line 219:
* Suggested by: Michal Privoznik <mprivozn@redhat.com>
* Suggested by: Michal Privoznik <mprivozn@redhat.com>


=== Running docker containers using virt-sandbox ===
'''Summary:''' The basic idea is to be able to grab container images from the docker registry
and run them using virt-sandbox. The application could then run as either a KVM or LXC guest.
Daniel already has a [https://berrange.fedorapeople.org/virt-sandbox-image.py proof of concept] to download
the docker images, but there is still quite some work to be done, like:
* transforming the docker image into a qcow2 image,
* getting libvirt to set the username of the process to run,
* getting libvirt to set environment variables
'''Links:'''
* https://libvirt.org
* https://libvirt.org/git
* Daniel's PoC: https://berrange.fedorapeople.org/virt-sandbox-image.py
* Dockerfile documentation: http://docs.docker.com/reference/builder/
'''Details:'''
* Skill level: advanced
* Language: C, possibly python too
* Mentor: Cedric Bosdonnat <cbosdonnat@suse.com>; cbosdonnat on IRC (OFTC and freenode)
* Suggested by: Cedric Bosdonnat


== Project idea template ==
== Project idea template ==

Revision as of 18:03, 23 February 2015

Introduction

QEMU is seeking funding to participate as a mentoring organization in Outreachy 2015 May-August. This page contains our ideas list and information for candidates and mentors.

Next deadline: February 17 for initial list of project ideas and mentors.

Find Us

  • IRC: #qemu-outreachy on irc.oftc.net
  • IRC (development):
    • QEMU: #qemu on irc.oftc.net
    • libvirt: #virt on irc.oftc.net
    • KVM: #kvm on chat.freenode.net

Please contact the mentor for the project idea you are interested in. IRC is usually the quickest way to get an answer.

For general questions about QEMU in Outreachy, please contact the following people:

How to get familiar with our software

See what people are developing and talking about on the mailing lists:

Grab the source code or browse it:

Build QEMU and run it: QEMU on Linux Hosts

Project Ideas

This is the listing of suggested project ideas. Candidates are free to suggest their own projects by emailing qemu-devel@nongnu.org and (optionally) CCing potential mentors.

QEMU/KVM projects

VT-d interrupt emulation with KVM support

Summary: Extend the existing VT-d model with support for interrupt remapping, including KVM irqchip support

QEMU gained basic Intel IOMMU support (VT-d) during last year's GSoC. DMA remapping (DMAR) works already, but interrupt remapping is still missing. There are some hacks that pleases Linux as well as the Jailhouse hypervisor (both shall serve as test cases here), but they are not upstream, lack error handling and only work if the KVM in-kernel irqchip is disabled.

This project shall overcome the limitations. It can start with addressing the open issues in -no-kvm-irqchip mode and get the changes upstream. Then it shall look into extending KVM to support interrupt remapping emulation as well. The primary challenge here is to ensure the interrupts of the in-kernel IOAPIC are properly remapped according to the user space VT-d model and its programming by the guest. This likely involves designing a new KVM user space interface, getting design and implementation reviewed and accepted by the KVM community and then make use of it in QEMU. Other in-kernel IRQ delivery paths will need a look as well as far as they are IOMMU compatible (virtio is not yet, thus can be skipped; vfio may be considered).

Possibly not much code needs to be written for this, but intensive interaction with KVM maintainers will be required because sensitive code will have to be touched. So this will be a unique chance to get deeply involved in the QEMU/KVM development process and work on both kernel as well as user space components.

Links:

Details:

  • Skill level: advanced
  • Language: C
  • Mentor: Jan Kiszka <jan.kiszka@web.de>
  • Suggested by: Jan Kiszka

Nested Virtualization hardening

Summary: Extend kvm-unit-tests with security and sanity checks

With Nested Virtualization, guest operating systems can behave as hypervisors and run their own guests utilizing host hardware extensions. In this context, the hypervisor that runs on baremetal is L0, the hypervisor/guest that L0 runs is L1 and so forth. A major challenge in improving the stability of this feature in KVM is to test various configurations with different hypervisors coupled with various guests. However, a more practical approach is to write unit tests that mimic the nested hypervisor's basic behavior. We already have a number of unit tests as part of the kvm-unit-tests framework. [0][1]

The scope of this project is to write unit tests for conditions that potentially let the level 2 guest kill or hang the level 1 guest or perhaps less importantly, the level 2 guest itself. First, we could visit known cases which have caused crashes/hangs in the past, and investigate whether we could write unit tests so we don't inadvertently introduce the same problems again when adding newer features and/or bug fixes. For example, a bug was introduced a while back that incorrectly injected L1's interrupts to L2 [2]. We could write a unit test that attempts to inject a few interrupts to L1(with help from L0) and verify that L1 indeed received all the interrupts. There are also cases[3] where emulate_invalid_guest_state=0 avoids hangs; which probably means that we could write tests that force certain paths during instruction emulation. This will probably also help with checks for BUG() at various places (and whether we should consider removing them with something less sinister :)) Another idea is to run nested unit tests with various values for kvm_intel module parameters - there have been cases where new commits break only if a certain feature is enabled/disabled.

Here's a tentative list of key hardening areas - 

 1. Writing unit tests for missing VMX features tests; eg- invvpid,
    load/store, posted interrupts. [1]
 2. Unit tests for emulated instructions[4] - can we write a test that 
    can hit a BUG() during instruction emulation ?
 3. A framework that can enable/disable various module parameters and run
    1. and 2.
 4. Unit tests for interrupt handling paths - can we write a test
    that times an interrupt injection such that results in an incorrect
     injection.
 5. Anything else that comes up as we start tackling these areas.

We should primarily focus on 1, 2 and 3 and get to the others if time permits.

Links:

Details:

  • Skill level: advanced
  • Language: C
  • Mentor: Bandan Das <bsd@redhat.com>
  • Suggested by: Bandan Das

QEMU projects

Integrate IDE ATAPI and SCSI CD-ROM emulation

Summary: Unify IDE ATAPI CD-ROM emulation with SCSI CD-ROM emulation to reduce code duplication and squash bugs

Currently the IDE ATAPI and SCSI CD-ROM emulated devices are two distinct implementations, and have different bugs/features. This leads to the situation that things which work when using the IDE emulation don't work when using the SCSI emulation and vice versa. So this project is for implementing a virtual ATA-to-SCSI bridge in QEMU, use this for emulating an IDE ATAPI drive, and merging the missing features from the IDE implementation into the SCSI one.

This project will allow you to expore IDE and SCSI storage protocols and learn how CD-ROM drives are operated.

Links:

Details:

  • Skill level: intermediate
  • Language: C
  • Mentor: Hannes Reinecke <hare@suse.de>, John Snow <jsnow@redhat.com> (jsnow on QEMU IRC)
  • Suggested by: Hannes Reinecke <hare@suse.de>

AMD IOMMU emulation

Summary: Rework existing AMD IOMMU emulation patches according to new IOMMU model of QEMU

There was already a proposal to add AMD IOMMU emulation in 2011, but QEMU was lacking a proper layer to integrate this effort smoothly. This is now possible like the successful integration of basic VT-d emulation demonstrated during last year's GSoC. So this task is a little bit simpler: study the old patches as well as the Intel version, port the old ones over the new interfaces, add ACPI tables (missing in the old version), ensure that the resulting IOMMU model works according to the AMD specification - and that guests like Linux/KVM can use it correctly. The model should also support interrupt remapping, but only for the user-space irqchip mode (kernel interface support is part of a different project proposal).

Links:

Details:

  • Skill level: intermediate to advanced
  • Language: C
  • Mentor: Jan Kiszka <jan.kiszka@web.de>
  • Suggested by: Jan Kiszka

Implement support for Mac OS 9 in QEMU

Summary: QEMU has gone a long way in emulating a Macintosh. But we can still improve. Adding support for Mac OS 9 would be a great imporovement. This would allow everyone who misses their older applications to be reacquainted with them. It would also expand QEMU's abilities.


Links:

Details:

  • Skill level: Advanced
  • Language: C
  • Mentor: Alexander Graf <agraf@suse.de>
  • Suggested by: John Arbuckle <programmingkidx@gmail.com>

Libvirt projects

Introducing job control to the storage driver

Summary: Currently, libvirt support job cancellation and progress reporting on domains. That is, if there's a long running job on a domain, e.g. migration, libvirt reports how much data has already been transferred to the destination and how much still needs to be transferred. However, libvirt lacks such information reporting in storage area, to which libvirt developers refer to as the storage driver. The aim is to report progress on several storage tasks, like volume wiping, file allocation an others.

  • Skill level: intermediate

Enhancing libvirt-designer

Summary: The project is in its very early stage of life. The libvirt-designer tries to ease generation of libvirt XML with coworking with libosinfo project. Contact me and we can find something suitable.

  • Skill level: beginer

Making virsh more bash like

Summary: If you have ever used virsh, you certainly reached the point where you stuggle with its user friendliness. Or unfriendliness I should rather say. Virsh is missing a lot of bash functionality that users consider natural: from automatic completion of object names, through redirecting command outputs through piping commands together. The aim would be to make these functions available in virsh and thus make user experience better.

  • Skill level: Advanced

Abstracting device address allocation

Summary: There are many types of addresses that devices can have in libvirt's XML description. Not all address types are properly assigned and checked for duplicates. The goal of this is to have an abstract data structure that would handle assigning all kinds of addresses, handle duplicates, etc.

  • Skill level: beginner

Admin interface APIs

Summary: We are currently working on an administrative interface for the libvirt daemon. This interface will be able to probe and change daemon settings live. Since this is going to be a new interface, there needs to be new APIs implemented. These APIs may do various interesting things strarting from showing the current logging filters up to force-disconnecting clients with certain properties. It is up to you what you would like to add, but of course this needs to be discussed and agreed on with the community.

  • Skill level: intermediate

Your own idea

Just catch me (Michal Privoznik) on IRC and we can discuss what interests you.

Links:

Details:

  • Component: libvirt
  • Skill level: (see description to each item)
  • Language: C
  • Mentor: Michal Privoznik <mprivozn@redhat.com>, mprivozn on IRC (#virt OFTC)
  • Suggested by: Michal Privoznik <mprivozn@redhat.com>

Running docker containers using virt-sandbox

Summary: The basic idea is to be able to grab container images from the docker registry and run them using virt-sandbox. The application could then run as either a KVM or LXC guest.

Daniel already has a proof of concept to download the docker images, but there is still quite some work to be done, like:

  • transforming the docker image into a qcow2 image,
  • getting libvirt to set the username of the process to run,
  • getting libvirt to set environment variables

Links:

Details:

  • Skill level: advanced
  • Language: C, possibly python too
  • Mentor: Cedric Bosdonnat <cbosdonnat@suse.com>; cbosdonnat on IRC (OFTC and freenode)
  • Suggested by: Cedric Bosdonnat

Project idea template

=== TITLE ===
 
 '''Summary:''' Short description of the project
 
 Detailed description of the project.
 
 '''Links:'''
 * Wiki links to relevant material
 * External links to mailing lists or web sites
 
 '''Details:'''
 * Skill level: beginner or intermediate or advanced
 * Language: C
 * Mentor: Email address and IRC nick
 * Suggested by: Person who suggested the idea

Information for mentors

Mentors are responsible for keeping in touch with their candidate and assessing the candidate's progress.

The mentor typically gives advice, reviews the candidate's code, and has regular communication with the candidate to ensure progress is being made.

Being a mentor is a significant time commitment, plan for 5 hours per week. Make sure you can make this commitment because backing out during the summer will affect the candidate's experience.

The mentor chooses their candidate by reviewing candidate application forms and conducting IRC interviews with candidates. Depending on the number of candidates, this can be time-consuming in itself. Choosing the right candidate is critical so that both the mentor and the candidate can have a successful experience.

Retrieved from "https://wiki.qemu.org/index.php?title=Outreachy_2015_MayAugust&oldid=4616"