Features/Channel I/O Passthrough

Channel I/O is a high-performance input/output (I/O) architecture that is implemented (especially) on s390 (see the wikipedia article).

Motivation

In the past, a guest virtualized via QEMU/KVM on s390 only sees paravirtualized virtio devices via the "Virtio Over Channel I/O (virtio-ccw)" transport. This makes virtio devices discoverable via standard operating system algorithms for handling channel devices.

However this is not enough. To make the majority of devices on s390, which use the standard Channel I/O based mechanism, usable in a QEMU virtual machine, a passthrough mechanism is needed. This includes devices that don't have a virtio counterpart (e.g. tape drives) or that have specific characteristics which guests want to exploit.

For passing a device to a guest, this uses the same interface as everybody else, namely vfio. Thus, the vfio support for channel devices is introduced, and this new vfio device is named "vfio-ccw".

Implementation

vfio-ccw is realized with a mdev implementation. It has two drivers for two types of devices in the kernel:

• The vfio_ccw driver for the physical subchannel device.
• The vfio_mdev driver for the mediated vfio ccw device.

The QEMU part introduces a basic Channel I/O passthrough infrastructure based on vfio.

• Focus on supporting dasd-eckd (cu_type/dev_type = 0x3990/0x3390) as the target device currently.
• Support new QEMU parameters in the style of:

   -machine s390-ccw-virtio(,s390-squash-mcss=on|off) \
   -device vfio-ccw,sysfsdev=$MDEV_PATH

The new machine option s390-squash-mcss is added to squash e.g. passed-through channel devices from their real css (0-3, or 0 for hosts not activating MCSS-E) into the default css, as all virtio-ccw devices are in css 0xfe (and show up in the default css 0 for guests not activating MCSS-E).

Setup

This example setup is done for a Linux guest running in an s390x-ccw-virtio machine.

Host

• Kernel Configuration

 CONFIG_S390_CCW_IOMMU=m
 CONFIG_VFIO=m
 CONFIG_VFIO_MDEV=m
 CONFIG_VFIO_MDEV_DEVICE=m
 CONFIG_VFIO_CCW=m

• Modules Required

 modprobe vfio.ko
 modprobe mdev.ko
 modprobe vfio_mdev.ko
 modprobe vfio_iommu_type1.ko
 modprobe vfio_ccw.ko

• You need to have a DASD/ECKD device as the target device.

Find the subchannel (0."$ssid"."$schid") of your target DASD/ECKD device and bind the subchannel to the vfio_ccw driver.

 #find the DASD you can use with lsdasd on your host, e.g.:
 #devno="7e52"
 #schid="16ca"
 #ssid="0"
 #for device 0.0.7e52 on subchannel 0.0.16ca.
 #unbind the CCW device from its subchannel
 echo 0."$ssid"."$devno" > /sys/bus/ccw/devices/0."$ssid"."$devno"/driver/unbind
 #unbind the subchannel from the I/O subchannel driver
 echo 0."$ssid"."$schid" > /sys/bus/css/devices/0."$ssid"."$schid"/driver/unbind
 #bind the subchannel to the vfio_ccw driver
 echo 0."$ssid"."$schid" > /sys/bus/css/drivers/vfio_ccw/bind

• Create a Mediated Device for the physical device

 #generate a uuid with uuidgen. e.g.:
 #uuid="6dfd3ec5-e8b3-4e18-a6fe-57bc9eceb920"
 echo "$uuid" > /sys/bus/css/devices/0."$ssid"."$schid"/mdev_supported_types/vfio_ccw-io/create

Starting QEMU

• Add a vfio-ccw device to your QEMU command line (machine needs to be s390x-ccw-virtio, with s390-squash-mcss=on). Example:

 -M s390-ccw-virtio,s390-squash-mcss=on \
 -device vfio-ccw,devno=0.0.1234,sysfsdev=/sys/bus/mdev/devices/$uuid \
 ... ...

• Start QEMU. Your guest will be presented with a real Channel I/O device, with the example above, to be more specific, a DASD/ECKD device.

Try the device on your Guest

• The guest will see the DASD/ECKD as a channel-attached device. With the example above, the device will show up as 0.0.1234 (verify e.g. via the lscss tool).

• You can online the device by calling chccwdev -e 0.0.1234 and use it as a block device.

Restrictions

• Only target I/O subchannels.
• Only basic commands (read/write) have been tested.
• Some commands may need special handling in the future, for example, anything related to path grouping.