Documentation/Platforms/S390X

QEMU s390x Guest Support

QEMU supports running s390x (aka zSystems aka mainframe) guests via the s390-virtio-ccw machine, with kvm or tcg acceleration.

Supported Devices

The s390-virtio-ccw machine emulates a virtual channel subsystem. Currently, only virtio-ccw proxy devices (for most virtio device types) are supported.

As a console device, supported types are SCLP VT220-compatible (preferred), SCLP line-mode or virtio-serial console.

For booting an s390x guest via network, see the dedicated page Features/S390xNetworkBoot.

Guest Operating Systems

Currently, the only supported guest operation system is Linux.

When using kvm, you should be able to use any recent distribution that you would be able to run on your host as well (RHEL, SLES, Ubuntu, Debian, Fedora). Older versions may not yet support virtio-ccw.

When using tcg with version 2.11 or before, the machine emulated is at an older level not supported on any of the recent enterprise distributions. Your best bet is probably using Debian.

When using tcg with version 2.12 or newer, the machine emulated should be good enough to run kernel 4.x based distributions. At least Fedora 26/27 and Ubuntu 18.04 (current build) have been reported to work.

Build Directions

 ./configure --target-list=s390x-softmmu && make

Minimal command-line

 qemu-system-s390x -M s390-ccw-virtio -kernel <kernel file name> -initrd <initrd file name> -m 512 -nographic

This command line will start a machine with the default SCLP VT220-compatible console. If you want any actual I/O devices, you need to add the respective virtio-*-ccw devices.

If you do not have a kernel and initrd image yet, you can have a try with the latest version from Debian stable for example: http://ftp.debian.org/debian/dists/stable/main/installer-s390x/current/images/generic/

Debian Install Example (TCG)

Having manually extracted (or downloaded) the kernel and initrd I successfully booted into the installer with the following:

 qemu-system-s390x -machine s390-ccw-virtio -cpu max,zpci=on -serial mon:stdio -display none -m 4096 \
   -nic user,model=virtio,hostfwd=tcp::2222-:22 \
   --cdrom debian-9.6.0-s390x-netinst.iso \
   -kernel kernel.s390 -initrd initrd.s390 \
   -drive file=/dev/zvol/zpool-0/debian-stretch-s390x,if=none,id=drive-virtio-disk0,format=raw,cache=none \
   -device virtio-blk-ccw,devno=fe.0.0001,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1,scsi=off

I had to manually specify the IP address (default for NAT is 10.0.2.15/10.0.2.2 with 10.0.2.3 for DNS, see [Documentation/Networking]

And then once installed I could drop the cdrom and direct kernel boot and start the system with:

 qemu-system-s390x -machine s390-ccw-virtio -cpu max,zpci=on -serial mon:stdio -display none -m 4096 \
   -nic user,model=virtio,hostfwd=tcp::2222-:22 \
   -drive file=/dev/zvol/zpool-0/debian-stretch-s390x,if=none,id=drive-virtio-disk0,format=raw,cache=none \
   -device virtio-blk-ccw,devno=fe.0.0001,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1,scsi=off

In this case I was using a block device via a ZFS pool, but a normal qcow2 device would work just as well.

Quick boot test with moonbuggy

If you want to simply test booting an s390x guest, a quick way to do so is grabbing the 'moonbuggy' image from the 2014 QEMU Advent Calendar: https://www.qemu-advent-calendar.org/2014/download/s390-moon-buggy.tar.xz

Extract the kernel and the initrd and then start a guest with

 qemu-system-s390x -M s390-ccw-virtio -nographic -kernel s390-bb.kernel -initrd s390-moon-buggy.initrd

You can of course also enjoy a round of the game while at it ;)

I/O devices

The virtio-ccw transport provides access to paravirtualized virtio devices. They are configured in a similar way to virtio-pci devices. For example,

 -drive file=/dev/dasdb,if=none,id=drive-virtio-disk0,format=raw,serial=ccwdasd1,cache=none \
 -device virtio-blk-ccw,devno=fe.0.0001,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1,scsi=off

will add a virtio-blk device that uses the host disk /dev/dasdb as backing and is seen in the guest as the channel device 0.0.0001.

Multiple subchannel sets are supported. For example,

 -device virtio-balloon-ccw,devno=fe.1.0101

will add a virtio-ballon device that the guest sees as the channel device 0.1.0101.

A note on PCI support

PCI devices are available on s390x, but they are not quite what you know from other platforms.

• PCI devices on s390x are not accessed via memory mapped I/O, but via special instructions that require kernel mode.
• No PCI topology; PCI functions are instead identified via a fid (function identifier) and a uid (unique identifier).
  • Any topology information you provide on the command line is basically invisible to the guest, as the device discovery instructions only return fid and uid.
  • You can specify fid and uid as properties of a zpci device that needs to be linked to the pci device.

As of 2.10, only devices passed through via vfio-pci have a chance of working. Prior to 2.10 (this includes compat machines), you may be able to add devices, but Linux guests will not use them.

As of 2.12, you should also be able to use MSI-X only virtio-pci devices, e.g. virtio-net-pci. This is also the first release that supports using pci with tcg on s390x.

PCI feature bits

PCI support is indicated as a cpu feature; Linux guests prior to kernel 4.15 also test for the AIS cpu feature (which requires support in the KVM kernel module; 4.12 or later). The emulated machine needs to be a zEC12 or later; for machines prior to z14, the bits need to be explicitly enabled. For 2.12 or later, the zpci feature bit can also be added to the qemu cpu model (under tcg, aen and ais are automatically enabled). For example,

 -cpu zEC12-base,zpci=on,aen=on,ais=on

turns on the PCI support, adapter-event notification and adapter interrupt suppression bits.

If you try to specify a PCI device for a machine without the PCI feature bit, for example

 -cpu zEC12-base,zpci=off (...) -device virtio-net-pci

qemu (2.11 or later) will fail with

 qemu-system-s390x: -device virtio-net-pci: No 'PCI' bus found for device 'virtio-net-pci'

This can be a bit confusing; but the root cause is the missing PCI feature bit.

Known limitations

• The -boot order=... command line parameter is not supported on s390x.
• Only the very first bootindex property is taken into consideration. If booting from the first device fails, no other device is tried afterwards.
• Memory hot-plugging is not supported.
• CPU hot-UNplugging is not supported.

Contact

Use the qemu-s390x mailing list or see the various S390 sections in the MAINTAINERS file of the QEMU sources.

Links

Wikipedia page for the System 390
Installing Ubuntu Linux
Installing SUSE Linux
Installing s390x emulator for Ubuntu
Debian S/390 port
Fedora s390x port