code:
===============================
Adjunct Processor (AP) facility
===============================

Introduction
============
The Adjunct Processor (AP) facility is an IBM Z cryptographic facility comprised
of three AP instructions and from 1 up to 256 PCIe cryptographic adapter cards.
The AP devices provide cryptographic functions to all CPUs assigned to a
linux system running in an IBM Z system LPAR.

The AP adapter cards are exposed via the AP bus. The motivation for vfio-ap
is to make AP cards available to KVM guests using the VFIO mediated device
framework. This implementation relies considerably on the s390 virtualization
facilities which do most of the hard work of providing direct access to AP
devices.

AP Architectural Overview
=========================
To facilitate the comprehension of the design, let's start with some
definitions:

* AP adapter

  An AP adapter is an IBM Z adapter card that can perform cryptographic
  functions. There can be from 0 to 256 adapters assigned to an LPAR. Adapters
  assigned to the LPAR in which a linux host is running will be available to
  the linux host. Each adapter is identified by a number from 0 to 255; however,
  the maximum adapter number is determined by machine model and/or adapter type.
  When installed, an AP adapter is accessed by AP instructions executed by any
  CPU.

  The AP adapter cards are assigned to a given LPAR via the system's Activation
  Profile which can be edited via the HMC. When the linux host system is IPL'd
  in the LPAR, the AP bus detects the AP adapter cards assigned to the LPAR and
  creates a sysfs device for each assigned adapter. For example, if AP adapters
  4 and 10 (0x0a) are assigned to the LPAR, the AP bus will create the following
  sysfs device entries::

    /sys/devices/ap/card04
    /sys/devices/ap/card0a

  Symbolic links to these devices will also be created in the AP bus devices
  sub-directory::

    /sys/bus/ap/devices/[card04]
    /sys/bus/ap/devices/[card04]

* AP domain

  An adapter is partitioned into domains. An adapter can hold up to 256 domains
  depending upon the adapter type and hardware configuration. A domain is
  identified by a number from 0 to 255; however, the maximum domain number is
  determined by machine model and/or adapter type.. A domain can be thought of
  as a set of hardware registers and memory used for processing AP commands. A
  domain can be configured with a secure private key used for clear key
  encryption. A domain is classified in one of two ways depending upon how it
  may be accessed:

    * Usage domains are domains that are targeted by an AP instruction to
      process an AP command.

    * Control domains are domains that are changed by an AP command sent to a
      usage domain; for example, to set the secure private key for the control
      domain.

  The AP usage and control domains are assigned to a given LPAR via the system's
  Activation Profile which can be edited via the HMC. When a linux host system
  is IPL'd in the LPAR, the AP bus module detects the AP usage and control
  domains assigned to the LPAR. The domain number of each usage domain and
  adapter number of each AP adapter are combined to create AP queue devices
  (see AP Queue section below). The domain number of each control domain will be
  represented in a bitmask and stored in a sysfs file
  /sys/bus/ap/ap_control_domain_mask. The bits in the mask, from most to least
  significant bit, correspond to domains 0-255.

* AP Queue

  An AP queue is the means by which an AP command is sent to a usage domain
  inside a specific adapter. An AP queue is identified by a tuple
  comprised of an AP adapter ID (APID) and an AP queue index (APQI). The
  APQI corresponds to a given usage domain number within the adapter. This tuple
  forms an AP Queue Number (APQN) uniquely identifying an AP queue. AP
  instructions include a field containing the APQN to identify the AP queue to
  which the AP command is to be sent for processing.

  The AP bus will create a sysfs device for each APQN that can be derived from
  the cross product of the AP adapter and usage domain numbers detected when the
  AP bus module is loaded. For example, if adapters 4 and 10 (0x0a) and usage
  domains 6 and 71 (0x47) are assigned to the LPAR, the AP bus will create the
  following sysfs entries::

    /sys/devices/ap/card04/04.0006
    /sys/devices/ap/card04/04.0047
    /sys/devices/ap/card0a/0a.0006
    /sys/devices/ap/card0a/0a.0047

  The following symbolic links to these devices will be created in the AP bus
  devices subdirectory::

    /sys/bus/ap/devices/[04.0006]
    /sys/bus/ap/devices/[04.0047]
    /sys/bus/ap/devices/[0a.0006]
    /sys/bus/ap/devices/[0a.0047]

* AP Instructions:

  There are three AP instructions:

  * NQAP: to enqueue an AP command-request message to a queue
  * DQAP: to dequeue an AP command-reply message from a queue
  * PQAP: to administer the queues

  AP instructions identify the domain that is targeted to process the AP
  command; this must be one of the usage domains. An AP command may modify a
  domain that is not one of the usage domains, but the modified domain
  must be one of the control domains.

AP and SIE
==========
Let's now take a look at how AP instructions executed on a guest are interpreted
by the hardware.

A satellite control block called the Crypto Control Block (CRYCB) is attached to
our main hardware virtualization control block. The CRYCB contains an AP Control
Block (APCB) that has three fields to identify the adapters, usage domains and
control domains assigned to the KVM guest:

* The AP Mask (APM) field is a bit mask that identifies the AP adapters assigned
  to the KVM guest. Each bit in the mask, from left to right, corresponds to
  an APID from 0-255. If a bit is set, the corresponding adapter is valid for
  use by the KVM guest.

* The AP Queue Mask (AQM) field is a bit mask identifying the AP usage domains
  assigned to the KVM guest. Each bit in the mask, from left to right,
  corresponds to an AP queue index (APQI) from 0-255. If a bit is set, the
  corresponding queue is valid for use by the KVM guest.

* The AP Domain Mask field is a bit mask that identifies the AP control domains
  assigned to the KVM guest. The ADM bit mask controls which domains can be
  changed by an AP command-request message sent to a usage domain from the
  guest. Each bit in the mask, from left to right, corresponds to a domain from
  0-255. If a bit is set, the corresponding domain can be modified by an AP
  command-request message sent to a usage domain.

If you recall from the description of an AP Queue, AP instructions include
an APQN to identify the AP queue to which an AP command-request message is to be
sent (NQAP and PQAP instructions), or from which a command-reply message is to
be received (DQAP instruction). The validity of an APQN is defined by the matrix
calculated from the APM and AQM; it is the Cartesian product of all assigned
adapter numbers (APM) with all assigned queue indexes (AQM). For example, if
adapters 1 and 2 and usage domains 5 and 6 are assigned to a guest, the APQNs
(1,5), (1,6), (2,5) and (2,6) will be valid for the guest.

The APQNs can provide secure key functionality - i.e., a private key is stored
on the adapter card for each of its domains - so each APQN must be assigned to
at most one guest or to the linux host::

   Example 1: Valid configuration:
   ------------------------------
   Guest1: adapters 1,2  domains 5,6
   Guest2: adapter  1,2  domain 7

   This is valid because both guests have a unique set of APQNs:
      Guest1 has APQNs (1,5), (1,6), (2,5), (2,6);
      Guest2 has APQNs (1,7), (2,7)

   Example 2: Valid configuration:
   ------------------------------
   Guest1: adapters 1,2 domains 5,6
   Guest2: adapters 3,4 domains 5,6

   This is also valid because both guests have a unique set of APQNs:
      Guest1 has APQNs (1,5), (1,6), (2,5), (2,6);
      Guest2 has APQNs (3,5), (3,6), (4,5), (4,6)

   Example 3: Invalid configuration:
   --------------------------------
   Guest1: adapters 1,2  domains 5,6
   Guest2: adapter  1    domains 6,7

   This is an invalid configuration because both guests have access to
   APQN (1,6).

The Design
==========
The design introduces three new objects:

1. AP matrix device
2. VFIO AP device driver (vfio_ap.ko)
3. VFIO AP mediated pass-through device

The VFIO AP device driver
-------------------------
The VFIO AP (vfio_ap) device driver serves the following purposes:

1. Provides the interfaces to secure APQNs for exclusive use of KVM guests.

2. Sets up the VFIO mediated device interfaces to manage a vfio_ap mediated
   device and creates the sysfs interfaces for assigning adapters, usage
   domains, and control domains comprising the matrix for a KVM guest.

3. Configures the APM, AQM and ADM in the APCB contained in the CRYCB referenced
   by a KVM guest's SIE state description to grant the guest access to a matrix
   of AP devices

Reserve APQNs for exclusive use of KVM guests
---------------------------------------------
The following block diagram illustrates the mechanism by which APQNs are
reserved::

				+------------------+
		 7 remove       |                  |
	   +--------------------> cex4queue driver |
	   |                    |                  |
	   |                    +------------------+
	   |
	   |
	   |                    +------------------+          +----------------+
	   |  5 register driver |                  | 3 create |                |
	   |   +---------------->   Device core    +---------->  matrix device |
	   |   |                |                  |          |                |
	   |   |                +--------^---------+          +----------------+
	   |   |                         |
	   |   |                         +-------------------+
	   |   | +-----------------------------------+       |
	   |   | |      4 register AP driver         |       | 2 register device
	   |   | |                                   |       |
  +--------+---+-v---+                      +--------+-------+-+
  |                  |                      |                  |
  |      ap_bus      +--------------------- >  vfio_ap driver  |
  |                  |       8 probe        |                  |
  +--------^---------+                      +--^--^------------+
  6 edit   |                                   |  |
    apmask |     +-----------------------------+  | 11 mdev create
    aqmask |     |           1 modprobe           |
  +--------+-----+---+           +----------------+-+         +----------------+
  |                  |           |                  |10 create|     mediated   |
  |      admin       |           | VFIO device core |--------->     matrix     |
  |                  +           |                  |         |     device     |
  +------+-+---------+           +--------^---------+         +--------^-------+
	 | |                              |                            |
	 | | 9 create vfio_ap-passthrough |                            |
	 | +------------------------------+                            |
	 +-------------------------------------------------------------+
		     12  assign adapter/domain/control domain

The process for reserving an AP queue for use by a KVM guest is:

1. The administrator loads the vfio_ap device driver
2. The vfio-ap driver during its initialization will register a single 'matrix'
   device with the device core. This will serve as the parent device for
   all vfio_ap mediated devices used to configure an AP matrix for a guest.
3. The /sys/devices/vfio_ap/matrix device is created by the device core
4. The vfio_ap device driver will register with the AP bus for AP queue devices
   of type 10 and higher (CEX4 and newer). The driver will provide the vfio_ap
   driver's probe and remove callback interfaces. Devices older than CEX4 queues
   are not supported to simplify the implementation by not needlessly
   complicating the design by supporting older devices that will go out of
   service in the relatively near future, and for which there are few older
   systems around on which to test.
5. The AP bus registers the vfio_ap device driver with the device core
6. The administrator edits the AP adapter and queue masks to reserve AP queues
   for use by the vfio_ap device driver.
7. The AP bus removes the AP queues reserved for the vfio_ap driver from the
   default zcrypt cex4queue driver.
8. The AP bus probes the vfio_ap device driver to bind the queues reserved for
   it.
9. The administrator creates a passthrough type vfio_ap mediated device to be
   used by a guest
10. The administrator assigns the adapters, usage domains and control domains
    to be exclusively used by a guest.

Set up the VFIO mediated device interfaces
------------------------------------------
The VFIO AP device driver utilizes the common interfaces of the VFIO mediated
device core driver to:

* Register an AP mediated bus driver to add a vfio_ap mediated device to and
  remove it from a VFIO group.
* Create and destroy a vfio_ap mediated device
* Add a vfio_ap mediated device to and remove it from the AP mediated bus driver
* Add a vfio_ap mediated device to and remove it from an IOMMU group

The following high-level block diagram shows the main components and interfaces
of the VFIO AP mediated device driver::

   +-------------+
   |             |
   | +---------+ | mdev_register_driver() +--------------+
   | |  Mdev   | +<-----------------------+              |
   | |  bus    | |                        | vfio_mdev.ko |
   | | driver  | +----------------------->+              |<-> VFIO user
   | +---------+ |    probe()/remove()    +--------------+    APIs
   |             |
   |  MDEV CORE  |
   |   MODULE    |
   |   mdev.ko   |
   | +---------+ | mdev_register_parent() +--------------+
   | |Physical | +<-----------------------+              |
   | | device  | |                        |  vfio_ap.ko  |<-> matrix
   | |interface| +----------------------->+              |    device
   | +---------+ |       callback         +--------------+
   +-------------+

During initialization of the vfio_ap module, the matrix device is registered
with an 'mdev_parent_ops' structure that provides the sysfs attribute
structures, mdev functions and callback interfaces for managing the mediated
matrix device.

* sysfs attribute structures:

  supported_type_groups
    The VFIO mediated device framework supports creation of user-defined
    mediated device types. These mediated device types are specified
    via the 'supported_type_groups' structure when a device is registered
    with the mediated device framework. The registration process creates the
    sysfs structures for each mediated device type specified in the
    'mdev_supported_types' sub-directory of the device being registered. Along
    with the device type, the sysfs attributes of the mediated device type are
    provided.

    The VFIO AP device driver will register one mediated device type for
    passthrough devices:

      /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough

    Only the read-only attributes required by the VFIO mdev framework will
    be provided::

	... name
	... device_api
	... available_instances
	... device_api

    Where:

	* name:
	    specifies the name of the mediated device type
	* device_api:
	    the mediated device type's API
	* available_instances:
	    the number of vfio_ap mediated passthrough devices
	    that can be created
	* device_api:
	    specifies the VFIO API
  mdev_attr_groups
    This attribute group identifies the user-defined sysfs attributes of the
    mediated device. When a device is registered with the VFIO mediated device
    framework, the sysfs attribute files identified in the 'mdev_attr_groups'
    structure will be created in the vfio_ap mediated device's directory. The
    sysfs attributes for a vfio_ap mediated device are:

    assign_adapter / unassign_adapter:
      Write-only attributes for assigning/unassigning an AP adapter to/from the
      vfio_ap mediated device. To assign/unassign an adapter, the APID of the
      adapter is echoed into the respective attribute file.
    assign_domain / unassign_domain:
      Write-only attributes for assigning/unassigning an AP usage domain to/from
      the vfio_ap mediated device. To assign/unassign a domain, the domain
      number of the usage domain is echoed into the respective attribute
      file.
    matrix:
      A read-only file for displaying the APQNs derived from the Cartesian
      product of the adapter and domain numbers assigned to the vfio_ap mediated
      device.
    guest_matrix:
      A read-only file for displaying the APQNs derived from the Cartesian
      product of the adapter and domain numbers assigned to the APM and AQM
      fields respectively of the KVM guest's CRYCB. This may differ from the
      the APQNs assigned to the vfio_ap mediated device if any APQN does not
      reference a queue device bound to the vfio_ap device driver (i.e., the
      queue is not in the host's AP configuration).
    assign_control_domain / unassign_control_domain:
      Write-only attributes for assigning/unassigning an AP control domain
      to/from the vfio_ap mediated device. To assign/unassign a control domain,
      the ID of the domain to be assigned/unassigned is echoed into the
      respective attribute file.
    control_domains:
      A read-only file for displaying the control domain numbers assigned to the
      vfio_ap mediated device.

* functions:

  create:
    allocates the ap_matrix_mdev structure used by the vfio_ap driver to:

    * Store the reference to the KVM structure for the guest using the mdev
    * Store the AP matrix configuration for the adapters, domains, and control
      domains assigned via the corresponding sysfs attributes files
    * Store the AP matrix configuration for the adapters, domains and control
      domains available to a guest. A guest may not be provided access to APQNs
      referencing queue devices that do not exist, or are not bound to the
      vfio_ap device driver.

  remove:
    deallocates the vfio_ap mediated device's ap_matrix_mdev structure.
    This will be allowed only if a running guest is not using the mdev.

* callback interfaces

  open_device:
    The vfio_ap driver uses this callback to register a
    VFIO_GROUP_NOTIFY_SET_KVM notifier callback function for the matrix mdev
    devices. The open_device callback is invoked by userspace to connect the
    VFIO iommu group for the matrix mdev device to the MDEV bus. Access to the
    KVM structure used to configure the KVM guest is provided via this callback.
    The KVM structure, is used to configure the guest's access to the AP matrix
    defined via the vfio_ap mediated device's sysfs attribute files.

  close_device:
    unregisters the VFIO_GROUP_NOTIFY_SET_KVM notifier callback function for the
    matrix mdev device and deconfigures the guest's AP matrix.

  ioctl:
    this callback handles the VFIO_DEVICE_GET_INFO and VFIO_DEVICE_RESET ioctls
    defined by the vfio framework.

Configure the guest's AP resources
----------------------------------
Configuring the AP resources for a KVM guest will be performed when the
VFIO_GROUP_NOTIFY_SET_KVM notifier callback is invoked. The notifier
function is called when userspace connects to KVM. The guest's AP resources are
configured via it's APCB by:

* Setting the bits in the APM corresponding to the APIDs assigned to the
  vfio_ap mediated device via its 'assign_adapter' interface.
* Setting the bits in the AQM corresponding to the domains assigned to the
  vfio_ap mediated device via its 'assign_domain' interface.
* Setting the bits in the ADM corresponding to the domain dIDs assigned to the
  vfio_ap mediated device via its 'assign_control_domains' interface.

The linux device model precludes passing a device through to a KVM guest that
is not bound to the device driver facilitating its pass-through. Consequently,
an APQN that does not reference a queue device bound to the vfio_ap device
driver will not be assigned to a KVM guest's matrix. The AP architecture,
however, does not provide a means to filter individual APQNs from the guest's
matrix, so the adapters, domains and control domains assigned to vfio_ap
mediated device via its sysfs 'assign_adapter', 'assign_domain' and
'assign_control_domain' interfaces will be filtered before providing the AP
configuration to a guest:

* The APIDs of the adapters, the APQIs of the domains and the domain numbers of
  the control domains assigned to the matrix mdev that are not also assigned to
  the host's AP configuration will be filtered.

* Each APQN derived from the Cartesian product of the APIDs and APQIs assigned
  to the vfio_ap mdev is examined and if any one of them does not reference a
  queue device bound to the vfio_ap device driver, the adapter will not be
  plugged into the guest (i.e., the bit corresponding to its APID will not be
  set in the APM of the guest's APCB).

The CPU model features for AP
-----------------------------
The AP stack relies on the presence of the AP instructions as well as three
facilities: The AP Facilities Test (APFT) facility; the AP Query
Configuration Information (QCI) facility; and the AP Queue Interruption Control
facility. These features/facilities are made available to a KVM guest via the
following CPU model features:

1. ap: Indicates whether the AP instructions are installed on the guest. This
   feature will be enabled by KVM only if the AP instructions are installed
   on the host.

2. apft: Indicates the APFT facility is available on the guest. This facility
   can be made available to the guest only if it is available on the host (i.e.,
   facility bit 15 is set).

3. apqci: Indicates the AP QCI facility is available on the guest. This facility
   can be made available to the guest only if it is available on the host (i.e.,
   facility bit 12 is set).

4. apqi: Indicates AP Queue Interruption Control faclity is available on the
   guest. This facility can be made available to the guest only if it is
   available on the host (i.e., facility bit 65 is set).

Note: If the user chooses to specify a CPU model different than the 'host'
model to QEMU, the CPU model features and facilities need to be turned on
explicitly; for example::

     /usr/bin/qemu-system-s390x ... -cpu z13,ap=on,apqci=on,apft=on,apqi=on

A guest can be precluded from using AP features/facilities by turning them off
explicitly; for example::

     /usr/bin/qemu-system-s390x ... -cpu host,ap=off,apqci=off,apft=off,apqi=off

Note: If the APFT facility is turned off (apft=off) for the guest, the guest
will not see any AP devices. The zcrypt device drivers on the guest that
register for type 10 and newer AP devices - i.e., the cex4card and cex4queue
device drivers - need the APFT facility to ascertain the facilities installed on
a given AP device. If the APFT facility is not installed on the guest, then no
adapter or domain devices will get created by the AP bus running on the
guest because only type 10 and newer devices can be configured for guest use.

Example
=======
Let's now provide an example to illustrate how KVM guests may be given
access to AP facilities. For this example, we will show how to configure
three guests such that executing the lszcrypt command on the guests would
look like this:

Guest1
------
=========== ===== ============
CARD.DOMAIN TYPE  MODE
=========== ===== ============
05          CEX5C CCA-Coproc
05.0004     CEX5C CCA-Coproc
05.00ab     CEX5C CCA-Coproc
06          CEX5A Accelerator
06.0004     CEX5A Accelerator
06.00ab     CEX5A Accelerator
=========== ===== ============

Guest2
------
=========== ===== ============
CARD.DOMAIN TYPE  MODE
=========== ===== ============
05          CEX5C CCA-Coproc
05.0047     CEX5C CCA-Coproc
05.00ff     CEX5C CCA-Coproc
=========== ===== ============

Guest3
------
=========== ===== ============
CARD.DOMAIN TYPE  MODE
=========== ===== ============
06          CEX5A Accelerator
06.0047     CEX5A Accelerator
06.00ff     CEX5A Accelerator
=========== ===== ============

These are the steps:

1. Install the vfio_ap module on the linux host. The dependency chain for the
   vfio_ap module is:
   * iommu
   * s390
   * zcrypt
   * vfio
   * vfio_mdev
   * vfio_mdev_device
   * KVM

   To build the vfio_ap module, the kernel build must be configured with the
   following Kconfig elements selected:
   * IOMMU_SUPPORT
   * S390
   * ZCRYPT
   * S390_AP_IOMMU
   * VFIO
   * VFIO_MDEV
   * KVM

   If using make menuconfig select the following to build the vfio_ap module::

     -> Device Drivers
	-> IOMMU Hardware Support
	   select S390 AP IOMMU Support
	-> VFIO Non-Privileged userspace driver framework
	   -> Mediated device driver frramework
	      -> VFIO driver for Mediated devices
     -> I/O subsystem
	-> VFIO support for AP devices

2. Secure the AP queues to be used by the three guests so that the host can not
   access them. To secure them, there are two sysfs files that specify
   bitmasks marking a subset of the APQN range as usable only by the default AP
   queue device drivers. All remaining APQNs are available for use by
   any other device driver. The vfio_ap device driver is currently the only
   non-default device driver. The location of the sysfs files containing the
   masks are::

     /sys/bus/ap/apmask
     /sys/bus/ap/aqmask

   The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs
   (APID). Each bit in the mask, from left to right, corresponds to an APID from
   0-255. If a bit is set, the APID belongs to the subset of APQNs marked as
   available only to the default AP queue device drivers.

   The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes
   (APQI). Each bit in the mask, from left to right, corresponds to an APQI from
   0-255. If a bit is set, the APQI belongs to the subset of APQNs marked as
   available only to the default AP queue device drivers.

   The Cartesian product of the APIDs corresponding to the bits set in the
   apmask and the APQIs corresponding to the bits set in the aqmask comprise
   the subset of APQNs that can be used only by the host default device drivers.
   All other APQNs are available to the non-default device drivers such as the
   vfio_ap driver.

   Take, for example, the following masks::

      apmask:
      0x7d00000000000000000000000000000000000000000000000000000000000000

      aqmask:
      0x8000000000000000000000000000000000000000000000000000000000000000

   The masks indicate:

   * Adapters 1, 2, 3, 4, 5, and 7 are available for use by the host default
     device drivers.

   * Domain 0 is available for use by the host default device drivers

   * The subset of APQNs available for use only by the default host device
     drivers are:

     (1,0), (2,0), (3,0), (4.0), (5,0) and (7,0)

   * All other APQNs are available for use by the non-default device drivers.

   The APQN of each AP queue device assigned to the linux host is checked by the
   AP bus against the set of APQNs derived from the Cartesian product of APIDs
   and APQIs marked as available to the default AP queue device drivers. If a
   match is detected,  only the default AP queue device drivers will be probed;
   otherwise, the vfio_ap device driver will be probed.

   By default, the two masks are set to reserve all APQNs for use by the default
   AP queue device drivers. There are two ways the default masks can be changed:

   1. The sysfs mask files can be edited by echoing a string into the
      respective sysfs mask file in one of two formats:

      * An absolute hex string starting with 0x - like "0x12345678" - sets
	the mask. If the given string is shorter than the mask, it is padded
	with 0s on the right; for example, specifying a mask value of 0x41 is
	the same as specifying::

	   0x4100000000000000000000000000000000000000000000000000000000000000

	Keep in mind that the mask reads from left to right, so the mask
	above identifies device numbers 1 and 7 (01000001).

	If the string is longer than the mask, the operation is terminated with
	an error (EINVAL).

      * Individual bits in the mask can be switched on and off by specifying
	each bit number to be switched in a comma separated list. Each bit
	number string must be prepended with a ('+') or minus ('-') to indicate
	the corresponding bit is to be switched on ('+') or off ('-'). Some
	valid values are:

	   - "+0"    switches bit 0 on
	   - "-13"   switches bit 13 off
	   - "+0x41" switches bit 65 on
	   - "-0xff" switches bit 255 off

	The following example:

	      +0,-6,+0x47,-0xf0

	Switches bits 0 and 71 (0x47) on

	Switches bits 6 and 240 (0xf0) off

	Note that the bits not specified in the list remain as they were before
	the operation.

   2. The masks can also be changed at boot time via parameters on the kernel
      command line like this:

	 ap.apmask=0xffff ap.aqmask=0x40

	 This would create the following masks::

	    apmask:
	    0xffff000000000000000000000000000000000000000000000000000000000000

	    aqmask:
	    0x4000000000000000000000000000000000000000000000000000000000000000

	 Resulting in these two pools::

	    default drivers pool:    adapter 0-15, domain 1
	    alternate drivers pool:  adapter 16-255, domains 0, 2-255

   **Note:**
   Changing a mask such that one or more APQNs will be taken from a vfio_ap
   mediated device (see below) will fail with an error (EBUSY). A message
   is logged to the kernel ring buffer which can be viewed with the 'dmesg'
   command. The output identifies each APQN flagged as 'in use' and identifies
   the vfio_ap mediated device to which it is assigned; for example:

   Userspace may not re-assign queue 05.0054 already assigned to 62177883-f1bb-47f0-914d-32a22e3a8804
   Userspace may not re-assign queue 04.0054 already assigned to cef03c3c-903d-4ecc-9a83-40694cb8aee4

Securing the APQNs for our example
----------------------------------
   To secure the AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, 06.0047,
   06.00ab, and 06.00ff for use by the vfio_ap device driver, the corresponding
   APQNs can be removed from the default masks using either of the following
   commands::

      echo -5,-6 > /sys/bus/ap/apmask

      echo -4,-0x47,-0xab,-0xff > /sys/bus/ap/aqmask

   Or the masks can be set as follows::

      echo 0xf9ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff \
      > apmask

      echo 0xf7fffffffffffffffeffffffffffffffffffffffffeffffffffffffffffffffe \
      > aqmask

   This will result in AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004,
   06.0047, 06.00ab, and 06.00ff getting bound to the vfio_ap device driver. The
   sysfs directory for the vfio_ap device driver will now contain symbolic links
   to the AP queue devices bound to it::

     /sys/bus/ap
     ... [drivers]
     ...... [vfio_ap]
     ......... [05.0004]
     ......... [05.0047]
     ......... [05.00ab]
     ......... [05.00ff]
     ......... [06.0004]
     ......... [06.0047]
     ......... [06.00ab]
     ......... [06.00ff]

   Keep in mind that only type 10 and newer adapters (i.e., CEX4 and later)
   can be bound to the vfio_ap device driver. The reason for this is to
   simplify the implementation by not needlessly complicating the design by
   supporting older devices that will go out of service in the relatively near
   future and for which there are few older systems on which to test.

   The administrator, therefore, must take care to secure only AP queues that
   can be bound to the vfio_ap device driver. The device type for a given AP
   queue device can be read from the parent card's sysfs directory. For example,
   to see the hardware type of the queue 05.0004:

     cat /sys/bus/ap/devices/card05/hwtype

   The hwtype must be 10 or higher (CEX4 or newer) in order to be bound to the
   vfio_ap device driver.

3. Create the mediated devices needed to configure the AP matrixes for the
   three guests and to provide an interface to the vfio_ap driver for
   use by the guests::

     /sys/devices/vfio_ap/matrix/
     --- [mdev_supported_types]
     ------ [vfio_ap-passthrough] (passthrough vfio_ap mediated device type)
     --------- create
     --------- [devices]

   To create the mediated devices for the three guests::

	uuidgen > create
	uuidgen > create
	uuidgen > create

	or

	echo $uuid1 > create
	echo $uuid2 > create
	echo $uuid3 > create

   This will create three mediated devices in the [devices] subdirectory named
   after the UUID written to the create attribute file. We call them $uuid1,
   $uuid2 and $uuid3 and this is the sysfs directory structure after creation::

     /sys/devices/vfio_ap/matrix/
     --- [mdev_supported_types]
     ------ [vfio_ap-passthrough]
     --------- [devices]
     ------------ [$uuid1]
     --------------- assign_adapter
     --------------- assign_control_domain
     --------------- assign_domain
     --------------- matrix
     --------------- unassign_adapter
     --------------- unassign_control_domain
     --------------- unassign_domain

     ------------ [$uuid2]
     --------------- assign_adapter
     --------------- assign_control_domain
     --------------- assign_domain
     --------------- matrix
     --------------- unassign_adapter
     ----------------unassign_control_domain
     ----------------unassign_domain

     ------------ [$uuid3]
     --------------- assign_adapter
     --------------- assign_control_domain
     --------------- assign_domain
     --------------- matrix
     --------------- unassign_adapter
     ----------------unassign_control_domain
     ----------------unassign_domain

   Note *****: The vfio_ap mdevs do not persist across reboots unless the
               mdevctl tool is used to create and persist them.

4. The administrator now needs to configure the matrixes for the mediated
   devices $uuid1 (for Guest1), $uuid2 (for Guest2) and $uuid3 (for Guest3).

   This is how the matrix is configured for Guest1::

      echo 5 > assign_adapter
      echo 6 > assign_adapter
      echo 4 > assign_domain
      echo 0xab > assign_domain

   Control domains can similarly be assigned using the assign_control_domain
   sysfs file.

   If a mistake is made configuring an adapter, domain or control domain,
   you can use the unassign_xxx files to unassign the adapter, domain or
   control domain.

   To display the matrix configuration for Guest1::

	 cat matrix

   To display the matrix that is or will be assigned to Guest1::

	 cat guest_matrix

   This is how the matrix is configured for Guest2::

      echo 5 > assign_adapter
      echo 0x47 > assign_domain
      echo 0xff > assign_domain

   This is how the matrix is configured for Guest3::

      echo 6 > assign_adapter
      echo 0x47 > assign_domain
      echo 0xff > assign_domain

   In order to successfully assign an adapter:

   * The adapter number specified must represent a value from 0 up to the
     maximum adapter number configured for the system. If an adapter number
     higher than the maximum is specified, the operation will terminate with
     an error (ENODEV).

     Note: The maximum adapter number can be obtained via the sysfs
	   /sys/bus/ap/ap_max_adapter_id attribute file.

   * Each APQN derived from the Cartesian product of the APID of the adapter
     being assigned and the APQIs of the domains previously assigned:

     - Must only be available to the vfio_ap device driver as specified in the
       sysfs /sys/bus/ap/apmask and /sys/bus/ap/aqmask attribute files. If even
       one APQN is reserved for use by the host device driver, the operation
       will terminate with an error (EADDRNOTAVAIL).

     - Must NOT be assigned to another vfio_ap mediated device. If even one APQN
       is assigned to another vfio_ap mediated device, the operation will
       terminate with an error (EBUSY).

     - Must NOT be assigned while the sysfs /sys/bus/ap/apmask and
       sys/bus/ap/aqmask attribute files are being edited or the operation may
       terminate with an error (EBUSY).

   In order to successfully assign a domain:

   * The domain number specified must represent a value from 0 up to the
     maximum domain number configured for the system. If a domain number
     higher than the maximum is specified, the operation will terminate with
     an error (ENODEV).

     Note: The maximum domain number can be obtained via the sysfs
	   /sys/bus/ap/ap_max_domain_id attribute file.

    * Each APQN derived from the Cartesian product of the APQI of the domain
      being assigned and the APIDs of the adapters previously assigned:

     - Must only be available to the vfio_ap device driver as specified in the
       sysfs /sys/bus/ap/apmask and /sys/bus/ap/aqmask attribute files. If even
       one APQN is reserved for use by the host device driver, the operation
       will terminate with an error (EADDRNOTAVAIL).

     - Must NOT be assigned to another vfio_ap mediated device. If even one APQN
       is assigned to another vfio_ap mediated device, the operation will
       terminate with an error (EBUSY).

     - Must NOT be assigned while the sysfs /sys/bus/ap/apmask and
       sys/bus/ap/aqmask attribute files are being edited or the operation may
       terminate with an error (EBUSY).

   In order to successfully assign a control domain:

   * The domain number specified must represent a value from 0 up to the maximum
     domain number configured for the system. If a control domain number higher
     than the maximum is specified, the operation will terminate with an
     error (ENODEV).

5. Start Guest1::

     /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on,apqi=on \
	-device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid1 ...

7. Start Guest2::

     /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on,apqi=on \
	-device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid2 ...

7. Start Guest3::

     /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on,apqi=on \
	-device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid3 ...

When the guest is shut down, the vfio_ap mediated devices may be removed.

Using our example again, to remove the vfio_ap mediated device $uuid1::

   /sys/devices/vfio_ap/matrix/
      --- [mdev_supported_types]
      ------ [vfio_ap-passthrough]
      --------- [devices]
      ------------ [$uuid1]
      --------------- remove

::

   echo 1 > remove

This will remove all of the matrix mdev device's sysfs structures including
the mdev device itself. To recreate and reconfigure the matrix mdev device,
all of the steps starting with step 3 will have to be performed again. Note
that the remove will fail if a guest using the vfio_ap mdev is still running.

It is not necessary to remove a vfio_ap mdev, but one may want to
remove it if no guest will use it during the remaining lifetime of the linux
host. If the vfio_ap mdev is removed, one may want to also reconfigure
the pool of adapters and queues reserved for use by the default drivers.

Hot plug/unplug support:
========================
An adapter, domain or control domain may be hot plugged into a running KVM
guest by assigning it to the vfio_ap mediated device being used by the guest if
the following conditions are met:

* The adapter, domain or control domain must also be assigned to the host's
  AP configuration.

* Each APQN derived from the Cartesian product comprised of the APID of the
  adapter being assigned and the APQIs of the domains assigned must reference a
  queue device bound to the vfio_ap device driver.

* To hot plug a domain, each APQN derived from the Cartesian product
  comprised of the APQI of the domain being assigned and the APIDs of the
  adapters assigned must reference a queue device bound to the vfio_ap device
  driver.

An adapter, domain or control domain may be hot unplugged from a running KVM
guest by unassigning it from the vfio_ap mediated device being used by the
guest.

Over-provisioning of AP queues for a KVM guest:
===============================================
Over-provisioning is defined herein as the assignment of adapters or domains to
a vfio_ap mediated device that do not reference AP devices in the host's AP
configuration. The idea here is that when the adapter or domain becomes
available, it will be automatically hot-plugged into the KVM guest using
the vfio_ap mediated device to which it is assigned as long as each new APQN
resulting from plugging it in references a queue device bound to the vfio_ap
device driver.

Limitations
===========
Live guest migration is not supported for guests using AP devices without
intervention by a system administrator. Before a KVM guest can be migrated,
the vfio_ap mediated device must be removed. Unfortunately, it can not be
removed manually (i.e., echo 1 > /sys/devices/vfio_ap/matrix/$UUID/remove) while
the mdev is in use by a KVM guest. If the guest is being emulated by QEMU,
its mdev can be hot unplugged from the guest in one of two ways:

1. If the KVM guest was started with libvirt, you can hot unplug the mdev via
   the following commands:

      virsh detach-device <guestname> <path-to-device-xml>

      For example, to hot unplug mdev 62177883-f1bb-47f0-914d-32a22e3a8804 from
      the guest named 'my-guest':

         virsh detach-device my-guest ~/config/my-guest-hostdev.xml

            The contents of my-guest-hostdev.xml:

.. code-block:: xml

            <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-ap'>
              <source>
                <address uuid='62177883-f1bb-47f0-914d-32a22e3a8804'/>
              </source>
            </hostdev>

      virsh qemu-monitor-command <guest-name> --hmp "device-del <device-id>"

      For example, to hot unplug the vfio_ap mediated device identified on the
      qemu command line with 'id=hostdev0' from the guest named 'my-guest':

.. code-block:: sh

         virsh qemu-monitor-command my-guest --hmp "device_del hostdev0"

2. A vfio_ap mediated device can be hot unplugged by attaching the qemu monitor
   to the guest and using the following qemu monitor command:

      (QEMU) device-del id=<device-id>

      For example, to hot unplug the vfio_ap mediated device that was specified
      on the qemu command line with 'id=hostdev0' when the guest was started:

         (QEMU) device-del id=hostdev0

After live migration of the KVM guest completes, an AP configuration can be
restored to the KVM guest by hot plugging a vfio_ap mediated device on the target
system into the guest in one of two ways:

1. If the KVM guest was started with libvirt, you can hot plug a matrix mediated
   device into the guest via the following virsh commands:

   virsh attach-device <guestname> <path-to-device-xml>

      For example, to hot plug mdev 62177883-f1bb-47f0-914d-32a22e3a8804 into
      the guest named 'my-guest':

         virsh attach-device my-guest ~/config/my-guest-hostdev.xml

            The contents of my-guest-hostdev.xml:

.. code-block:: xml

            <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-ap'>
              <source>
                <address uuid='62177883-f1bb-47f0-914d-32a22e3a8804'/>
              </source>
            </hostdev>

   virsh qemu-monitor-command <guest-name> --hmp \
   "device_add vfio-ap,sysfsdev=<path-to-mdev>,id=<device-id>"

      For example, to hot plug the vfio_ap mediated device
      62177883-f1bb-47f0-914d-32a22e3a8804 into the guest named 'my-guest' with
      device-id hostdev0:

      virsh qemu-monitor-command my-guest --hmp \
      "device_add vfio-ap,\
      sysfsdev=/sys/devices/vfio_ap/matrix/62177883-f1bb-47f0-914d-32a22e3a8804,\
      id=hostdev0"

2. A vfio_ap mediated device can be hot plugged by attaching the qemu monitor
   to the guest and using the following qemu monitor command:

      (qemu) device_add "vfio-ap,sysfsdev=<path-to-mdev>,id=<device-id>"

      For example, to plug the vfio_ap mediated device
      62177883-f1bb-47f0-914d-32a22e3a8804 into the guest with the device-id
      hostdev0:

         (QEMU) device-add "vfio-ap,\
         sysfsdev=/sys/devices/vfio_ap/matrix/62177883-f1bb-47f0-914d-32a22e3a8804,\
         id=hostdev0"


code:
==================================
vfio-ccw: the basic infrastructure
==================================

Introduction
------------

Here we describe the vfio support for I/O subchannel devices for
Linux/s390. Motivation for vfio-ccw is to passthrough subchannels to a
virtual machine, while vfio is the means.

Different than other hardware architectures, s390 has defined a unified
I/O access method, which is so called Channel I/O. It has its own access
patterns:

- Channel programs run asynchronously on a separate (co)processor.
- The channel subsystem will access any memory designated by the caller
  in the channel program directly, i.e. there is no iommu involved.

Thus when we introduce vfio support for these devices, we realize it
with a mediated device (mdev) implementation. The vfio mdev will be
added to an iommu group, so as to make itself able to be managed by the
vfio framework. And we add read/write callbacks for special vfio I/O
regions to pass the channel programs from the mdev to its parent device
(the real I/O subchannel device) to do further address translation and
to perform I/O instructions.

This document does not intend to explain the s390 I/O architecture in
every detail. More information/reference could be found here:

- A good start to know Channel I/O in general:
  https://en.wikipedia.org/wiki/Channel_I/O
- s390 architecture:
  s390 Principles of Operation manual (IBM Form. No. SA22-7832)
- The existing QEMU code which implements a simple emulated channel
  subsystem could also be a good reference. It makes it easier to follow
  the flow.
  qemu/hw/s390x/css.c

For vfio mediated device framework:
- Documentation/driver-api/vfio-mediated-device.rst

Motivation of vfio-ccw
----------------------

Typically, a guest virtualized via QEMU/KVM on s390 only sees
paravirtualized virtio devices via the "Virtio Over Channel I/O
(virtio-ccw)" transport. This makes virtio devices discoverable via
standard operating system algorithms for handling channel devices.

However this is not enough. On s390 for the majority of devices, which
use the standard Channel I/O based mechanism, we also need to provide
the functionality of passing through them to a QEMU virtual machine.
This includes devices that don't have a virtio counterpart (e.g. tape
drives) or that have specific characteristics which guests want to
exploit.

For passing a device to a guest, we want to use the same interface as
everybody else, namely vfio. We implement this vfio support for channel
devices via the vfio mediated device framework and the subchannel device
driver "vfio_ccw".

Access patterns of CCW devices
------------------------------

s390 architecture has implemented a so called channel subsystem, that
provides a unified view of the devices physically attached to the
systems. Though the s390 hardware platform knows about a huge variety of
different peripheral attachments like disk devices (aka. DASDs), tapes,
communication controllers, etc. They can all be accessed by a well
defined access method and they are presenting I/O completion a unified
way: I/O interruptions.

All I/O requires the use of channel command words (CCWs). A CCW is an
instruction to a specialized I/O channel processor. A channel program is
a sequence of CCWs which are executed by the I/O channel subsystem.  To
issue a channel program to the channel subsystem, it is required to
build an operation request block (ORB), which can be used to point out
the format of the CCW and other control information to the system. The
operating system signals the I/O channel subsystem to begin executing
the channel program with a SSCH (start sub-channel) instruction. The
central processor is then free to proceed with non-I/O instructions
until interrupted. The I/O completion result is received by the
interrupt handler in the form of interrupt response block (IRB).

Back to vfio-ccw, in short:

- ORBs and channel programs are built in guest kernel (with guest
  physical addresses).
- ORBs and channel programs are passed to the host kernel.
- Host kernel translates the guest physical addresses to real addresses
  and starts the I/O with issuing a privileged Channel I/O instruction
  (e.g SSCH).
- channel programs run asynchronously on a separate processor.
- I/O completion will be signaled to the host with I/O interruptions.
  And it will be copied as IRB to user space to pass it back to the
  guest.

Physical vfio ccw device and its child mdev
-------------------------------------------

As mentioned above, we realize vfio-ccw with a mdev implementation.

Channel I/O does not have IOMMU hardware support, so the physical
vfio-ccw device does not have an IOMMU level translation or isolation.

Subchannel I/O instructions are all privileged instructions. When
handling the I/O instruction interception, vfio-ccw has the software
policing and translation how the channel program is programmed before
it gets sent to hardware.

Within this implementation, we have two drivers for two types of
devices:

- The vfio_ccw driver for the physical subchannel device.
  This is an I/O subchannel driver for the real subchannel device.  It
  realizes a group of callbacks and registers to the mdev framework as a
  parent (physical) device. As a consequence, mdev provides vfio_ccw a
  generic interface (sysfs) to create mdev devices. A vfio mdev could be
  created by vfio_ccw then and added to the mediated bus. It is the vfio
  device that added to an IOMMU group and a vfio group.
  vfio_ccw also provides an I/O region to accept channel program
  request from user space and store I/O interrupt result for user
  space to retrieve. To notify user space an I/O completion, it offers
  an interface to setup an eventfd fd for asynchronous signaling.

- The vfio_mdev driver for the mediated vfio ccw device.
  This is provided by the mdev framework. It is a vfio device driver for
  the mdev that created by vfio_ccw.
  It realizes a group of vfio device driver callbacks, adds itself to a
  vfio group, and registers itself to the mdev framework as a mdev
  driver.
  It uses a vfio iommu backend that uses the existing map and unmap
  ioctls, but rather than programming them into an IOMMU for a device,
  it simply stores the translations for use by later requests. This
  means that a device programmed in a VM with guest physical addresses
  can have the vfio kernel convert that address to process virtual
  address, pin the page and program the hardware with the host physical
  address in one step.
  For a mdev, the vfio iommu backend will not pin the pages during the
  VFIO_IOMMU_MAP_DMA ioctl. Mdev framework will only maintain a database
  of the iova<->vaddr mappings in this operation. And they export a
  vfio_pin_pages and a vfio_unpin_pages interfaces from the vfio iommu
  backend for the physical devices to pin and unpin pages by demand.

Below is a high Level block diagram::

 +-------------+
 |             |
 | +---------+ | mdev_register_driver() +--------------+
 | |  Mdev   | +<-----------------------+              |
 | |  bus    | |                        | vfio_mdev.ko |
 | | driver  | +----------------------->+              |<-> VFIO user
 | +---------+ |    probe()/remove()    +--------------+    APIs
 |             |
 |  MDEV CORE  |
 |   MODULE    |
 |   mdev.ko   |
 | +---------+ | mdev_register_parent() +--------------+
 | |Physical | +<-----------------------+              |
 | | device  | |                        |  vfio_ccw.ko |<-> subchannel
 | |interface| +----------------------->+              |     device
 | +---------+ |       callback         +--------------+
 +-------------+

The process of how these work together.

1. vfio_ccw.ko drives the physical I/O subchannel, and registers the
   physical device (with callbacks) to mdev framework.
   When vfio_ccw probing the subchannel device, it registers device
   pointer and callbacks to the mdev framework. Mdev related file nodes
   under the device node in sysfs would be created for the subchannel
   device, namely 'mdev_create', 'mdev_destroy' and
   'mdev_supported_types'.
2. Create a mediated vfio ccw device.
   Use the 'mdev_create' sysfs file, we need to manually create one (and
   only one for our case) mediated device.
3. vfio_mdev.ko drives the mediated ccw device.
   vfio_mdev is also the vfio device drvier. It will probe the mdev and
   add it to an iommu_group and a vfio_group. Then we could pass through
   the mdev to a guest.

VFIO-CCW Regions
----------------

The vfio-ccw driver exposes MMIO regions to accept requests from and return
results to userspace.

vfio-ccw I/O region
-------------------

An I/O region is used to accept channel program request from user
space and store I/O interrupt result for user space to retrieve. The
definition of the region is::

  struct ccw_io_region {
  #define ORB_AREA_SIZE 12
	  __u8    orb_area[ORB_AREA_SIZE];
  #define SCSW_AREA_SIZE 12
	  __u8    scsw_area[SCSW_AREA_SIZE];
  #define IRB_AREA_SIZE 96
	  __u8    irb_area[IRB_AREA_SIZE];
	  __u32   ret_code;
  } __packed;

This region is always available.

While starting an I/O request, orb_area should be filled with the
guest ORB, and scsw_area should be filled with the SCSW of the Virtual
Subchannel.

irb_area stores the I/O result.

ret_code stores a return code for each access of the region. The following
values may occur:

``0``
  The operation was successful.

``-EOPNOTSUPP``
  The orb specified transport mode or an unidentified IDAW format, or the
  scsw specified a function other than the start function.

``-EIO``
  A request was issued while the device was not in a state ready to accept
  requests, or an internal error occurred.

``-EBUSY``
  The subchannel was status pending or busy, or a request is already active.

``-EAGAIN``
  A request was being processed, and the caller should retry.

``-EACCES``
  The channel path(s) used for the I/O were found to be not operational.

``-ENODEV``
  The device was found to be not operational.

``-EINVAL``
  The orb specified a chain longer than 255 ccws, or an internal error
  occurred.

vfio-ccw cmd region
-------------------

The vfio-ccw cmd region is used to accept asynchronous instructions
from userspace::

  #define VFIO_CCW_ASYNC_CMD_HSCH (1 << 0)
  #define VFIO_CCW_ASYNC_CMD_CSCH (1 << 1)
  struct ccw_cmd_region {
         __u32 command;
         __u32 ret_code;
  } __packed;

This region is exposed via region type VFIO_REGION_SUBTYPE_CCW_ASYNC_CMD.

Currently, CLEAR SUBCHANNEL and HALT SUBCHANNEL use this region.

command specifies the command to be issued; ret_code stores a return code
for each access of the region. The following values may occur:

``0``
  The operation was successful.

``-ENODEV``
  The device was found to be not operational.

``-EINVAL``
  A command other than halt or clear was specified.

``-EIO``
  A request was issued while the device was not in a state ready to accept
  requests.

``-EAGAIN``
  A request was being processed, and the caller should retry.

``-EBUSY``
  The subchannel was status pending or busy while processing a halt request.

vfio-ccw schib region
---------------------

The vfio-ccw schib region is used to return Subchannel-Information
Block (SCHIB) data to userspace::

  struct ccw_schib_region {
  #define SCHIB_AREA_SIZE 52
         __u8 schib_area[SCHIB_AREA_SIZE];
  } __packed;

This region is exposed via region type VFIO_REGION_SUBTYPE_CCW_SCHIB.

Reading this region triggers a STORE SUBCHANNEL to be issued to the
associated hardware.

vfio-ccw crw region
---------------------

The vfio-ccw crw region is used to return Channel Report Word (CRW)
data to userspace::

  struct ccw_crw_region {
         __u32 crw;
         __u32 pad;
  } __packed;

This region is exposed via region type VFIO_REGION_SUBTYPE_CCW_CRW.

Reading this region returns a CRW if one that is relevant for this
subchannel (e.g. one reporting changes in channel path state) is
pending, or all zeroes if not. If multiple CRWs are pending (including
possibly chained CRWs), reading this region again will return the next
one, until no more CRWs are pending and zeroes are returned. This is
similar to how STORE CHANNEL REPORT WORD works.

vfio-ccw operation details
--------------------------

vfio-ccw follows what vfio-pci did on the s390 platform and uses
vfio-iommu-type1 as the vfio iommu backend.

* CCW translation APIs
  A group of APIs (start with `cp_`) to do CCW translation. The CCWs
  passed in by a user space program are organized with their guest
  physical memory addresses. These APIs will copy the CCWs into kernel
  space, and assemble a runnable kernel channel program by updating the
  guest physical addresses with their corresponding host physical addresses.
  Note that we have to use IDALs even for direct-access CCWs, as the
  referenced memory can be located anywhere, including above 2G.

* vfio_ccw device driver
  This driver utilizes the CCW translation APIs and introduces
  vfio_ccw, which is the driver for the I/O subchannel devices you want
  to pass through.
  vfio_ccw implements the following vfio ioctls::

    VFIO_DEVICE_GET_INFO
    VFIO_DEVICE_GET_IRQ_INFO
    VFIO_DEVICE_GET_REGION_INFO
    VFIO_DEVICE_RESET
    VFIO_DEVICE_SET_IRQS

  This provides an I/O region, so that the user space program can pass a
  channel program to the kernel, to do further CCW translation before
  issuing them to a real device.
  This also provides the SET_IRQ ioctl to setup an event notifier to
  notify the user space program the I/O completion in an asynchronous
  way.

The use of vfio-ccw is not limited to QEMU, while QEMU is definitely a
good example to get understand how these patches work. Here is a little
bit more detail how an I/O request triggered by the QEMU guest will be
handled (without error handling).

Explanation:

- Q1-Q7: QEMU side process.
- K1-K5: Kernel side process.

Q1.
    Get I/O region info during initialization.

Q2.
    Setup event notifier and handler to handle I/O completion.

... ...

Q3.
    Intercept a ssch instruction.
Q4.
    Write the guest channel program and ORB to the I/O region.

    K1.
	Copy from guest to kernel.
    K2.
	Translate the guest channel program to a host kernel space
	channel program, which becomes runnable for a real device.
    K3.
	With the necessary information contained in the orb passed in
	by QEMU, issue the ccwchain to the device.
    K4.
	Return the ssch CC code.
Q5.
    Return the CC code to the guest.

... ...

    K5.
	Interrupt handler gets the I/O result and write the result to
	the I/O region.
    K6.
	Signal QEMU to retrieve the result.

Q6.
    Get the signal and event handler reads out the result from the I/O
    region.
Q7.
    Update the irb for the guest.

Limitations
-----------

The current vfio-ccw implementation focuses on supporting basic commands
needed to implement block device functionality (read/write) of DASD/ECKD
device only. Some commands may need special handling in the future, for
example, anything related to path grouping.

DASD is a kind of storage device. While ECKD is a data recording format.
More information for DASD and ECKD could be found here:
https://en.wikipedia.org/wiki/Direct-access_storage_device
https://en.wikipedia.org/wiki/Count_key_data

Together with the corresponding work in QEMU, we can bring the passed
through DASD/ECKD device online in a guest now and use it as a block
device.

The current code allows the guest to start channel programs via
START SUBCHANNEL, and to issue HALT SUBCHANNEL, CLEAR SUBCHANNEL,
and STORE SUBCHANNEL.

Currently all channel programs are prefetched, regardless of the
p-bit setting in the ORB.  As a result, self modifying channel
programs are not supported.  For this reason, IPL has to be handled as
a special case by a userspace/guest program; this has been implemented
in QEMU's s390-ccw bios as of QEMU 4.1.

vfio-ccw supports classic (command mode) channel I/O only. Transport
mode (HPF) is not supported.

QDIO subchannels are currently not supported. Classic devices other than
DASD/ECKD might work, but have not been tested.

Reference
---------
1. ESA/s390 Principles of Operation manual (IBM Form. No. SA22-7832)
2. ESA/390 Common I/O Device Commands manual (IBM Form. No. SA22-7204)
3. https://en.wikipedia.org/wiki/Channel_I/O
4. Documentation/s390/cds.rst
5. Documentation/driver-api/vfio.rst
6. Documentation/driver-api/vfio-mediated-device.rst

repo_name: angery/linux


code:
==================================
The s390 SCSI dump tool (zfcpdump)
==================================

System z machines (z900 or higher) provide hardware support for creating system
dumps on SCSI disks. The dump process is initiated by booting a dump tool, which
has to create a dump of the current (probably crashed) Linux image. In order to
not overwrite memory of the crashed Linux with data of the dump tool, the
hardware saves some memory plus the register sets of the boot CPU before the
dump tool is loaded. There exists an SCLP hardware interface to obtain the saved
memory afterwards. Currently 32 MB are saved.

This zfcpdump implementation consists of a Linux dump kernel together with
a user space dump tool, which are loaded together into the saved memory region
below 32 MB. zfcpdump is installed on a SCSI disk using zipl (as contained in
the s390-tools package) to make the device bootable. The operator of a Linux
system can then trigger a SCSI dump by booting the SCSI disk, where zfcpdump
resides on.

The user space dump tool accesses the memory of the crashed system by means
of the /proc/vmcore interface. This interface exports the crashed system's
memory and registers in ELF core dump format. To access the memory which has
been saved by the hardware SCLP requests will be created at the time the data
is needed by /proc/vmcore. The tail part of the crashed systems memory which
has not been stashed by hardware can just be copied from real memory.

To build a dump enabled kernel the kernel config option CONFIG_CRASH_DUMP
has to be set.

To get a valid zfcpdump kernel configuration use "make zfcpdump_defconfig".

The s390 zipl tool looks for the zfcpdump kernel and optional initrd/initramfs
under the following locations:

* kernel:  <zfcpdump directory>/zfcpdump.image
* ramdisk: <zfcpdump directory>/zfcpdump.rd

The zfcpdump directory is defined in the s390-tools package.

The user space application of zfcpdump can reside in an intitramfs or an
initrd. It can also be included in a built-in kernel initramfs. The application
reads from /proc/vmcore or zcore/mem and writes the system dump to a SCSI disk.

The s390-tools package version 1.24.0 and above builds an external zfcpdump
initramfs with a user space application that writes the dump to a SCSI
partition.

For more information on how to use zfcpdump refer to the s390 'Using the Dump
Tools' book, which is available from IBM Knowledge Center:
https://www.ibm.com/support/knowledgecenter/linuxonibm/liaaf/lnz_r_dt.html


code:
================================================
Completions - "wait for completion" barrier APIs
================================================

Introduction:
-------------

If you have one or more threads that must wait for some kernel activity
to have reached a point or a specific state, completions can provide a
race-free solution to this problem. Semantically they are somewhat like a
pthread_barrier() and have similar use-cases.

Completions are a code synchronization mechanism which is preferable to any
misuse of locks/semaphores and busy-loops. Any time you think of using
yield() or some quirky msleep(1) loop to allow something else to proceed,
you probably want to look into using one of the wait_for_completion*()
calls and complete() instead.

The advantage of using completions is that they have a well defined, focused
purpose which makes it very easy to see the intent of the code, but they
also result in more efficient code as all threads can continue execution
until the result is actually needed, and both the waiting and the signalling
is highly efficient using low level scheduler sleep/wakeup facilities.

Completions are built on top of the waitqueue and wakeup infrastructure of
the Linux scheduler. The event the threads on the waitqueue are waiting for
is reduced to a simple flag in 'struct completion', appropriately called "done".

As completions are scheduling related, the code can be found in
kernel/sched/completion.c.

Usage:
------

There are three main parts to using completions:

 - the initialization of the 'struct completion' synchronization object
 - the waiting part through a call to one of the variants of wait_for_completion(),
 - the signaling side through a call to complete() or complete_all().

There are also some helper functions for checking the state of completions.
Note that while initialization must happen first, the waiting and signaling
part can happen in any order. I.e. it's entirely normal for a thread
to have marked a completion as 'done' before another thread checks whether
it has to wait for it.

To use completions you need to #include <linux/completion.h> and
create a static or dynamic variable of type 'struct completion',
which has only two fields::

	struct completion {
		unsigned int done;
		wait_queue_head_t wait;
	};

This provides the ->wait waitqueue to place tasks on for waiting (if any), and
the ->done completion flag for indicating whether it's completed or not.

Completions should be named to refer to the event that is being synchronized on.
A good example is::

	wait_for_completion(&early_console_added);

	complete(&early_console_added);

Good, intuitive naming (as always) helps code readability. Naming a completion
'complete' is not helpful unless the purpose is super obvious...

Initializing completions:
-------------------------

Dynamically allocated completion objects should preferably be embedded in data
structures that are assured to be alive for the life-time of the function/driver,
to prevent races with asynchronous complete() calls from occurring.

Particular care should be taken when using the _timeout() or _killable()/_interruptible()
variants of wait_for_completion(), as it must be assured that memory de-allocation
does not happen until all related activities (complete() or reinit_completion())
have taken place, even if these wait functions return prematurely due to a timeout
or a signal triggering.

Initializing of dynamically allocated completion objects is done via a call to
init_completion()::

	init_completion(&dynamic_object->done);

In this call we initialize the waitqueue and set ->done to 0, i.e. "not completed"
or "not done".

The re-initialization function, reinit_completion(), simply resets the
->done field to 0 ("not done"), without touching the waitqueue.
Callers of this function must make sure that there are no racy
wait_for_completion() calls going on in parallel.

Calling init_completion() on the same completion object twice is
most likely a bug as it re-initializes the queue to an empty queue and
enqueued tasks could get "lost" - use reinit_completion() in that case,
but be aware of other races.

For static declaration and initialization, macros are available.

For static (or global) declarations in file scope you can use
DECLARE_COMPLETION()::

	static DECLARE_COMPLETION(setup_done);
	DECLARE_COMPLETION(setup_done);

Note that in this case the completion is boot time (or module load time)
initialized to 'not done' and doesn't require an init_completion() call.

When a completion is declared as a local variable within a function,
then the initialization should always use DECLARE_COMPLETION_ONSTACK()
explicitly, not just to make lockdep happy, but also to make it clear
that limited scope had been considered and is intentional::

	DECLARE_COMPLETION_ONSTACK(setup_done)

Note that when using completion objects as local variables you must be
acutely aware of the short life time of the function stack: the function
must not return to a calling context until all activities (such as waiting
threads) have ceased and the completion object is completely unused.

To emphasise this again: in particular when using some of the waiting API variants
with more complex outcomes, such as the timeout or signalling (_timeout(),
_killable() and _interruptible()) variants, the wait might complete
prematurely while the object might still be in use by another thread - and a return
from the wait_on_completion*() caller function will deallocate the function
stack and cause subtle data corruption if a complete() is done in some
other thread. Simple testing might not trigger these kinds of races.

If unsure, use dynamically allocated completion objects, preferably embedded
in some other long lived object that has a boringly long life time which
exceeds the life time of any helper threads using the completion object,
or has a lock or other synchronization mechanism to make sure complete()
is not called on a freed object.

A naive DECLARE_COMPLETION() on the stack triggers a lockdep warning.

Waiting for completions:
------------------------

For a thread to wait for some concurrent activity to finish, it
calls wait_for_completion() on the initialized completion structure::

	void wait_for_completion(struct completion *done)

A typical usage scenario is::

	CPU#1					CPU#2

	struct completion setup_done;

	init_completion(&setup_done);
	initialize_work(...,&setup_done,...);

	/* run non-dependent code */		/* do setup */

	wait_for_completion(&setup_done);	complete(setup_done);

This is not implying any particular order between wait_for_completion() and
the call to complete() - if the call to complete() happened before the call
to wait_for_completion() then the waiting side simply will continue
immediately as all dependencies are satisfied; if not, it will block until
completion is signaled by complete().

Note that wait_for_completion() is calling spin_lock_irq()/spin_unlock_irq(),
so it can only be called safely when you know that interrupts are enabled.
Calling it from IRQs-off atomic contexts will result in hard-to-detect
spurious enabling of interrupts.

The default behavior is to wait without a timeout and to mark the task as
uninterruptible. wait_for_completion() and its variants are only safe
in process context (as they can sleep) but not in atomic context,
interrupt context, with disabled IRQs, or preemption is disabled - see also
try_wait_for_completion() below for handling completion in atomic/interrupt
context.

As all variants of wait_for_completion() can (obviously) block for a long
time depending on the nature of the activity they are waiting for, so in
most cases you probably don't want to call this with held mutexes.

wait_for_completion*() variants available:
------------------------------------------

The below variants all return status and this status should be checked in
most(/all) cases - in cases where the status is deliberately not checked you
probably want to make a note explaining this (e.g. see
arch/arm/kernel/smp.c:__cpu_up()).

A common problem that occurs is to have unclean assignment of return types,
so take care to assign return-values to variables of the proper type.

Checking for the specific meaning of return values also has been found
to be quite inaccurate, e.g. constructs like::

	if (!wait_for_completion_interruptible_timeout(...))

... would execute the same code path for successful completion and for the
interrupted case - which is probably not what you want::

	int wait_for_completion_interruptible(struct completion *done)

This function marks the task TASK_INTERRUPTIBLE while it is waiting.
If a signal was received while waiting it will return -ERESTARTSYS; 0 otherwise::

	unsigned long wait_for_completion_timeout(struct completion *done, unsigned long timeout)

The task is marked as TASK_UNINTERRUPTIBLE and will wait at most 'timeout'
jiffies. If a timeout occurs it returns 0, else the remaining time in
jiffies (but at least 1).

Timeouts are preferably calculated with msecs_to_jiffies() or usecs_to_jiffies(),
to make the code largely HZ-invariant.

If the returned timeout value is deliberately ignored a comment should probably explain
why (e.g. see drivers/mfd/wm8350-core.c wm8350_read_auxadc())::

	long wait_for_completion_interruptible_timeout(struct completion *done, unsigned long timeout)

This function passes a timeout in jiffies and marks the task as
TASK_INTERRUPTIBLE. If a signal was received it will return -ERESTARTSYS;
otherwise it returns 0 if the completion timed out, or the remaining time in
jiffies if completion occurred.

Further variants include _killable which uses TASK_KILLABLE as the
designated tasks state and will return -ERESTARTSYS if it is interrupted,
or 0 if completion was achieved.  There is a _timeout variant as well::

	long wait_for_completion_killable(struct completion *done)
	long wait_for_completion_killable_timeout(struct completion *done, unsigned long timeout)

The _io variants wait_for_completion_io() behave the same as the non-_io
variants, except for accounting waiting time as 'waiting on IO', which has
an impact on how the task is accounted in scheduling/IO stats::

	void wait_for_completion_io(struct completion *done)
	unsigned long wait_for_completion_io_timeout(struct completion *done, unsigned long timeout)

Signaling completions:
----------------------

A thread that wants to signal that the conditions for continuation have been
achieved calls complete() to signal exactly one of the waiters that it can
continue::

	void complete(struct completion *done)

... or calls complete_all() to signal all current and future waiters::

	void complete_all(struct completion *done)

The signaling will work as expected even if completions are signaled before
a thread starts waiting. This is achieved by the waiter "consuming"
(decrementing) the done field of 'struct completion'. Waiting threads
wakeup order is the same in which they were enqueued (FIFO order).

If complete() is called multiple times then this will allow for that number
of waiters to continue - each call to complete() will simply increment the
done field. Calling complete_all() multiple times is a bug though. Both
complete() and complete_all() can be called in IRQ/atomic context safely.

There can only be one thread calling complete() or complete_all() on a
particular 'struct completion' at any time - serialized through the wait
queue spinlock. Any such concurrent calls to complete() or complete_all()
probably are a design bug.

Signaling completion from IRQ context is fine as it will appropriately
lock with spin_lock_irqsave()/spin_unlock_irqrestore() and it will never
sleep.

try_wait_for_completion()/completion_done():
--------------------------------------------

The try_wait_for_completion() function will not put the thread on the wait
queue but rather returns false if it would need to enqueue (block) the thread,
else it consumes one posted completion and returns true::

	bool try_wait_for_completion(struct completion *done)

Finally, to check the state of a completion without changing it in any way,
call completion_done(), which returns false if there are no posted
completions that were not yet consumed by waiters (implying that there are
waiters) and true otherwise::

	bool completion_done(struct completion *done)

Both try_wait_for_completion() and completion_done() are safe to be called in
IRQ or atomic context.


code:
===============
Linux Scheduler
===============

.. toctree::
    :maxdepth: 1

    completion
    sched-arch
    sched-bwc
    sched-deadline
    sched-design-CFS
    sched-domains
    sched-capacity
    sched-energy
    schedutil
    sched-nice-design
    sched-rt-group
    sched-stats
    sched-debug

    text_files

.. only::  subproject and html

   Indices
   =======

   * :ref:`genindex`


code:
=================================================================
CPU Scheduler implementation hints for architecture specific code
=================================================================

	Nick Piggin, 2005

Context switch
==============
1. Runqueue locking
By default, the switch_to arch function is called with the runqueue
locked. This is usually not a problem unless switch_to may need to
take the runqueue lock. This is usually due to a wake up operation in
the context switch. See arch/ia64/include/asm/switch_to.h for an example.

To request the scheduler call switch_to with the runqueue unlocked,
you must `#define __ARCH_WANT_UNLOCKED_CTXSW` in a header file
(typically the one where switch_to is defined).

Unlocked context switches introduce only a very minor performance
penalty to the core scheduler implementation in the CONFIG_SMP case.

CPU idle
========
Your cpu_idle routines need to obey the following rules:

1. Preempt should now disabled over idle routines. Should only
   be enabled to call schedule() then disabled again.

2. need_resched/TIF_NEED_RESCHED is only ever set, and will never
   be cleared until the running task has called schedule(). Idle
   threads need only ever query need_resched, and may never set or
   clear it.

3. When cpu_idle finds (need_resched() == 'true'), it should call
   schedule(). It should not call schedule() otherwise.

4. The only time interrupts need to be disabled when checking
   need_resched is if we are about to sleep the processor until
   the next interrupt (this doesn't provide any protection of
   need_resched, it prevents losing an interrupt):

	4a. Common problem with this type of sleep appears to be::

	        local_irq_disable();
	        if (!need_resched()) {
	                local_irq_enable();
	                *** resched interrupt arrives here ***
	                __asm__("sleep until next interrupt");
	        }

5. TIF_POLLING_NRFLAG can be set by idle routines that do not
   need an interrupt to wake them up when need_resched goes high.
   In other words, they must be periodically polling need_resched,
   although it may be reasonable to do some background work or enter
   a low CPU priority.

      - 5a. If TIF_POLLING_NRFLAG is set, and we do decide to enter
	an interrupt sleep, it needs to be cleared then a memory
	barrier issued (followed by a test of need_resched with
	interrupts disabled, as explained in 3).

arch/x86/kernel/process.c has examples of both polling and
sleeping idle functions.

Possible arch/ problems
=======================

Possible arch problems I found (and either tried to fix or didn't):

ia64 - is safe_halt call racy vs interrupts? (does it sleep?) (See #4a)

sh64 - Is sleeping racy vs interrupts? (See #4a)

sparc - IRQs on at this point(?), change local_irq_save to _disable.
      - TODO: needs secondary CPUs to disable preempt (See #1)


code:
=====================
CFS Bandwidth Control
=====================

.. note::
   This document only discusses CPU bandwidth control for SCHED_NORMAL.
   The SCHED_RT case is covered in Documentation/scheduler/sched-rt-group.rst

CFS bandwidth control is a CONFIG_FAIR_GROUP_SCHED extension which allows the
specification of the maximum CPU bandwidth available to a group or hierarchy.

The bandwidth allowed for a group is specified using a quota and period. Within
each given "period" (microseconds), a task group is allocated up to "quota"
microseconds of CPU time. That quota is assigned to per-cpu run queues in
slices as threads in the cgroup become runnable. Once all quota has been
assigned any additional requests for quota will result in those threads being
throttled. Throttled threads will not be able to run again until the next
period when the quota is replenished.

A group's unassigned quota is globally tracked, being refreshed back to
cfs_quota units at each period boundary. As threads consume this bandwidth it
is transferred to cpu-local "silos" on a demand basis. The amount transferred
within each of these updates is tunable and described as the "slice".

Burst feature
-------------
This feature borrows time now against our future underrun, at the cost of
increased interference against the other system users. All nicely bounded.

Traditional (UP-EDF) bandwidth control is something like:

  (U = \Sum u_i) <= 1

This guaranteeds both that every deadline is met and that the system is
stable. After all, if U were > 1, then for every second of walltime,
we'd have to run more than a second of program time, and obviously miss
our deadline, but the next deadline will be further out still, there is
never time to catch up, unbounded fail.

The burst feature observes that a workload doesn't always executes the full
quota; this enables one to describe u_i as a statistical distribution.

For example, have u_i = {x,e}_i, where x is the p(95) and x+e p(100)
(the traditional WCET). This effectively allows u to be smaller,
increasing the efficiency (we can pack more tasks in the system), but at
the cost of missing deadlines when all the odds line up. However, it
does maintain stability, since every overrun must be paired with an
underrun as long as our x is above the average.

That is, suppose we have 2 tasks, both specify a p(95) value, then we
have a p(95)*p(95) = 90.25% chance both tasks are within their quota and
everything is good. At the same time we have a p(5)p(5) = 0.25% chance
both tasks will exceed their quota at the same time (guaranteed deadline
fail). Somewhere in between there's a threshold where one exceeds and
the other doesn't underrun enough to compensate; this depends on the
specific CDFs.

At the same time, we can say that the worst case deadline miss, will be
\Sum e_i; that is, there is a bounded tardiness (under the assumption
that x+e is indeed WCET).

The interferenece when using burst is valued by the possibilities for
missing the deadline and the average WCET. Test results showed that when
there many cgroups or CPU is under utilized, the interference is
limited. More details are shown in:
https://lore.kernel.org/lkml/5371BD36-55AE-4F71-B9D7-B86DC32E3D2B@linux.alibaba.com/

Management
----------
Quota, period and burst are managed within the cpu subsystem via cgroupfs.

.. note::
   The cgroupfs files described in this section are only applicable
   to cgroup v1. For cgroup v2, see
   :ref:`Documentation/admin-guide/cgroup-v2.rst <cgroup-v2-cpu>`.

- cpu.cfs_quota_us: run-time replenished within a period (in microseconds)
- cpu.cfs_period_us: the length of a period (in microseconds)
- cpu.stat: exports throttling statistics [explained further below]
- cpu.cfs_burst_us: the maximum accumulated run-time (in microseconds)

The default values are::

	cpu.cfs_period_us=100ms
	cpu.cfs_quota_us=-1
	cpu.cfs_burst_us=0

A value of -1 for cpu.cfs_quota_us indicates that the group does not have any
bandwidth restriction in place, such a group is described as an unconstrained
bandwidth group. This represents the traditional work-conserving behavior for
CFS.

Writing any (valid) positive value(s) no smaller than cpu.cfs_burst_us will
enact the specified bandwidth limit. The minimum quota allowed for the quota or
period is 1ms. There is also an upper bound on the period length of 1s.
Additional restrictions exist when bandwidth limits are used in a hierarchical
fashion, these are explained in more detail below.

Writing any negative value to cpu.cfs_quota_us will remove the bandwidth limit
and return the group to an unconstrained state once more.

A value of 0 for cpu.cfs_burst_us indicates that the group can not accumulate
any unused bandwidth. It makes the traditional bandwidth control behavior for
CFS unchanged. Writing any (valid) positive value(s) no larger than
cpu.cfs_quota_us into cpu.cfs_burst_us will enact the cap on unused bandwidth
accumulation.

Any updates to a group's bandwidth specification will result in it becoming
unthrottled if it is in a constrained state.

System wide settings
--------------------
For efficiency run-time is transferred between the global pool and CPU local
"silos" in a batch fashion. This greatly reduces global accounting pressure
on large systems. The amount transferred each time such an update is required
is described as the "slice".

This is tunable via procfs::

	/proc/sys/kernel/sched_cfs_bandwidth_slice_us (default=5ms)

Larger slice values will reduce transfer overheads, while smaller values allow
for more fine-grained consumption.

Statistics
----------
A group's bandwidth statistics are exported via 5 fields in cpu.stat.

cpu.stat:

- nr_periods: Number of enforcement intervals that have elapsed.
- nr_throttled: Number of times the group has been throttled/limited.
- throttled_time: The total time duration (in nanoseconds) for which entities
  of the group have been throttled.
- nr_bursts: Number of periods burst occurs.
- burst_time: Cumulative wall-time (in nanoseconds) that any CPUs has used
  above quota in respective periods.

This interface is read-only.

Hierarchical considerations
---------------------------
The interface enforces that an individual entity's bandwidth is always
attainable, that is: max(c_i) <= C. However, over-subscription in the
aggregate case is explicitly allowed to enable work-conserving semantics
within a hierarchy:

  e.g. \Sum (c_i) may exceed C

[ Where C is the parent's bandwidth, and c_i its children ]

There are two ways in which a group may become throttled:

	a. it fully consumes its own quota within a period
	b. a parent's quota is fully consumed within its period

In case b) above, even though the child may have runtime remaining it will not
be allowed to until the parent's runtime is refreshed.

CFS Bandwidth Quota Caveats
---------------------------
Once a slice is assigned to a cpu it does not expire.  However all but 1ms of
the slice may be returned to the global pool if all threads on that cpu become
unrunnable. This is configured at compile time by the min_cfs_rq_runtime
variable. This is a performance tweak that helps prevent added contention on
the global lock.

The fact that cpu-local slices do not expire results in some interesting corner
cases that should be understood.

For cgroup cpu constrained applications that are cpu limited this is a
relatively moot point because they will naturally consume the entirety of their
quota as well as the entirety of each cpu-local slice in each period. As a
result it is expected that nr_periods roughly equal nr_throttled, and that
cpuacct.usage will increase roughly equal to cfs_quota_us in each period.

For highly-threaded, non-cpu bound applications this non-expiration nuance
allows applications to briefly burst past their quota limits by the amount of
unused slice on each cpu that the task group is running on (typically at most
1ms per cpu or as defined by min_cfs_rq_runtime).  This slight burst only
applies if quota had been assigned to a cpu and then not fully used or returned
in previous periods. This burst amount will not be transferred between cores.
As a result, this mechanism still strictly limits the task group to quota
average usage, albeit over a longer time window than a single period.  This
also limits the burst ability to no more than 1ms per cpu.  This provides
better more predictable user experience for highly threaded applications with
small quota limits on high core count machines. It also eliminates the
propensity to throttle these applications while simultanously using less than
quota amounts of cpu. Another way to say this, is that by allowing the unused
portion of a slice to remain valid across periods we have decreased the
possibility of wastefully expiring quota on cpu-local silos that don't need a
full slice's amount of cpu time.

The interaction between cpu-bound and non-cpu-bound-interactive applications
should also be considered, especially when single core usage hits 100%. If you
gave each of these applications half of a cpu-core and they both got scheduled
on the same CPU it is theoretically possible that the non-cpu bound application
will use up to 1ms additional quota in some periods, thereby preventing the
cpu-bound application from fully using its quota by that same amount. In these
instances it will be up to the CFS algorithm (see sched-design-CFS.rst) to
decide which application is chosen to run, as they will both be runnable and
have remaining quota. This runtime discrepancy will be made up in the following
periods when the interactive application idles.

Examples
--------
1. Limit a group to 1 CPU worth of runtime::

	If period is 250ms and quota is also 250ms, the group will get
	1 CPU worth of runtime every 250ms.

	# echo 250000 > cpu.cfs_quota_us /* quota = 250ms */
	# echo 250000 > cpu.cfs_period_us /* period = 250ms */

2. Limit a group to 2 CPUs worth of runtime on a multi-CPU machine

   With 500ms period and 1000ms quota, the group can get 2 CPUs worth of
   runtime every 500ms::

	# echo 1000000 > cpu.cfs_quota_us /* quota = 1000ms */
	# echo 500000 > cpu.cfs_period_us /* period = 500ms */

	The larger period here allows for increased burst capacity.

3. Limit a group to 20% of 1 CPU.

   With 50ms period, 10ms quota will be equivalent to 20% of 1 CPU::

	# echo 10000 > cpu.cfs_quota_us /* quota = 10ms */
	# echo 50000 > cpu.cfs_period_us /* period = 50ms */

   By using a small period here we are ensuring a consistent latency
   response at the expense of burst capacity.

4. Limit a group to 40% of 1 CPU, and allow accumulate up to 20% of 1 CPU
   additionally, in case accumulation has been done.

   With 50ms period, 20ms quota will be equivalent to 40% of 1 CPU.
   And 10ms burst will be equivalent to 20% of 1 CPU::

	# echo 20000 > cpu.cfs_quota_us /* quota = 20ms */
	# echo 50000 > cpu.cfs_period_us /* period = 50ms */
	# echo 10000 > cpu.cfs_burst_us /* burst = 10ms */

   Larger buffer setting (no larger than quota) allows greater burst capacity.