code:
.. SPDX-License-Identifier: GPL-2.0
.. _ultravisor:

============================
Protected Execution Facility
============================

.. contents::
    :depth: 3

Introduction
############

    Protected Execution Facility (PEF) is an architectural change for
    POWER 9 that enables Secure Virtual Machines (SVMs). DD2.3 chips
    (PVR=0x004e1203) or greater will be PEF-capable. A new ISA release
    will include the PEF RFC02487 changes.

    When enabled, PEF adds a new higher privileged mode, called Ultravisor
    mode, to POWER architecture. Along with the new mode there is new
    firmware called the Protected Execution Ultravisor (or Ultravisor
    for short). Ultravisor mode is the highest privileged mode in POWER
    architecture.

	+------------------+
	| Privilege States |
	+==================+
	|  Problem         |
	+------------------+
	|  Supervisor      |
	+------------------+
	|  Hypervisor      |
	+------------------+
	|  Ultravisor      |
	+------------------+

    PEF protects SVMs from the hypervisor, privileged users, and other
    VMs in the system. SVMs are protected while at rest and can only be
    executed by an authorized machine. All virtual machines utilize
    hypervisor services. The Ultravisor filters calls between the SVMs
    and the hypervisor to assure that information does not accidentally
    leak. All hypercalls except H_RANDOM are reflected to the hypervisor.
    H_RANDOM is not reflected to prevent the hypervisor from influencing
    random values in the SVM.

    To support this there is a refactoring of the ownership of resources
    in the CPU. Some of the resources which were previously hypervisor
    privileged are now ultravisor privileged.

Hardware
========

    The hardware changes include the following:

    * There is a new bit in the MSR that determines whether the current
      process is running in secure mode, MSR(S) bit 41. MSR(S)=1, process
      is in secure mode, MSR(s)=0 process is in normal mode.

    * The MSR(S) bit can only be set by the Ultravisor.

    * HRFID cannot be used to set the MSR(S) bit. If the hypervisor needs
      to return to a SVM it must use an ultracall. It can determine if
      the VM it is returning to is secure.

    * There is a new Ultravisor privileged register, SMFCTRL, which has an
      enable/disable bit SMFCTRL(E).

    * The privilege of a process is now determined by three MSR bits,
      MSR(S, HV, PR). In each of the tables below the modes are listed
      from least privilege to highest privilege. The higher privilege
      modes can access all the resources of the lower privilege modes.

      **Secure Mode MSR Settings**

      +---+---+---+---------------+
      | S | HV| PR|Privilege      |
      +===+===+===+===============+
      | 1 | 0 | 1 | Problem       |
      +---+---+---+---------------+
      | 1 | 0 | 0 | Privileged(OS)|
      +---+---+---+---------------+
      | 1 | 1 | 0 | Ultravisor    |
      +---+---+---+---------------+
      | 1 | 1 | 1 | Reserved      |
      +---+---+---+---------------+

      **Normal Mode MSR Settings**

      +---+---+---+---------------+
      | S | HV| PR|Privilege      |
      +===+===+===+===============+
      | 0 | 0 | 1 | Problem       |
      +---+---+---+---------------+
      | 0 | 0 | 0 | Privileged(OS)|
      +---+---+---+---------------+
      | 0 | 1 | 0 | Hypervisor    |
      +---+---+---+---------------+
      | 0 | 1 | 1 | Problem (Host)|
      +---+---+---+---------------+

    * Memory is partitioned into secure and normal memory. Only processes
      that are running in secure mode can access secure memory.

    * The hardware does not allow anything that is not running secure to
      access secure memory. This means that the Hypervisor cannot access
      the memory of the SVM without using an ultracall (asking the
      Ultravisor). The Ultravisor will only allow the hypervisor to see
      the SVM memory encrypted.

    * I/O systems are not allowed to directly address secure memory. This
      limits the SVMs to virtual I/O only.

    * The architecture allows the SVM to share pages of memory with the
      hypervisor that are not protected with encryption. However, this
      sharing must be initiated by the SVM.

    * When a process is running in secure mode all hypercalls
      (syscall lev=1) go to the Ultravisor.

    * When a process is in secure mode all interrupts go to the
      Ultravisor.

    * The following resources have become Ultravisor privileged and
      require an Ultravisor interface to manipulate:

      * Processor configurations registers (SCOMs).

      * Stop state information.

      * The debug registers CIABR, DAWR, and DAWRX when SMFCTRL(D) is set.
        If SMFCTRL(D) is not set they do not work in secure mode. When set,
        reading and writing requires an Ultravisor call, otherwise that
        will cause a Hypervisor Emulation Assistance interrupt.

      * PTCR and partition table entries (partition table is in secure
        memory). An attempt to write to PTCR will cause a Hypervisor
        Emulation Assitance interrupt.

      * LDBAR (LD Base Address Register) and IMC (In-Memory Collection)
        non-architected registers. An attempt to write to them will cause a
        Hypervisor Emulation Assistance interrupt.

      * Paging for an SVM, sharing of memory with Hypervisor for an SVM.
        (Including Virtual Processor Area (VPA) and virtual I/O).

Software/Microcode
==================

    The software changes include:

    * SVMs are created from normal VM using (open source) tooling supplied
      by IBM.

    * All SVMs start as normal VMs and utilize an ultracall, UV_ESM
      (Enter Secure Mode), to make the transition.

    * When the UV_ESM ultracall is made the Ultravisor copies the VM into
      secure memory, decrypts the verification information, and checks the
      integrity of the SVM. If the integrity check passes the Ultravisor
      passes control in secure mode.

    * The verification information includes the pass phrase for the
      encrypted disk associated with the SVM. This pass phrase is given
      to the SVM when requested.

    * The Ultravisor is not involved in protecting the encrypted disk of
      the SVM while at rest.

    * For external interrupts the Ultravisor saves the state of the SVM,
      and reflects the interrupt to the hypervisor for processing.
      For hypercalls, the Ultravisor inserts neutral state into all
      registers not needed for the hypercall then reflects the call to
      the hypervisor for processing. The H_RANDOM hypercall is performed
      by the Ultravisor and not reflected.

    * For virtual I/O to work bounce buffering must be done.

    * The Ultravisor uses AES (IAPM) for protection of SVM memory. IAPM
      is a mode of AES that provides integrity and secrecy concurrently.

    * The movement of data between normal and secure pages is coordinated
      with the Ultravisor by a new HMM plug-in in the Hypervisor.

    The Ultravisor offers new services to the hypervisor and SVMs. These
    are accessed through ultracalls.

Terminology
===========

    * Hypercalls: special system calls used to request services from
      Hypervisor.

    * Normal memory: Memory that is accessible to Hypervisor.

    * Normal page: Page backed by normal memory and available to
      Hypervisor.

    * Shared page: A page backed by normal memory and available to both
      the Hypervisor/QEMU and the SVM (i.e page has mappings in SVM and
      Hypervisor/QEMU).

    * Secure memory: Memory that is accessible only to Ultravisor and
      SVMs.

    * Secure page: Page backed by secure memory and only available to
      Ultravisor and SVM.

    * SVM: Secure Virtual Machine.

    * Ultracalls: special system calls used to request services from
      Ultravisor.

Ultravisor calls API
####################

    This section describes Ultravisor calls (ultracalls) needed to
    support Secure Virtual Machines (SVM)s and Paravirtualized KVM. The
    ultracalls allow the SVMs and Hypervisor to request services from the
    Ultravisor such as accessing a register or memory region that can only
    be accessed when running in Ultravisor-privileged mode.

    The specific service needed from an ultracall is specified in register
    R3 (the first parameter to the ultracall). Other parameters to the
    ultracall, if any, are specified in registers R4 through R12.

    Return value of all ultracalls is in register R3. Other output values
    from the ultracall, if any, are returned in registers R4 through R12.
    The only exception to this register usage is the ``UV_RETURN``
    ultracall described below.

    Each ultracall returns specific error codes, applicable in the context
    of the ultracall. However, like with the PowerPC Architecture Platform
    Reference (PAPR), if no specific error code is defined for a
    particular situation, then the ultracall will fallback to an erroneous
    parameter-position based code. i.e U_PARAMETER, U_P2, U_P3 etc
    depending on the ultracall parameter that may have caused the error.

    Some ultracalls involve transferring a page of data between Ultravisor
    and Hypervisor.  Secure pages that are transferred from secure memory
    to normal memory may be encrypted using dynamically generated keys.
    When the secure pages are transferred back to secure memory, they may
    be decrypted using the same dynamically generated keys. Generation and
    management of these keys will be covered in a separate document.

    For now this only covers ultracalls currently implemented and being
    used by Hypervisor and SVMs but others can be added here when it
    makes sense.

    The full specification for all hypercalls/ultracalls will eventually
    be made available in the public/OpenPower version of the PAPR
    specification.

    .. note::

        If PEF is not enabled, the ultracalls will be redirected to the
        Hypervisor which must handle/fail the calls.

Ultracalls used by Hypervisor
=============================

    This section describes the virtual memory management ultracalls used
    by the Hypervisor to manage SVMs.

UV_PAGE_OUT
-----------

    Encrypt and move the contents of a page from secure memory to normal
    memory.

Syntax
~~~~~~

.. code-block:: c

	uint64_t ultracall(const uint64_t UV_PAGE_OUT,
		uint16_t lpid,		/* LPAR ID */
		uint64_t dest_ra,	/* real address of destination page */
		uint64_t src_gpa,	/* source guest-physical-address */
		uint8_t  flags,		/* flags */
		uint64_t order)		/* page size order */

Return values
~~~~~~~~~~~~~

    One of the following values:

	* U_SUCCESS	on success.
	* U_PARAMETER	if ``lpid`` is invalid.
	* U_P2 		if ``dest_ra`` is invalid.
	* U_P3		if the ``src_gpa`` address is invalid.
	* U_P4		if any bit in the ``flags`` is unrecognized
	* U_P5		if the ``order`` parameter is unsupported.
	* U_FUNCTION	if functionality is not supported.
	* U_BUSY	if page cannot be currently paged-out.

Description
~~~~~~~~~~~

    Encrypt the contents of a secure-page and make it available to
    Hypervisor in a normal page.

    By default, the source page is unmapped from the SVM's partition-
    scoped page table. But the Hypervisor can provide a hint to the
    Ultravisor to retain the page mapping by setting the ``UV_SNAPSHOT``
    flag in ``flags`` parameter.

    If the source page is already a shared page the call returns
    U_SUCCESS, without doing anything.

Use cases
~~~~~~~~~

    #. QEMU attempts to access an address belonging to the SVM but the
       page frame for that address is not mapped into QEMU's address
       space. In this case, the Hypervisor will allocate a page frame,
       map it into QEMU's address space and issue the ``UV_PAGE_OUT``
       call to retrieve the encrypted contents of the page.

    #. When Ultravisor runs low on secure memory and it needs to page-out
       an LRU page. In this case, Ultravisor will issue the
       ``H_SVM_PAGE_OUT`` hypercall to the Hypervisor. The Hypervisor will
       then allocate a normal page and issue the ``UV_PAGE_OUT`` ultracall
       and the Ultravisor will encrypt and move the contents of the secure
       page into the normal page.

    #. When Hypervisor accesses SVM data, the Hypervisor requests the
       Ultravisor to transfer the corresponding page into a insecure page,
       which the Hypervisor can access. The data in the normal page will
       be encrypted though.

UV_PAGE_IN
----------

    Move the contents of a page from normal memory to secure memory.

Syntax
~~~~~~

.. code-block:: c

	uint64_t ultracall(const uint64_t UV_PAGE_IN,
		uint16_t lpid,		/* the LPAR ID */
		uint64_t src_ra,	/* source real address of page */
		uint64_t dest_gpa,	/* destination guest physical address */
		uint64_t flags,		/* flags */
		uint64_t order)		/* page size order */

Return values
~~~~~~~~~~~~~

    One of the following values:

	* U_SUCCESS	on success.
	* U_BUSY	if page cannot be currently paged-in.
	* U_FUNCTION	if functionality is not supported
	* U_PARAMETER	if ``lpid`` is invalid.
	* U_P2 		if ``src_ra`` is invalid.
	* U_P3		if the ``dest_gpa`` address is invalid.
	* U_P4		if any bit in the ``flags`` is unrecognized
	* U_P5		if the ``order`` parameter is unsupported.

Description
~~~~~~~~~~~

    Move the contents of the page identified by ``src_ra`` from normal
    memory to secure memory and map it to the guest physical address
    ``dest_gpa``.

    If `dest_gpa` refers to a shared address, map the page into the
    partition-scoped page-table of the SVM.  If `dest_gpa` is not shared,
    copy the contents of the page into the corresponding secure page.
    Depending on the context, decrypt the page before being copied.

    The caller provides the attributes of the page through the ``flags``
    parameter. Valid values for ``flags`` are:

	* CACHE_INHIBITED
	* CACHE_ENABLED
	* WRITE_PROTECTION

    The Hypervisor must pin the page in memory before making
    ``UV_PAGE_IN`` ultracall.

Use cases
~~~~~~~~~

    #. When a normal VM switches to secure mode, all its pages residing
       in normal memory, are moved into secure memory.

    #. When an SVM requests to share a page with Hypervisor the Hypervisor
       allocates a page and informs the Ultravisor.

    #. When an SVM accesses a secure page that has been paged-out,
       Ultravisor invokes the Hypervisor to locate the page. After
       locating the page, the Hypervisor uses UV_PAGE_IN to make the
       page available to Ultravisor.

UV_PAGE_INVAL
-------------

    Invalidate the Ultravisor mapping of a page.

Syntax
~~~~~~

.. code-block:: c

	uint64_t ultracall(const uint64_t UV_PAGE_INVAL,
		uint16_t lpid,		/* the LPAR ID */
		uint64_t guest_pa,	/* destination guest-physical-address */
		uint64_t order)		/* page size order */

Return values
~~~~~~~~~~~~~

    One of the following values:

	* U_SUCCESS	on success.
	* U_PARAMETER	if ``lpid`` is invalid.
	* U_P2 		if ``guest_pa`` is invalid (or corresponds to a secure
                        page mapping).
	* U_P3		if the ``order`` is invalid.
	* U_FUNCTION	if functionality is not supported.
	* U_BUSY	if page cannot be currently invalidated.

Description
~~~~~~~~~~~

    This ultracall informs Ultravisor that the page mapping in Hypervisor
    corresponding to the given guest physical address has been invalidated
    and that the Ultravisor should not access the page. If the specified
    ``guest_pa`` corresponds to a secure page, Ultravisor will ignore the
    attempt to invalidate the page and return U_P2.

Use cases
~~~~~~~~~

    #. When a shared page is unmapped from the QEMU's page table, possibly
       because it is paged-out to disk, Ultravisor needs to know that the
       page should not be accessed from its side too.

UV_WRITE_PATE
-------------

    Validate and write the partition table entry (PATE) for a given
    partition.

Syntax
~~~~~~

.. code-block:: c

	uint64_t ultracall(const uint64_t UV_WRITE_PATE,
		uint32_t lpid,		/* the LPAR ID */
		uint64_t dw0		/* the first double word to write */
		uint64_t dw1)		/* the second double word to write */

Return values
~~~~~~~~~~~~~

    One of the following values:

	* U_SUCCESS	on success.
	* U_BUSY	if PATE cannot be currently written to.
	* U_FUNCTION	if functionality is not supported.
	* U_PARAMETER	if ``lpid`` is invalid.
	* U_P2 		if ``dw0`` is invalid.
	* U_P3		if the ``dw1`` address is invalid.
	* U_PERMISSION	if the Hypervisor is attempting to change the PATE
			of a secure virtual machine or if called from a
			context other than Hypervisor.

Description
~~~~~~~~~~~

    Validate and write a LPID and its partition-table-entry for the given
    LPID.  If the LPID is already allocated and initialized, this call
    results in changing the partition table entry.

Use cases
~~~~~~~~~

    #. The Partition table resides in Secure memory and its entries,
       called PATE (Partition Table Entries), point to the partition-
       scoped page tables for the Hypervisor as well as each of the
       virtual machines (both secure and normal). The Hypervisor
       operates in partition 0 and its partition-scoped page tables
       reside in normal memory.

    #. This ultracall allows the Hypervisor to register the partition-
       scoped and process-scoped page table entries for the Hypervisor
       and other partitions (virtual machines) with the Ultravisor.

    #. If the value of the PATE for an existing partition (VM) changes,
       the TLB cache for the partition is flushed.

    #. The Hypervisor is responsible for allocating LPID. The LPID and
       its PATE entry are registered together.  The Hypervisor manages
       the PATE entries for a normal VM and can change the PATE entry
       anytime. Ultravisor manages the PATE entries for an SVM and
       Hypervisor is not allowed to modify them.

UV_RETURN
---------

    Return control from the Hypervisor back to the Ultravisor after
    processing an hypercall or interrupt that was forwarded (aka
    *reflected*) to the Hypervisor.

Syntax
~~~~~~

.. code-block:: c

	uint64_t ultracall(const uint64_t UV_RETURN)

Return values
~~~~~~~~~~~~~

     This call never returns to Hypervisor on success.  It returns
     U_INVALID if ultracall is not made from a Hypervisor context.

Description
~~~~~~~~~~~

    When an SVM makes an hypercall or incurs some other exception, the
    Ultravisor usually forwards (aka *reflects*) the exceptions to the
    Hypervisor.  After processing the exception, Hypervisor uses the
    ``UV_RETURN`` ultracall to return control back to the SVM.

    The expected register state on entry to this ultracall is:

    * Non-volatile registers are restored to their original values.
    * If returning from an hypercall, register R0 contains the return
      value (**unlike other ultracalls**) and, registers R4 through R12
      contain any output values of the hypercall.
    * R3 contains the ultracall number, i.e UV_RETURN.
    * If returning with a synthesized interrupt, R2 contains the
      synthesized interrupt number.

Use cases
~~~~~~~~~

    #. Ultravisor relies on the Hypervisor to provide several services to
       the SVM such as processing hypercall and other exceptions. After
       processing the exception, Hypervisor uses UV_RETURN to return
       control back to the Ultravisor.

    #. Hypervisor has to use this ultracall to return control to the SVM.

UV_REGISTER_MEM_SLOT
--------------------

    Register an SVM address-range with specified properties.

Syntax
~~~~~~

.. code-block:: c

	uint64_t ultracall(const uint64_t UV_REGISTER_MEM_SLOT,
		uint64_t lpid,		/* LPAR ID of the SVM */
		uint64_t start_gpa,	/* start guest physical address */
		uint64_t size,		/* size of address range in bytes */
		uint64_t flags		/* reserved for future expansion */
		uint16_t slotid)	/* slot identifier */

Return values
~~~~~~~~~~~~~

    One of the following values:

	* U_SUCCESS	on success.
	* U_PARAMETER	if ``lpid`` is invalid.
	* U_P2 		if ``start_gpa`` is invalid.
	* U_P3		if ``size`` is invalid.
	* U_P4		if any bit in the ``flags`` is unrecognized.
	* U_P5		if the ``slotid`` parameter is unsupported.
	* U_PERMISSION	if called from context other than Hypervisor.
	* U_FUNCTION	if functionality is not supported.

Description
~~~~~~~~~~~

    Register a memory range for an SVM.  The memory range starts at the
    guest physical address ``start_gpa`` and is ``size`` bytes long.

Use cases
~~~~~~~~~

    #. When a virtual machine goes secure, all the memory slots managed by
       the Hypervisor move into secure memory. The Hypervisor iterates
       through each of memory slots, and registers the slot with
       Ultravisor.  Hypervisor may discard some slots such as those used
       for firmware (SLOF).

    #. When new memory is hot-plugged, a new memory slot gets registered.

UV_UNREGISTER_MEM_SLOT
----------------------

    Unregister an SVM address-range that was previously registered using
    UV_REGISTER_MEM_SLOT.

Syntax
~~~~~~

.. code-block:: c

	uint64_t ultracall(const uint64_t UV_UNREGISTER_MEM_SLOT,
		uint64_t lpid,		/* LPAR ID of the SVM */
		uint64_t slotid)	/* reservation slotid */

Return values
~~~~~~~~~~~~~

    One of the following values:

	* U_SUCCESS	on success.
	* U_FUNCTION	if functionality is not supported.
	* U_PARAMETER	if ``lpid`` is invalid.
	* U_P2 		if ``slotid`` is invalid.
	* U_PERMISSION	if called from context other than Hypervisor.

Description
~~~~~~~~~~~

    Release the memory slot identified by ``slotid`` and free any
    resources allocated towards the reservation.

Use cases
~~~~~~~~~

    #. Memory hot-remove.

UV_SVM_TERMINATE
----------------

    Terminate an SVM and release its resources.

Syntax
~~~~~~

.. code-block:: c

	uint64_t ultracall(const uint64_t UV_SVM_TERMINATE,
		uint64_t lpid,		/* LPAR ID of the SVM */)

Return values
~~~~~~~~~~~~~

    One of the following values:

	* U_SUCCESS	on success.
	* U_FUNCTION	if functionality is not supported.
	* U_PARAMETER	if ``lpid`` is invalid.
	* U_INVALID	if VM is not secure.
	* U_PERMISSION  if not called from a Hypervisor context.

Description
~~~~~~~~~~~

    Terminate an SVM and release all its resources.

Use cases
~~~~~~~~~

    #. Called by Hypervisor when terminating an SVM.

Ultracalls used by SVM
======================

UV_SHARE_PAGE
-------------

    Share a set of guest physical pages with the Hypervisor.

Syntax
~~~~~~

.. code-block:: c

	uint64_t ultracall(const uint64_t UV_SHARE_PAGE,
		uint64_t gfn,	/* guest page frame number */
		uint64_t num)	/* number of pages of size PAGE_SIZE */

Return values
~~~~~~~~~~~~~

    One of the following values:

	* U_SUCCESS	on success.
	* U_FUNCTION	if functionality is not supported.
	* U_INVALID	if the VM is not secure.
	* U_PARAMETER	if ``gfn`` is invalid.
	* U_P2 		if ``num`` is invalid.

Description
~~~~~~~~~~~

    Share the ``num`` pages starting at guest physical frame number ``gfn``
    with the Hypervisor. Assume page size is PAGE_SIZE bytes. Zero the
    pages before returning.

    If the address is already backed by a secure page, unmap the page and
    back it with an insecure page, with the help of the Hypervisor. If it
    is not backed by any page yet, mark the PTE as insecure and back it
    with an insecure page when the address is accessed. If it is already
    backed by an insecure page, zero the page and return.

Use cases
~~~~~~~~~

    #. The Hypervisor cannot access the SVM pages since they are backed by
       secure pages. Hence an SVM must explicitly request Ultravisor for
       pages it can share with Hypervisor.

    #. Shared pages are needed to support virtio and Virtual Processor Area
       (VPA) in SVMs.

UV_UNSHARE_PAGE
---------------

    Restore a shared SVM page to its initial state.

Syntax
~~~~~~

.. code-block:: c

	uint64_t ultracall(const uint64_t UV_UNSHARE_PAGE,
		uint64_t gfn,	/* guest page frame number */
		uint73 num)	/* number of pages of size PAGE_SIZE*/

Return values
~~~~~~~~~~~~~

    One of the following values:

	* U_SUCCESS	on success.
	* U_FUNCTION	if functionality is not supported.
	* U_INVALID	if VM is not secure.
	* U_PARAMETER	if ``gfn`` is invalid.
	* U_P2 		if ``num`` is invalid.

Description
~~~~~~~~~~~

    Stop sharing ``num`` pages starting at ``gfn`` with the Hypervisor.
    Assume that the page size is PAGE_SIZE. Zero the pages before
    returning.

    If the address is already backed by an insecure page, unmap the page
    and back it with a secure page. Inform the Hypervisor to release
    reference to its shared page. If the address is not backed by a page
    yet, mark the PTE as secure and back it with a secure page when that
    address is accessed. If it is already backed by an secure page zero
    the page and return.

Use cases
~~~~~~~~~

    #. The SVM may decide to unshare a page from the Hypervisor.

UV_UNSHARE_ALL_PAGES
--------------------

    Unshare all pages the SVM has shared with Hypervisor.

Syntax
~~~~~~

.. code-block:: c

	uint64_t ultracall(const uint64_t UV_UNSHARE_ALL_PAGES)

Return values
~~~~~~~~~~~~~

    One of the following values:

	* U_SUCCESS	on success.
	* U_FUNCTION	if functionality is not supported.
	* U_INVAL	if VM is not secure.

Description
~~~~~~~~~~~

    Unshare all shared pages from the Hypervisor. All unshared pages are
    zeroed on return. Only pages explicitly shared by the SVM with the
    Hypervisor (using UV_SHARE_PAGE ultracall) are unshared. Ultravisor
    may internally share some pages with the Hypervisor without explicit
    request from the SVM.  These pages will not be unshared by this
    ultracall.

Use cases
~~~~~~~~~

    #. This call is needed when ``kexec`` is used to boot a different
       kernel. It may also be needed during SVM reset.

UV_ESM
------

    Secure the virtual machine (*enter secure mode*).

Syntax
~~~~~~

.. code-block:: c

	uint64_t ultracall(const uint64_t UV_ESM,
		uint64_t esm_blob_addr,	/* location of the ESM blob */
		unint64_t fdt)		/* Flattened device tree */

Return values
~~~~~~~~~~~~~

    One of the following values:

	* U_SUCCESS	on success (including if VM is already secure).
	* U_FUNCTION	if functionality is not supported.
	* U_INVALID	if VM is not secure.
	* U_PARAMETER	if ``esm_blob_addr`` is invalid.
	* U_P2 		if ``fdt`` is invalid.
	* U_PERMISSION	if any integrity checks fail.
	* U_RETRY	insufficient memory to create SVM.
	* U_NO_KEY	symmetric key unavailable.

Description
~~~~~~~~~~~

    Secure the virtual machine. On successful completion, return
    control to the virtual machine at the address specified in the
    ESM blob.

Use cases
~~~~~~~~~

    #. A normal virtual machine can choose to switch to a secure mode.

Hypervisor Calls API
####################

    This document describes the Hypervisor calls (hypercalls) that are
    needed to support the Ultravisor. Hypercalls are services provided by
    the Hypervisor to virtual machines and Ultravisor.

    Register usage for these hypercalls is identical to that of the other
    hypercalls defined in the Power Architecture Platform Reference (PAPR)
    document.  i.e on input, register R3 identifies the specific service
    that is being requested and registers R4 through R11 contain
    additional parameters to the hypercall, if any. On output, register
    R3 contains the return value and registers R4 through R9 contain any
    other output values from the hypercall.

    This document only covers hypercalls currently implemented/planned
    for Ultravisor usage but others can be added here when it makes sense.

    The full specification for all hypercalls/ultracalls will eventually
    be made available in the public/OpenPower version of the PAPR
    specification.

Hypervisor calls to support Ultravisor
======================================

    Following are the set of hypercalls needed to support Ultravisor.

H_SVM_INIT_START
----------------

    Begin the process of converting a normal virtual machine into an SVM.

Syntax
~~~~~~

.. code-block:: c

	uint64_t hypercall(const uint64_t H_SVM_INIT_START)

Return values
~~~~~~~~~~~~~

    One of the following values:

	* H_SUCCESS	 on success.
        * H_STATE        if the VM is not in a position to switch to secure.

Description
~~~~~~~~~~~

    Initiate the process of securing a virtual machine. This involves
    coordinating with the Ultravisor, using ultracalls, to allocate
    resources in the Ultravisor for the new SVM, transferring the VM's
    pages from normal to secure memory etc. When the process is
    completed, Ultravisor issues the H_SVM_INIT_DONE hypercall.

Use cases
~~~~~~~~~

     #. Ultravisor uses this hypercall to inform Hypervisor that a VM
        has initiated the process of switching to secure mode.

H_SVM_INIT_DONE
---------------

    Complete the process of securing an SVM.

Syntax
~~~~~~

.. code-block:: c

	uint64_t hypercall(const uint64_t H_SVM_INIT_DONE)

Return values
~~~~~~~~~~~~~

    One of the following values:

	* H_SUCCESS 		on success.
	* H_UNSUPPORTED		if called from the wrong context (e.g.
				from an SVM or before an H_SVM_INIT_START
				hypercall).
	* H_STATE		if the hypervisor could not successfully
                                transition the VM to Secure VM.

Description
~~~~~~~~~~~

    Complete the process of securing a virtual machine. This call must
    be made after a prior call to ``H_SVM_INIT_START`` hypercall.

Use cases
~~~~~~~~~

    On successfully securing a virtual machine, the Ultravisor informs
    Hypervisor about it. Hypervisor can use this call to finish setting
    up its internal state for this virtual machine.

H_SVM_INIT_ABORT
----------------

    Abort the process of securing an SVM.

Syntax
~~~~~~

.. code-block:: c

	uint64_t hypercall(const uint64_t H_SVM_INIT_ABORT)

Return values
~~~~~~~~~~~~~

    One of the following values:

	* H_PARAMETER 		on successfully cleaning up the state,
				Hypervisor will return this value to the
				**guest**, to indicate that the underlying
				UV_ESM ultracall failed.

	* H_STATE		if called after a VM has gone secure (i.e
				H_SVM_INIT_DONE hypercall was successful).

	* H_UNSUPPORTED		if called from a wrong context (e.g. from a
				normal VM).

Description
~~~~~~~~~~~

    Abort the process of securing a virtual machine. This call must
    be made after a prior call to ``H_SVM_INIT_START`` hypercall and
    before a call to ``H_SVM_INIT_DONE``.

    On entry into this hypercall the non-volatile GPRs and FPRs are
    expected to contain the values they had at the time the VM issued
    the UV_ESM ultracall. Further ``SRR0`` is expected to contain the
    address of the instruction after the ``UV_ESM`` ultracall and ``SRR1``
    the MSR value with which to return to the VM.

    This hypercall will cleanup any partial state that was established for
    the VM since the prior ``H_SVM_INIT_START`` hypercall, including paging
    out pages that were paged-into secure memory, and issue the
    ``UV_SVM_TERMINATE`` ultracall to terminate the VM.

    After the partial state is cleaned up, control returns to the VM
    (**not Ultravisor**), at the address specified in ``SRR0`` with the
    MSR values set to the value in ``SRR1``.

Use cases
~~~~~~~~~

    If after a successful call to ``H_SVM_INIT_START``, the Ultravisor
    encounters an error while securing a virtual machine, either due
    to lack of resources or because the VM's security information could
    not be validated, Ultravisor informs the Hypervisor about it.
    Hypervisor should use this call to clean up any internal state for
    this virtual machine and return to the VM.

H_SVM_PAGE_IN
-------------

    Move the contents of a page from normal memory to secure memory.

Syntax
~~~~~~

.. code-block:: c

	uint64_t hypercall(const uint64_t H_SVM_PAGE_IN,
		uint64_t guest_pa,	/* guest-physical-address */
		uint64_t flags,		/* flags */
		uint64_t order)		/* page size order */

Return values
~~~~~~~~~~~~~

    One of the following values:

	* H_SUCCESS	on success.
	* H_PARAMETER	if ``guest_pa`` is invalid.
	* H_P2		if ``flags`` is invalid.
	* H_P3		if ``order`` of page is invalid.

Description
~~~~~~~~~~~

    Retrieve the content of the page, belonging to the VM at the specified
    guest physical address.

    Only valid value(s) in ``flags`` are:

        * H_PAGE_IN_SHARED which indicates that the page is to be shared
	  with the Ultravisor.

	* H_PAGE_IN_NONSHARED indicates that the UV is not anymore
          interested in the page. Applicable if the page is a shared page.

    The ``order`` parameter must correspond to the configured page size.

Use cases
~~~~~~~~~

    #. When a normal VM becomes a secure VM (using the UV_ESM ultracall),
       the Ultravisor uses this hypercall to move contents of each page of
       the VM from normal memory to secure memory.

    #. Ultravisor uses this hypercall to ask Hypervisor to provide a page
       in normal memory that can be shared between the SVM and Hypervisor.

    #. Ultravisor uses this hypercall to page-in a paged-out page. This
       can happen when the SVM touches a paged-out page.

    #. If SVM wants to disable sharing of pages with Hypervisor, it can
       inform Ultravisor to do so. Ultravisor will then use this hypercall
       and inform Hypervisor that it has released access to the normal
       page.

H_SVM_PAGE_OUT
---------------

    Move the contents of the page to normal memory.

Syntax
~~~~~~

.. code-block:: c

	uint64_t hypercall(const uint64_t H_SVM_PAGE_OUT,
		uint64_t guest_pa,	/* guest-physical-address */
		uint64_t flags,		/* flags (currently none) */
		uint64_t order)		/* page size order */

Return values
~~~~~~~~~~~~~

    One of the following values:

	* H_SUCCESS	on success.
	* H_PARAMETER	if ``guest_pa`` is invalid.
	* H_P2		if ``flags`` is invalid.
	* H_P3		if ``order`` is invalid.

Description
~~~~~~~~~~~

    Move the contents of the page identified by ``guest_pa`` to normal
    memory.

    Currently ``flags`` is unused and must be set to 0. The ``order``
    parameter must correspond to the configured page size.

Use cases
~~~~~~~~~

    #. If Ultravisor is running low on secure pages, it can move the
       contents of some secure pages, into normal pages using this
       hypercall. The content will be encrypted.

References
##########

- `Supporting Protected Computing on IBM Power Architecture <https://developer.ibm.com/articles/l-support-protected-computing/>`_


code:
.. SPDX-License-Identifier: GPL-2.0
.. _VAS-API:

===================================================
Virtual Accelerator Switchboard (VAS) userspace API
===================================================

Introduction
============

Power9 processor introduced Virtual Accelerator Switchboard (VAS) which
allows both userspace and kernel communicate to co-processor
(hardware accelerator) referred to as the Nest Accelerator (NX). The NX
unit comprises of one or more hardware engines or co-processor types
such as 842 compression, GZIP compression and encryption. On power9,
userspace applications will have access to only GZIP Compression engine
which supports ZLIB and GZIP compression algorithms in the hardware.

To communicate with NX, kernel has to establish a channel or window and
then requests can be submitted directly without kernel involvement.
Requests to the GZIP engine must be formatted as a co-processor Request
Block (CRB) and these CRBs must be submitted to the NX using COPY/PASTE
instructions to paste the CRB to hardware address that is associated with
the engine's request queue.

The GZIP engine provides two priority levels of requests: Normal and
High. Only Normal requests are supported from userspace right now.

This document explains userspace API that is used to interact with
kernel to setup channel / window which can be used to send compression
requests directly to NX accelerator.

Overview
========

Application access to the GZIP engine is provided through
/dev/crypto/nx-gzip device node implemented by the VAS/NX device driver.
An application must open the /dev/crypto/nx-gzip device to obtain a file
descriptor (fd). Then should issue VAS_TX_WIN_OPEN ioctl with this fd to
establish connection to the engine. It means send window is opened on GZIP
engine for this process. Once a connection is established, the application
should use the mmap() system call to map the hardware address of engine's
request queue into the application's virtual address space.

The application can then submit one or more requests to the engine by
using copy/paste instructions and pasting the CRBs to the virtual address
(aka paste_address) returned by mmap(). User space can close the
established connection or send window by closing the file descriptior
(close(fd)) or upon the process exit.

Note that applications can send several requests with the same window or
can establish multiple windows, but one window for each file descriptor.

Following sections provide additional details and references about the
individual steps.

NX-GZIP Device Node
===================

There is one /dev/crypto/nx-gzip node in the system and it provides
access to all GZIP engines in the system. The only valid operations on
/dev/crypto/nx-gzip are:

	* open() the device for read and write.
	* issue VAS_TX_WIN_OPEN ioctl
	* mmap() the engine's request queue into application's virtual
	  address space (i.e. get a paste_address for the co-processor
	  engine).
	* close the device node.

Other file operations on this device node are undefined.

Note that the copy and paste operations go directly to the hardware and
do not go through this device. Refer COPY/PASTE document for more
details.

Although a system may have several instances of the NX co-processor
engines (typically, one per P9 chip) there is just one
/dev/crypto/nx-gzip device node in the system. When the nx-gzip device
node is opened, Kernel opens send window on a suitable instance of NX
accelerator. It finds CPU on which the user process is executing and
determine the NX instance for the corresponding chip on which this CPU
belongs.

Applications may chose a specific instance of the NX co-processor using
the vas_id field in the VAS_TX_WIN_OPEN ioctl as detailed below.

A userspace library libnxz is available here but still in development:

	 https://github.com/abalib/power-gzip

Applications that use inflate / deflate calls can link with libnxz
instead of libz and use NX GZIP compression without any modification.

Open /dev/crypto/nx-gzip
========================

The nx-gzip device should be opened for read and write. No special
privileges are needed to open the device. Each window corresponds to one
file descriptor. So if the userspace process needs multiple windows,
several open calls have to be issued.

See open(2) system call man pages for other details such as return values,
error codes and restrictions.

VAS_TX_WIN_OPEN ioctl
=====================

Applications should use the VAS_TX_WIN_OPEN ioctl as follows to establish
a connection with NX co-processor engine:

	::

		struct vas_tx_win_open_attr {
			__u32   version;
			__s16   vas_id; /* specific instance of vas or -1
						for default */
			__u16   reserved1;
			__u64   flags;	/* For future use */
			__u64   reserved2[6];
		};

	version:
		The version field must be currently set to 1.
	vas_id:
		If '-1' is passed, kernel will make a best-effort attempt
		to assign an optimal instance of NX for the process. To
		select the specific VAS instance, refer
		"Discovery of available VAS engines" section below.

	flags, reserved1 and reserved2[6] fields are for future extension
	and must be set to 0.

	The attributes attr for the VAS_TX_WIN_OPEN ioctl are defined as
	follows::

		#define VAS_MAGIC 'v'
		#define VAS_TX_WIN_OPEN _IOW(VAS_MAGIC, 1,
						struct vas_tx_win_open_attr)

		struct vas_tx_win_open_attr attr;
		rc = ioctl(fd, VAS_TX_WIN_OPEN, &attr);

	The VAS_TX_WIN_OPEN ioctl returns 0 on success. On errors, it
	returns -1 and sets the errno variable to indicate the error.

	Error conditions:

		======	================================================
		EINVAL	fd does not refer to a valid VAS device.
		EINVAL	Invalid vas ID
		EINVAL	version is not set with proper value
		EEXIST	Window is already opened for the given fd
		ENOMEM	Memory is not available to allocate window
		ENOSPC	System has too many active windows (connections)
			opened
		EINVAL	reserved fields are not set to 0.
		======	================================================

	See the ioctl(2) man page for more details, error codes and
	restrictions.

mmap() NX-GZIP device
=====================

The mmap() system call for a NX-GZIP device fd returns a paste_address
that the application can use to copy/paste its CRB to the hardware engines.

	::

		paste_addr = mmap(addr, size, prot, flags, fd, offset);

	Only restrictions on mmap for a NX-GZIP device fd are:

		* size should be PAGE_SIZE
		* offset parameter should be 0ULL

	Refer to mmap(2) man page for additional details/restrictions.
	In addition to the error conditions listed on the mmap(2) man
	page, can also fail with one of the following error codes:

		======	=============================================
		EINVAL	fd is not associated with an open window
			(i.e mmap() does not follow a successful call
			to the VAS_TX_WIN_OPEN ioctl).
		EINVAL	offset field is not 0ULL.
		======	=============================================

Discovery of available VAS engines
==================================

Each available VAS instance in the system will have a device tree node
like /proc/device-tree/vas@* or /proc/device-tree/xscom@*/vas@*.
Determine the chip or VAS instance and use the corresponding ibm,vas-id
property value in this node to select specific VAS instance.

Copy/Paste operations
=====================

Applications should use the copy and paste instructions to send CRB to NX.
Refer section 4.4 in PowerISA for Copy/Paste instructions:
https://openpowerfoundation.org/?resource_lib=power-isa-version-3-0

CRB Specification and use NX
============================

Applications should format requests to the co-processor using the
co-processor Request Block (CRBs). Refer NX-GZIP user's manual for the format
of CRB and use NX from userspace such as sending requests and checking
request status.

NX Fault handling
=================

Applications send requests to NX and wait for the status by polling on
co-processor Status Block (CSB) flags. NX updates status in CSB after each
request is processed. Refer NX-GZIP user's manual for the format of CSB and
status flags.

In case if NX encounters translation error (called NX page fault) on CSB
address or any request buffer, raises an interrupt on the CPU to handle the
fault. Page fault can happen if an application passes invalid addresses or
request buffers are not in memory. The operating system handles the fault by
updating CSB with the following data::

	csb.flags = CSB_V;
	csb.cc = CSB_CC_FAULT_ADDRESS;
	csb.ce = CSB_CE_TERMINATION;
	csb.address = fault_address;

When an application receives translation error, it can touch or access
the page that has a fault address so that this page will be in memory. Then
the application can resend this request to NX.

If the OS can not update CSB due to invalid CSB address, sends SEGV signal
to the process who opened the send window on which the original request was
issued. This signal returns with the following siginfo struct::

	siginfo.si_signo = SIGSEGV;
	siginfo.si_errno = EFAULT;
	siginfo.si_code = SEGV_MAPERR;
	siginfo.si_addr = CSB adress;

In the case of multi-thread applications, NX send windows can be shared
across all threads. For example, a child thread can open a send window,
but other threads can send requests to NX using this window. These
requests will be successful even in the case of OS handling faults as long
as CSB address is valid. If the NX request contains an invalid CSB address,
the signal will be sent to the child thread that opened the window. But if
the thread is exited without closing the window and the request is issued
using this window. the signal will be issued to the thread group leader
(tgid). It is up to the application whether to ignore or handle these
signals.

NX-GZIP User's Manual:
https://github.com/libnxz/power-gzip/blob/master/doc/power_nx_gzip_um.pdf

Simple example
==============

	::

		int use_nx_gzip()
		{
			int rc, fd;
			void *addr;
			struct vas_setup_attr txattr;

			fd = open("/dev/crypto/nx-gzip", O_RDWR);
			if (fd < 0) {
				fprintf(stderr, "open nx-gzip failed\n");
				return -1;
			}
			memset(&txattr, 0, sizeof(txattr));
			txattr.version = 1;
			txattr.vas_id = -1
			rc = ioctl(fd, VAS_TX_WIN_OPEN,
					(unsigned long)&txattr);
			if (rc < 0) {
				fprintf(stderr, "ioctl() n %d, error %d\n",
						rc, errno);
				return rc;
			}
			addr = mmap(NULL, 4096, PROT_READ|PROT_WRITE,
					MAP_SHARED, fd, 0ULL);
			if (addr == MAP_FAILED) {
				fprintf(stderr, "mmap() failed, errno %d\n",
						errno);
				return -errno;
			}
			do {
				//Format CRB request with compression or
				//uncompression
				// Refer tests for vas_copy/vas_paste
				vas_copy((&crb, 0, 1);
				vas_paste(addr, 0, 1);
				// Poll on csb.flags with timeout
				// csb address is listed in CRB
			} while (true)
			close(fd) or window can be closed upon process exit
		}

	Refer https://github.com/libnxz/power-gzip for tests or more
	use cases.

repo_name: angery/linux


code:
.. SPDX-License-Identifier: GPL-2.0

========================
VCPU Dispatch Statistics
========================

For Shared Processor LPARs, the POWER Hypervisor maintains a relatively
static mapping of the LPAR processors (vcpus) to physical processor
chips (representing the "home" node) and tries to always dispatch vcpus
on their associated physical processor chip. However, under certain
scenarios, vcpus may be dispatched on a different processor chip (away
from its home node).

/proc/powerpc/vcpudispatch_stats can be used to obtain statistics
related to the vcpu dispatch behavior. Writing '1' to this file enables
collecting the statistics, while writing '0' disables the statistics.
By default, the DTLB log for each vcpu is processed 50 times a second so
as not to miss any entries. This processing frequency can be changed
through /proc/powerpc/vcpudispatch_stats_freq.

The statistics themselves are available by reading the procfs file
/proc/powerpc/vcpudispatch_stats. Each line in the output corresponds to
a vcpu as represented by the first field, followed by 8 numbers.

The first number corresponds to:

1. total vcpu dispatches since the beginning of statistics collection

The next 4 numbers represent vcpu dispatch dispersions:

2. number of times this vcpu was dispatched on the same processor as last
   time
3. number of times this vcpu was dispatched on a different processor core
   as last time, but within the same chip
4. number of times this vcpu was dispatched on a different chip
5. number of times this vcpu was dispatches on a different socket/drawer
   (next numa boundary)

The final 3 numbers represent statistics in relation to the home node of
the vcpu:

6. number of times this vcpu was dispatched in its home node (chip)
7. number of times this vcpu was dispatched in a different node
8. number of times this vcpu was dispatched in a node further away (numa
   distance)

An example output::

    $ sudo cat /proc/powerpc/vcpudispatch_stats
    cpu0 6839 4126 2683 30 0 6821 18 0
    cpu1 2515 1274 1229 12 0 2509 6 0
    cpu2 2317 1198 1109 10 0 2312 5 0
    cpu3 2259 1165 1088 6 0 2256 3 0
    cpu4 2205 1143 1056 6 0 2202 3 0
    cpu5 2165 1121 1038 6 0 2162 3 0
    cpu6 2183 1127 1050 6 0 2180 3 0
    cpu7 2193 1133 1052 8 0 2187 6 0
    cpu8 2165 1115 1032 18 0 2156 9 0
    cpu9 2301 1252 1033 16 0 2293 8 0
    cpu10 2197 1138 1041 18 0 2187 10 0
    cpu11 2273 1185 1062 26 0 2260 13 0
    cpu12 2186 1125 1043 18 0 2177 9 0
    cpu13 2161 1115 1030 16 0 2153 8 0
    cpu14 2206 1153 1033 20 0 2196 10 0
    cpu15 2163 1115 1032 16 0 2155 8 0

In the output above, for vcpu0, there have been 6839 dispatches since
statistics were enabled. 4126 of those dispatches were on the same
physical cpu as the last time. 2683 were on a different core, but within
the same chip, while 30 dispatches were on a different chip compared to
its last dispatch.

Also, out of the total of 6839 dispatches, we see that there have been
6821 dispatches on the vcpu's home node, while 18 dispatches were
outside its home node, on a neighbouring chip.


code:
.. _development_process_intro:

Introduction
============

Executive summary
-----------------

The rest of this section covers the scope of the kernel development process
and the kinds of frustrations that developers and their employers can
encounter there.  There are a great many reasons why kernel code should be
merged into the official ("mainline") kernel, including automatic
availability to users, community support in many forms, and the ability to
influence the direction of kernel development.  Code contributed to the
Linux kernel must be made available under a GPL-compatible license.

:ref:`development_process` introduces the development process, the kernel
release cycle, and the mechanics of the merge window.  The various phases in
the patch development, review, and merging cycle are covered.  There is some
discussion of tools and mailing lists.  Developers wanting to get started
with kernel development are encouraged to track down and fix bugs as an
initial exercise.

:ref:`development_early_stage` covers early-stage project planning, with an
emphasis on involving the development community as soon as possible.

:ref:`development_coding` is about the coding process; several pitfalls which
have been encountered by other developers are discussed.  Some requirements for
patches are covered, and there is an introduction to some of the tools
which can help to ensure that kernel patches are correct.

:ref:`development_posting` talks about the process of posting patches for
review. To be taken seriously by the development community, patches must be
properly formatted and described, and they must be sent to the right place.
Following the advice in this section should help to ensure the best
possible reception for your work.

:ref:`development_followthrough` covers what happens after posting patches; the
job is far from done at that point.  Working with reviewers is a crucial part
of the development process; this section offers a number of tips on how to
avoid problems at this important stage.  Developers are cautioned against
assuming that the job is done when a patch is merged into the mainline.

:ref:`development_advancedtopics` introduces a couple of "advanced" topics:
managing patches with git and reviewing patches posted by others.

:ref:`development_conclusion` concludes the document with pointers to sources
for more information on kernel development.

What this document is about
---------------------------

The Linux kernel, at over 8 million lines of code and well over 1000
contributors to each release, is one of the largest and most active free
software projects in existence.  Since its humble beginning in 1991, this
kernel has evolved into a best-of-breed operating system component which
runs on pocket-sized digital music players, desktop PCs, the largest
supercomputers in existence, and all types of systems in between.  It is a
robust, efficient, and scalable solution for almost any situation.

With the growth of Linux has come an increase in the number of developers
(and companies) wishing to participate in its development.  Hardware
vendors want to ensure that Linux supports their products well, making
those products attractive to Linux users.  Embedded systems vendors, who
use Linux as a component in an integrated product, want Linux to be as
capable and well-suited to the task at hand as possible.  Distributors and
other software vendors who base their products on Linux have a clear
interest in the capabilities, performance, and reliability of the Linux
kernel.  And end users, too, will often wish to change Linux to make it
better suit their needs.

One of the most compelling features of Linux is that it is accessible to
these developers; anybody with the requisite skills can improve Linux and
influence the direction of its development.  Proprietary products cannot
offer this kind of openness, which is a characteristic of the free software
process.  But, if anything, the kernel is even more open than most other
free software projects.  A typical three-month kernel development cycle can
involve over 1000 developers working for more than 100 different companies
(or for no company at all).

Working with the kernel development community is not especially hard.  But,
that notwithstanding, many potential contributors have experienced
difficulties when trying to do kernel work.  The kernel community has
evolved its own distinct ways of operating which allow it to function
smoothly (and produce a high-quality product) in an environment where
thousands of lines of code are being changed every day.  So it is not
surprising that Linux kernel development process differs greatly from
proprietary development methods.

The kernel's development process may come across as strange and
intimidating to new developers, but there are good reasons and solid
experience behind it.  A developer who does not understand the kernel
community's ways (or, worse, who tries to flout or circumvent them) will
have a frustrating experience in store.  The development community, while
being helpful to those who are trying to learn, has little time for those
who will not listen or who do not care about the development process.

It is hoped that those who read this document will be able to avoid that
frustrating experience.  There is a lot of material here, but the effort
involved in reading it will be repaid in short order.  The development
community is always in need of developers who will help to make the kernel
better; the following text should help you - or those who work for you -
join our community.

Credits
-------

This document was written by Jonathan Corbet, corbet@lwn.net.  It has been
improved by comments from Johannes Berg, James Berry, Alex Chiang, Roland
Dreier, Randy Dunlap, Jake Edge, Jiri Kosina, Matt Mackall, Arthur Marsh,
Amanda McPherson, Andrew Morton, Andrew Price, Tsugikazu Shibata, and
Jochen Voß.

This work was supported by the Linux Foundation; thanks especially to
Amanda McPherson, who saw the value of this effort and made it all happen.

The importance of getting code into the mainline
------------------------------------------------

Some companies and developers occasionally wonder why they should bother
learning how to work with the kernel community and get their code into the
mainline kernel (the "mainline" being the kernel maintained by Linus
Torvalds and used as a base by Linux distributors).  In the short term,
contributing code can look like an avoidable expense; it seems easier to
just keep the code separate and support users directly.  The truth of the
matter is that keeping code separate ("out of tree") is a false economy.

As a way of illustrating the costs of out-of-tree code, here are a few
relevant aspects of the kernel development process; most of these will be
discussed in greater detail later in this document.  Consider:

- Code which has been merged into the mainline kernel is available to all
  Linux users.  It will automatically be present on all distributions which
  enable it.  There is no need for driver disks, downloads, or the hassles
  of supporting multiple versions of multiple distributions; it all just
  works, for the developer and for the user.  Incorporation into the
  mainline solves a large number of distribution and support problems.

- While kernel developers strive to maintain a stable interface to user
  space, the internal kernel API is in constant flux.  The lack of a stable
  internal interface is a deliberate design decision; it allows fundamental
  improvements to be made at any time and results in higher-quality code.
  But one result of that policy is that any out-of-tree code requires
  constant upkeep if it is to work with new kernels.  Maintaining
  out-of-tree code requires significant amounts of work just to keep that
  code working.

  Code which is in the mainline, instead, does not require this work as the
  result of a simple rule requiring any developer who makes an API change
  to also fix any code that breaks as the result of that change.  So code
  which has been merged into the mainline has significantly lower
  maintenance costs.

- Beyond that, code which is in the kernel will often be improved by other
  developers.  Surprising results can come from empowering your user
  community and customers to improve your product.

- Kernel code is subjected to review, both before and after merging into
  the mainline.  No matter how strong the original developer's skills are,
  this review process invariably finds ways in which the code can be
  improved.  Often review finds severe bugs and security problems.  This is
  especially true for code which has been developed in a closed
  environment; such code benefits strongly from review by outside
  developers.  Out-of-tree code is lower-quality code.

- Participation in the development process is your way to influence the
  direction of kernel development.  Users who complain from the sidelines
  are heard, but active developers have a stronger voice - and the ability
  to implement changes which make the kernel work better for their needs.

- When code is maintained separately, the possibility that a third party
  will contribute a different implementation of a similar feature always
  exists.  Should that happen, getting your code merged will become much
  harder - to the point of impossibility.  Then you will be faced with the
  unpleasant alternatives of either (1) maintaining a nonstandard feature
  out of tree indefinitely, or (2) abandoning your code and migrating your
  users over to the in-tree version.

- Contribution of code is the fundamental action which makes the whole
  process work.  By contributing your code you can add new functionality to
  the kernel and provide capabilities and examples which are of use to
  other kernel developers.  If you have developed code for Linux (or are
  thinking about doing so), you clearly have an interest in the continued
  success of this platform; contributing code is one of the best ways to
  help ensure that success.

All of the reasoning above applies to any out-of-tree kernel code,
including code which is distributed in proprietary, binary-only form.
There are, however, additional factors which should be taken into account
before considering any sort of binary-only kernel code distribution.  These
include:

- The legal issues around the distribution of proprietary kernel modules
  are cloudy at best; quite a few kernel copyright holders believe that
  most binary-only modules are derived products of the kernel and that, as
  a result, their distribution is a violation of the GNU General Public
  license (about which more will be said below).  Your author is not a
  lawyer, and nothing in this document can possibly be considered to be
  legal advice.  The true legal status of closed-source modules can only be
  determined by the courts.  But the uncertainty which haunts those modules
  is there regardless.

- Binary modules greatly increase the difficulty of debugging kernel
  problems, to the point that most kernel developers will not even try.  So
  the distribution of binary-only modules will make it harder for your
  users to get support from the community.

- Support is also harder for distributors of binary-only modules, who must
  provide a version of the module for every distribution and every kernel
  version they wish to support.  Dozens of builds of a single module can
  be required to provide reasonably comprehensive coverage, and your users
  will have to upgrade your module separately every time they upgrade their
  kernel.

- Everything that was said above about code review applies doubly to
  closed-source code.  Since this code is not available at all, it cannot
  have been reviewed by the community and will, beyond doubt, have serious
  problems.

Makers of embedded systems, in particular, may be tempted to disregard much
of what has been said in this section in the belief that they are shipping
a self-contained product which uses a frozen kernel version and requires no
more development after its release.  This argument misses the value of
widespread code review and the value of allowing your users to add
capabilities to your product.  But these products, too, have a limited
commercial life, after which a new version must be released.  At that
point, vendors whose code is in the mainline and well maintained will be
much better positioned to get the new product ready for market quickly.

Licensing
---------

Code is contributed to the Linux kernel under a number of licenses, but all
code must be compatible with version 2 of the GNU General Public License
(GPLv2), which is the license covering the kernel distribution as a whole.
In practice, that means that all code contributions are covered either by
GPLv2 (with, optionally, language allowing distribution under later
versions of the GPL) or the three-clause BSD license.  Any contributions
which are not covered by a compatible license will not be accepted into the
kernel.

Copyright assignments are not required (or requested) for code contributed
to the kernel.  All code merged into the mainline kernel retains its
original ownership; as a result, the kernel now has thousands of owners.

One implication of this ownership structure is that any attempt to change
the licensing of the kernel is doomed to almost certain failure.  There are
few practical scenarios where the agreement of all copyright holders could
be obtained (or their code removed from the kernel).  So, in particular,
there is no prospect of a migration to version 3 of the GPL in the
foreseeable future.

It is imperative that all code contributed to the kernel be legitimately
free software.  For that reason, code from anonymous (or pseudonymous)
contributors will not be accepted.  All contributors are required to "sign
off" on their code, stating that the code can be distributed with the
kernel under the GPL.  Code which has not been licensed as free software by
its owner, or which risks creating copyright-related problems for the
kernel (such as code which derives from reverse-engineering efforts lacking
proper safeguards) cannot be contributed.

Questions about copyright-related issues are common on Linux development
mailing lists.  Such questions will normally receive no shortage of
answers, but one should bear in mind that the people answering those
questions are not lawyers and cannot provide legal advice.  If you have
legal questions relating to Linux source code, there is no substitute for
talking with a lawyer who understands this field.  Relying on answers
obtained on technical mailing lists is a risky affair.


code:
.. _development_process:

How the development process works
=================================

Linux kernel development in the early 1990's was a pretty loose affair,
with relatively small numbers of users and developers involved.  With a
user base in the millions and with some 2,000 developers involved over the
course of one year, the kernel has since had to evolve a number of
processes to keep development happening smoothly.  A solid understanding of
how the process works is required in order to be an effective part of it.

The big picture
---------------

The kernel developers use a loosely time-based release process, with a new
major kernel release happening every two or three months.  The recent
release history looks like this:

	======  =================
	5.0	March 3, 2019
	5.1	May 5, 2019
	5.2	July 7, 2019
	5.3	September 15, 2019
	5.4	November 24, 2019
	5.5	January 6, 2020
	======  =================

Every 5.x release is a major kernel release with new features, internal
API changes, and more.  A typical release can contain about 13,000
changesets with changes to several hundred thousand lines of code.  5.x is
the leading edge of Linux kernel development; the kernel uses a
rolling development model which is continually integrating major changes.

A relatively straightforward discipline is followed with regard to the
merging of patches for each release.  At the beginning of each development
cycle, the "merge window" is said to be open.  At that time, code which is
deemed to be sufficiently stable (and which is accepted by the development
community) is merged into the mainline kernel.  The bulk of changes for a
new development cycle (and all of the major changes) will be merged during
this time, at a rate approaching 1,000 changes ("patches," or "changesets")
per day.

(As an aside, it is worth noting that the changes integrated during the
merge window do not come out of thin air; they have been collected, tested,
and staged ahead of time.  How that process works will be described in
detail later on).

The merge window lasts for approximately two weeks.  At the end of this
time, Linus Torvalds will declare that the window is closed and release the
first of the "rc" kernels.  For the kernel which is destined to be 5.6,
for example, the release which happens at the end of the merge window will
be called 5.6-rc1.  The -rc1 release is the signal that the time to
merge new features has passed, and that the time to stabilize the next
kernel has begun.

Over the next six to ten weeks, only patches which fix problems should be
submitted to the mainline.  On occasion a more significant change will be
allowed, but such occasions are rare; developers who try to merge new
features outside of the merge window tend to get an unfriendly reception.
As a general rule, if you miss the merge window for a given feature, the
best thing to do is to wait for the next development cycle.  (An occasional
exception is made for drivers for previously-unsupported hardware; if they
touch no in-tree code, they cannot cause regressions and should be safe to
add at any time).

As fixes make their way into the mainline, the patch rate will slow over
time.  Linus releases new -rc kernels about once a week; a normal series
will get up to somewhere between -rc6 and -rc9 before the kernel is
considered to be sufficiently stable and the final release is made.
At that point the whole process starts over again.

As an example, here is how the 5.4 development cycle went (all dates in
2019):

	==============  ===============================
	September 15	5.3 stable release
	September 30	5.4-rc1, merge window closes
	October 6	5.4-rc2
	October 13	5.4-rc3
	October 20	5.4-rc4
	October 27	5.4-rc5
	November 3	5.4-rc6
	November 10	5.4-rc7
	November 17	5.4-rc8
	November 24	5.4 stable release
	==============  ===============================

How do the developers decide when to close the development cycle and create
the stable release?  The most significant metric used is the list of
regressions from previous releases.  No bugs are welcome, but those which
break systems which worked in the past are considered to be especially
serious.  For this reason, patches which cause regressions are looked upon
unfavorably and are quite likely to be reverted during the stabilization
period.

The developers' goal is to fix all known regressions before the stable
release is made.  In the real world, this kind of perfection is hard to
achieve; there are just too many variables in a project of this size.
There comes a point where delaying the final release just makes the problem
worse; the pile of changes waiting for the next merge window will grow
larger, creating even more regressions the next time around.  So most 5.x
kernels go out with a handful of known regressions though, hopefully, none
of them are serious.

Once a stable release is made, its ongoing maintenance is passed off to the
"stable team," currently Greg Kroah-Hartman. The stable team will release
occasional updates to the stable release using the 5.x.y numbering scheme.
To be considered for an update release, a patch must (1) fix a significant
bug, and (2) already be merged into the mainline for the next development
kernel. Kernels will typically receive stable updates for a little more
than one development cycle past their initial release. So, for example, the
5.2 kernel's history looked like this (all dates in 2019):

	==============  ===============================
	July 7		5.2 stable release
	July 14		5.2.1
	July 21		5.2.2
	July 26		5.2.3
	July 28		5.2.4
	July 31  	5.2.5
	...		...
	October 11	5.2.21
	==============  ===============================

5.2.21 was the final stable update of the 5.2 release.

Some kernels are designated "long term" kernels; they will receive support
for a longer period.  Please refer to the following link for the list of active
long term kernel versions and their maintainers:

	https://www.kernel.org/category/releases.html

The selection of a kernel for long-term support is purely a matter of a
maintainer having the need and the time to maintain that release.  There
are no known plans for long-term support for any specific upcoming
release.

The lifecycle of a patch
------------------------

Patches do not go directly from the developer's keyboard into the mainline
kernel.  There is, instead, a somewhat involved (if somewhat informal)
process designed to ensure that each patch is reviewed for quality and that
each patch implements a change which is desirable to have in the mainline.
This process can happen quickly for minor fixes, or, in the case of large
and controversial changes, go on for years.  Much developer frustration
comes from a lack of understanding of this process or from attempts to
circumvent it.

In the hopes of reducing that frustration, this document will describe how
a patch gets into the kernel.  What follows below is an introduction which
describes the process in a somewhat idealized way.  A much more detailed
treatment will come in later sections.

The stages that a patch goes through are, generally:

 - Design.  This is where the real requirements for the patch - and the way
   those requirements will be met - are laid out.  Design work is often
   done without involving the community, but it is better to do this work
   in the open if at all possible; it can save a lot of time redesigning
   things later.

 - Early review.  Patches are posted to the relevant mailing list, and
   developers on that list reply with any comments they may have.  This
   process should turn up any major problems with a patch if all goes
   well.

 - Wider review.  When the patch is getting close to ready for mainline
   inclusion, it should be accepted by a relevant subsystem maintainer -
   though this acceptance is not a guarantee that the patch will make it
   all the way to the mainline.  The patch will show up in the maintainer's
   subsystem tree and into the -next trees (described below).  When the
   process works, this step leads to more extensive review of the patch and
   the discovery of any problems resulting from the integration of this
   patch with work being done by others.

-  Please note that most maintainers also have day jobs, so merging
   your patch may not be their highest priority.  If your patch is
   getting feedback about changes that are needed, you should either
   make those changes or justify why they should not be made.  If your
   patch has no review complaints but is not being merged by its
   appropriate subsystem or driver maintainer, you should be persistent
   in updating the patch to the current kernel so that it applies cleanly
   and keep sending it for review and merging.

 - Merging into the mainline.  Eventually, a successful patch will be
   merged into the mainline repository managed by Linus Torvalds.  More
   comments and/or problems may surface at this time; it is important that
   the developer be responsive to these and fix any issues which arise.

 - Stable release.  The number of users potentially affected by the patch
   is now large, so, once again, new problems may arise.

 - Long-term maintenance.  While it is certainly possible for a developer
   to forget about code after merging it, that sort of behavior tends to
   leave a poor impression in the development community.  Merging code
   eliminates some of the maintenance burden, in that others will fix
   problems caused by API changes.  But the original developer should
   continue to take responsibility for the code if it is to remain useful
   in the longer term.

One of the largest mistakes made by kernel developers (or their employers)
is to try to cut the process down to a single "merging into the mainline"
step.  This approach invariably leads to frustration for everybody
involved.

How patches get into the Kernel
-------------------------------

There is exactly one person who can merge patches into the mainline kernel
repository: Linus Torvalds. But, for example, of the over 9,500 patches
which went into the 2.6.38 kernel, only 112 (around 1.3%) were directly
chosen by Linus himself. The kernel project has long since grown to a size
where no single developer could possibly inspect and select every patch
unassisted. The way the kernel developers have addressed this growth is
through the use of a lieutenant system built around a chain of trust.

The kernel code base is logically broken down into a set of subsystems:
networking, specific architecture support, memory management, video
devices, etc.  Most subsystems have a designated maintainer, a developer
who has overall responsibility for the code within that subsystem.  These
subsystem maintainers are the gatekeepers (in a loose way) for the portion
of the kernel they manage; they are the ones who will (usually) accept a
patch for inclusion into the mainline kernel.

Subsystem maintainers each manage their own version of the kernel source
tree, usually (but certainly not always) using the git source management
tool.  Tools like git (and related tools like quilt or mercurial) allow
maintainers to track a list of patches, including authorship information
and other metadata.  At any given time, the maintainer can identify which
patches in his or her repository are not found in the mainline.

When the merge window opens, top-level maintainers will ask Linus to "pull"
the patches they have selected for merging from their repositories.  If
Linus agrees, the stream of patches will flow up into his repository,
becoming part of the mainline kernel.  The amount of attention that Linus
pays to specific patches received in a pull operation varies.  It is clear
that, sometimes, he looks quite closely.  But, as a general rule, Linus
trusts the subsystem maintainers to not send bad patches upstream.

Subsystem maintainers, in turn, can pull patches from other maintainers.
For example, the networking tree is built from patches which accumulated
first in trees dedicated to network device drivers, wireless networking,
etc.  This chain of repositories can be arbitrarily long, though it rarely
exceeds two or three links.  Since each maintainer in the chain trusts
those managing lower-level trees, this process is known as the "chain of
trust."

Clearly, in a system like this, getting patches into the kernel depends on
finding the right maintainer.  Sending patches directly to Linus is not
normally the right way to go.

Next trees
----------

The chain of subsystem trees guides the flow of patches into the kernel,
but it also raises an interesting question: what if somebody wants to look
at all of the patches which are being prepared for the next merge window?
Developers will be interested in what other changes are pending to see
whether there are any conflicts to worry about; a patch which changes a
core kernel function prototype, for example, will conflict with any other
patches which use the older form of that function.  Reviewers and testers
want access to the changes in their integrated form before all of those
changes land in the mainline kernel.  One could pull changes from all of
the interesting subsystem trees, but that would be a big and error-prone
job.

The answer comes in the form of -next trees, where subsystem trees are
collected for testing and review.  The older of these trees, maintained by
Andrew Morton, is called "-mm" (for memory management, which is how it got
started).  The -mm tree integrates patches from a long list of subsystem
trees; it also has some patches aimed at helping with debugging.

Beyond that, -mm contains a significant collection of patches which have
been selected by Andrew directly.  These patches may have been posted on a
mailing list, or they may apply to a part of the kernel for which there is
no designated subsystem tree.  As a result, -mm operates as a sort of
subsystem tree of last resort; if there is no other obvious path for a
patch into the mainline, it is likely to end up in -mm.  Miscellaneous
patches which accumulate in -mm will eventually either be forwarded on to
an appropriate subsystem tree or be sent directly to Linus.  In a typical
development cycle, approximately 5-10% of the patches going into the
mainline get there via -mm.

The current -mm patch is available in the "mmotm" (-mm of the moment)
directory at:

	https://www.ozlabs.org/~akpm/mmotm/

Use of the MMOTM tree is likely to be a frustrating experience, though;
there is a definite chance that it will not even compile.

The primary tree for next-cycle patch merging is linux-next, maintained by
Stephen Rothwell.  The linux-next tree is, by design, a snapshot of what
the mainline is expected to look like after the next merge window closes.
Linux-next trees are announced on the linux-kernel and linux-next mailing
lists when they are assembled; they can be downloaded from:

	https://www.kernel.org/pub/linux/kernel/next/

Linux-next has become an integral part of the kernel development process;
all patches merged during a given merge window should really have found
their way into linux-next some time before the merge window opens.

Staging trees
-------------

The kernel source tree contains the drivers/staging/ directory, where
many sub-directories for drivers or filesystems that are on their way to
being added to the kernel tree live.  They remain in drivers/staging while
they still need more work; once complete, they can be moved into the
kernel proper.  This is a way to keep track of drivers that aren't
up to Linux kernel coding or quality standards, but people may want to use
them and track development.

Greg Kroah-Hartman currently maintains the staging tree.  Drivers that
still need work are sent to him, with each driver having its own
subdirectory in drivers/staging/.  Along with the driver source files, a
TODO file should be present in the directory as well.  The TODO file lists
the pending work that the driver needs for acceptance into the kernel
proper, as well as a list of people that should be Cc'd for any patches to
the driver.  Current rules require that drivers contributed to staging
must, at a minimum, compile properly.

Staging can be a relatively easy way to get new drivers into the mainline
where, with luck, they will come to the attention of other developers and
improve quickly.  Entry into staging is not the end of the story, though;
code in staging which is not seeing regular progress will eventually be
removed.  Distributors also tend to be relatively reluctant to enable
staging drivers.  So staging is, at best, a stop on the way toward becoming
a proper mainline driver.

Tools
-----

As can be seen from the above text, the kernel development process depends
heavily on the ability to herd collections of patches in various
directions.  The whole thing would not work anywhere near as well as it
does without suitably powerful tools.  Tutorials on how to use these tools
are well beyond the scope of this document, but there is space for a few
pointers.

By far the dominant source code management system used by the kernel
community is git.  Git is one of a number of distributed version control
systems being developed in the free software community.  It is well tuned
for kernel development, in that it performs quite well when dealing with
large repositories and large numbers of patches.  It also has a reputation
for being difficult to learn and use, though it has gotten better over
time.  Some sort of familiarity with git is almost a requirement for kernel
developers; even if they do not use it for their own work, they'll need git
to keep up with what other developers (and the mainline) are doing.

Git is now packaged by almost all Linux distributions.  There is a home
page at:

	https://git-scm.com/

That page has pointers to documentation and tutorials.

Among the kernel developers who do not use git, the most popular choice is
almost certainly Mercurial:

	https://www.selenic.com/mercurial/

Mercurial shares many features with git, but it provides an interface which
many find easier to use.

The other tool worth knowing about is Quilt:

	https://savannah.nongnu.org/projects/quilt/

Quilt is a patch management system, rather than a source code management
system.  It does not track history over time; it is, instead, oriented
toward tracking a specific set of changes against an evolving code base.
Some major subsystem maintainers use quilt to manage patches intended to go
upstream.  For the management of certain kinds of trees (-mm, for example),
quilt is the best tool for the job.

Mailing lists
-------------

A great deal of Linux kernel development work is done by way of mailing
lists.  It is hard to be a fully-functioning member of the community
without joining at least one list somewhere.  But Linux mailing lists also
represent a potential hazard to developers, who risk getting buried under a
load of electronic mail, running afoul of the conventions used on the Linux
lists, or both.

Most kernel mailing lists are run on vger.kernel.org; the master list can
be found at:

	http://vger.kernel.org/vger-lists.html

There are lists hosted elsewhere, though; a number of them are at
redhat.com/mailman/listinfo.

The core mailing list for kernel development is, of course, linux-kernel.
This list is an intimidating place to be; volume can reach 500 messages per
day, the amount of noise is high, the conversation can be severely
technical, and participants are not always concerned with showing a high
degree of politeness.  But there is no other place where the kernel
development community comes together as a whole; developers who avoid this
list will miss important information.

There are a few hints which can help with linux-kernel survival:

- Have the list delivered to a separate folder, rather than your main
  mailbox.  One must be able to ignore the stream for sustained periods of
  time.

- Do not try to follow every conversation - nobody else does.  It is
  important to filter on both the topic of interest (though note that
  long-running conversations can drift away from the original subject
  without changing the email subject line) and the people who are
  participating.

- Do not feed the trolls.  If somebody is trying to stir up an angry
  response, ignore them.

- When responding to linux-kernel email (or that on other lists) preserve
  the Cc: header for all involved.  In the absence of a strong reason (such
  as an explicit request), you should never remove recipients.  Always make
  sure that the person you are responding to is in the Cc: list.  This
  convention also makes it unnecessary to explicitly ask to be copied on
  replies to your postings.

- Search the list archives (and the net as a whole) before asking
  questions.  Some developers can get impatient with people who clearly
  have not done their homework.

- Avoid top-posting (the practice of putting your answer above the quoted
  text you are responding to).  It makes your response harder to read and
  makes a poor impression.

- Ask on the correct mailing list.  Linux-kernel may be the general meeting
  point, but it is not the best place to find developers from all
  subsystems.

The last point - finding the correct mailing list - is a common place for
beginning developers to go wrong.  Somebody who asks a networking-related
question on linux-kernel will almost certainly receive a polite suggestion
to ask on the netdev list instead, as that is the list frequented by most
networking developers.  Other lists exist for the SCSI, video4linux, IDE,
filesystem, etc. subsystems.  The best place to look for mailing lists is
in the MAINTAINERS file packaged with the kernel source.

Getting started with Kernel development
---------------------------------------

Questions about how to get started with the kernel development process are
common - from both individuals and companies.  Equally common are missteps
which make the beginning of the relationship harder than it has to be.

Companies often look to hire well-known developers to get a development
group started.  This can, in fact, be an effective technique.  But it also
tends to be expensive and does not do much to grow the pool of experienced
kernel developers.  It is possible to bring in-house developers up to speed
on Linux kernel development, given the investment of a bit of time.  Taking
this time can endow an employer with a group of developers who understand
the kernel and the company both, and who can help to train others as well.
Over the medium term, this is often the more profitable approach.

Individual developers are often, understandably, at a loss for a place to
start.  Beginning with a large project can be intimidating; one often wants
to test the waters with something smaller first.  This is the point where
some developers jump into the creation of patches fixing spelling errors or
minor coding style issues.  Unfortunately, such patches create a level of
noise which is distracting for the development community as a whole, so,
increasingly, they are looked down upon.  New developers wishing to
introduce themselves to the community will not get the sort of reception
they wish for by these means.

Andrew Morton gives this advice for aspiring kernel developers

::

	The #1 project for all kernel beginners should surely be "make sure
	that the kernel runs perfectly at all times on all machines which
	you can lay your hands on".  Usually the way to do this is to work
	with others on getting things fixed up (this can require
	persistence!) but that's fine - it's a part of kernel development.

(https://lwn.net/Articles/283982/).

In the absence of obvious problems to fix, developers are advised to look
at the current lists of regressions and open bugs in general.  There is
never any shortage of issues in need of fixing; by addressing these issues,
developers will gain experience with the process while, at the same time,
building respect with the rest of the development community.


code:
.. _development_early_stage:

Early-stage planning
====================

When contemplating a Linux kernel development project, it can be tempting
to jump right in and start coding.  As with any significant project,
though, much of the groundwork for success is best laid before the first
line of code is written.  Some time spent in early planning and
communication can save far more time later on.

Specifying the problem
----------------------

Like any engineering project, a successful kernel enhancement starts with a
clear description of the problem to be solved.  In some cases, this step is
easy: when a driver is needed for a specific piece of hardware, for
example.  In others, though, it is tempting to confuse the real problem
with the proposed solution, and that can lead to difficulties.

Consider an example: some years ago, developers working with Linux audio
sought a way to run applications without dropouts or other artifacts caused
by excessive latency in the system.  The solution they arrived at was a
kernel module intended to hook into the Linux Security Module (LSM)
framework; this module could be configured to give specific applications
access to the realtime scheduler.  This module was implemented and sent to
the linux-kernel mailing list, where it immediately ran into problems.

To the audio developers, this security module was sufficient to solve their
immediate problem.  To the wider kernel community, though, it was seen as a
misuse of the LSM framework (which is not intended to confer privileges
onto processes which they would not otherwise have) and a risk to system
stability.  Their preferred solutions involved realtime scheduling access
via the rlimit mechanism for the short term, and ongoing latency reduction
work in the long term.

The audio community, however, could not see past the particular solution
they had implemented; they were unwilling to accept alternatives.  The
resulting disagreement left those developers feeling disillusioned with the
entire kernel development process; one of them went back to an audio list
and posted this:

	There are a number of very good Linux kernel developers, but they
	tend to get outshouted by a large crowd of arrogant fools. Trying
	to communicate user requirements to these people is a waste of
	time. They are much too "intelligent" to listen to lesser mortals.

(https://lwn.net/Articles/131776/).

The reality of the situation was different; the kernel developers were far
more concerned about system stability, long-term maintenance, and finding
the right solution to the problem than they were with a specific module.
The moral of the story is to focus on the problem - not a specific solution
- and to discuss it with the development community before investing in the
creation of a body of code.

So, when contemplating a kernel development project, one should obtain
answers to a short set of questions:

 - What, exactly, is the problem which needs to be solved?

 - Who are the users affected by this problem?  Which use cases should the
   solution address?

 - How does the kernel fall short in addressing that problem now?

Only then does it make sense to start considering possible solutions.

Early discussion
----------------

When planning a kernel development project, it makes great sense to hold
discussions with the community before launching into implementation.  Early
communication can save time and trouble in a number of ways:

 - It may well be that the problem is addressed by the kernel in ways which
   you have not understood.  The Linux kernel is large and has a number of
   features and capabilities which are not immediately obvious.  Not all
   kernel capabilities are documented as well as one might like, and it is
   easy to miss things.  Your author has seen the posting of a complete
   driver which duplicated an existing driver that the new author had been
   unaware of.  Code which reinvents existing wheels is not only wasteful;
   it will also not be accepted into the mainline kernel.

 - There may be elements of the proposed solution which will not be
   acceptable for mainline merging.  It is better to find out about
   problems like this before writing the code.

 - It's entirely possible that other developers have thought about the
   problem; they may have ideas for a better solution, and may be willing
   to help in the creation of that solution.

Years of experience with the kernel development community have taught a
clear lesson: kernel code which is designed and developed behind closed
doors invariably has problems which are only revealed when the code is
released into the community.  Sometimes these problems are severe,
requiring months or years of effort before the code can be brought up to
the kernel community's standards.  Some examples include:

 - The Devicescape network stack was designed and implemented for
   single-processor systems.  It could not be merged into the mainline
   until it was made suitable for multiprocessor systems.  Retrofitting
   locking and such into code is a difficult task; as a result, the merging
   of this code (now called mac80211) was delayed for over a year.

 - The Reiser4 filesystem included a number of capabilities which, in the
   core kernel developers' opinion, should have been implemented in the
   virtual filesystem layer instead.  It also included features which could
   not easily be implemented without exposing the system to user-caused
   deadlocks.  The late revelation of these problems - and refusal to
   address some of them - has caused Reiser4 to stay out of the mainline
   kernel.

 - The AppArmor security module made use of internal virtual filesystem
   data structures in ways which were considered to be unsafe and
   unreliable.  This concern (among others) kept AppArmor out of the
   mainline for years.

In each of these cases, a great deal of pain and extra work could have been
avoided with some early discussion with the kernel developers.

Who do you talk to?
-------------------

When developers decide to take their plans public, the next question will
be: where do we start?  The answer is to find the right mailing list(s) and
the right maintainer.  For mailing lists, the best approach is to look in
the MAINTAINERS file for a relevant place to post.  If there is a suitable
subsystem list, posting there is often preferable to posting on
linux-kernel; you are more likely to reach developers with expertise in the
relevant subsystem and the environment may be more supportive.

Finding maintainers can be a bit harder.  Again, the MAINTAINERS file is
the place to start.  That file tends to not always be up to date, though,
and not all subsystems are represented there.  The person listed in the
MAINTAINERS file may, in fact, not be the person who is actually acting in
that role currently.  So, when there is doubt about who to contact, a
useful trick is to use git (and "git log" in particular) to see who is
currently active within the subsystem of interest.  Look at who is writing
patches, and who, if anybody, is attaching Signed-off-by lines to those
patches.  Those are the people who will be best placed to help with a new
development project.

The task of finding the right maintainer is sometimes challenging enough
that the kernel developers have added a script to ease the process:

::

	.../scripts/get_maintainer.pl

This script will return the current maintainer(s) for a given file or
directory when given the "-f" option.  If passed a patch on the
command line, it will list the maintainers who should probably receive
copies of the patch.  This is the preferred way (unlike "-f" option) to get the
list of people to Cc for your patches.  There are a number of options
regulating how hard get_maintainer.pl will search for maintainers; please be
careful about using the more aggressive options as you may end up including
developers who have no real interest in the code you are modifying.

If all else fails, talking to Andrew Morton can be an effective way to
track down a maintainer for a specific piece of code.

When to post?
-------------

If possible, posting your plans during the early stages can only be
helpful.  Describe the problem being solved and any plans that have been
made on how the implementation will be done.  Any information you can
provide can help the development community provide useful input on the
project.

One discouraging thing which can happen at this stage is not a hostile
reaction, but, instead, little or no reaction at all.  The sad truth of the
matter is (1) kernel developers tend to be busy, (2) there is no shortage
of people with grand plans and little code (or even prospect of code) to
back them up, and (3) nobody is obligated to review or comment on ideas
posted by others.  Beyond that, high-level designs often hide problems
which are only revealed when somebody actually tries to implement those
designs; for that reason, kernel developers would rather see the code.

If a request-for-comments posting yields little in the way of comments, do
not assume that it means there is no interest in the project.
Unfortunately, you also cannot assume that there are no problems with your
idea.  The best thing to do in this situation is to proceed, keeping the
community informed as you go.

Getting official buy-in
-----------------------

If your work is being done in a corporate environment - as most Linux
kernel work is - you must, obviously, have permission from suitably
empowered managers before you can post your company's plans or code to a
public mailing list.  The posting of code which has not been cleared for
release under a GPL-compatible license can be especially problematic; the
sooner that a company's management and legal staff can agree on the posting
of a kernel development project, the better off everybody involved will be.

Some readers may be thinking at this point that their kernel work is
intended to support a product which does not yet have an officially
acknowledged existence.  Revealing their employer's plans on a public
mailing list may not be a viable option.  In cases like this, it is worth
considering whether the secrecy is really necessary; there is often no real
need to keep development plans behind closed doors.

That said, there are also cases where a company legitimately cannot
disclose its plans early in the development process.  Companies with
experienced kernel developers may choose to proceed in an open-loop manner
on the assumption that they will be able to avoid serious integration
problems later.  For companies without that sort of in-house expertise, the
best option is often to hire an outside developer to review the plans under
a non-disclosure agreement.  The Linux Foundation operates an NDA program
designed to help with this sort of situation; more information can be found
at:

    https://www.linuxfoundation.org/nda/

This kind of review is often enough to avoid serious problems later on
without requiring public disclosure of the project.