code:
.. _device_link:

============
Device links
============

By default, the driver core only enforces dependencies between devices
that are borne out of a parent/child relationship within the device
hierarchy: When suspending, resuming or shutting down the system, devices
are ordered based on this relationship, i.e. children are always suspended
before their parent, and the parent is always resumed before its children.

Sometimes there is a need to represent device dependencies beyond the
mere parent/child relationship, e.g. between siblings, and have the
driver core automatically take care of them.

Secondly, the driver core by default does not enforce any driver presence
dependencies, i.e. that one device must be bound to a driver before
another one can probe or function correctly.

Often these two dependency types come together, so a device depends on
another one both with regards to driver presence *and* with regards to
suspend/resume and shutdown ordering.

Device links allow representation of such dependencies in the driver core.

In its standard or *managed* form, a device link combines *both* dependency
types:  It guarantees correct suspend/resume and shutdown ordering between a
"supplier" device and its "consumer" devices, and it guarantees driver
presence on the supplier.  The consumer devices are not probed before the
supplier is bound to a driver, and they're unbound before the supplier
is unbound.

When driver presence on the supplier is irrelevant and only correct
suspend/resume and shutdown ordering is needed, the device link may
simply be set up with the ``DL_FLAG_STATELESS`` flag.  In other words,
enforcing driver presence on the supplier is optional.

Another optional feature is runtime PM integration:  By setting the
``DL_FLAG_PM_RUNTIME`` flag on addition of the device link, the PM core
is instructed to runtime resume the supplier and keep it active
whenever and for as long as the consumer is runtime resumed.

Usage
=====

The earliest point in time when device links can be added is after
:c:func:`device_add()` has been called for the supplier and
:c:func:`device_initialize()` has been called for the consumer.

It is legal to add them later, but care must be taken that the system
remains in a consistent state:  E.g. a device link cannot be added in
the midst of a suspend/resume transition, so either commencement of
such a transition needs to be prevented with :c:func:`lock_system_sleep()`,
or the device link needs to be added from a function which is guaranteed
not to run in parallel to a suspend/resume transition, such as from a
device ``->probe`` callback or a boot-time PCI quirk.

Another example for an inconsistent state would be a device link that
represents a driver presence dependency, yet is added from the consumer's
``->probe`` callback while the supplier hasn't started to probe yet:  Had the
driver core known about the device link earlier, it wouldn't have probed the
consumer in the first place.  The onus is thus on the consumer to check
presence of the supplier after adding the link, and defer probing on
non-presence.  [Note that it is valid to create a link from the consumer's
``->probe`` callback while the supplier is still probing, but the consumer must
know that the supplier is functional already at the link creation time (that is
the case, for instance, if the consumer has just acquired some resources that
would not have been available had the supplier not been functional then).]

If a device link with ``DL_FLAG_STATELESS`` set (i.e. a stateless device link)
is added in the ``->probe`` callback of the supplier or consumer driver, it is
typically deleted in its ``->remove`` callback for symmetry.  That way, if the
driver is compiled as a module, the device link is added on module load and
orderly deleted on unload.  The same restrictions that apply to device link
addition (e.g. exclusion of a parallel suspend/resume transition) apply equally
to deletion.  Device links managed by the driver core are deleted automatically
by it.

Several flags may be specified on device link addition, two of which
have already been mentioned above:  ``DL_FLAG_STATELESS`` to express that no
driver presence dependency is needed (but only correct suspend/resume and
shutdown ordering) and ``DL_FLAG_PM_RUNTIME`` to express that runtime PM
integration is desired.

Two other flags are specifically targeted at use cases where the device
link is added from the consumer's ``->probe`` callback:  ``DL_FLAG_RPM_ACTIVE``
can be specified to runtime resume the supplier and prevent it from suspending
before the consumer is runtime suspended.  ``DL_FLAG_AUTOREMOVE_CONSUMER``
causes the device link to be automatically purged when the consumer fails to
probe or later unbinds.

Similarly, when the device link is added from supplier's ``->probe`` callback,
``DL_FLAG_AUTOREMOVE_SUPPLIER`` causes the device link to be automatically
purged when the supplier fails to probe or later unbinds.

If neither ``DL_FLAG_AUTOREMOVE_CONSUMER`` nor ``DL_FLAG_AUTOREMOVE_SUPPLIER``
is set, ``DL_FLAG_AUTOPROBE_CONSUMER`` can be used to request the driver core
to probe for a driver for the consumer driver on the link automatically after
a driver has been bound to the supplier device.

Note, however, that any combinations of ``DL_FLAG_AUTOREMOVE_CONSUMER``,
``DL_FLAG_AUTOREMOVE_SUPPLIER`` or ``DL_FLAG_AUTOPROBE_CONSUMER`` with
``DL_FLAG_STATELESS`` are invalid and cannot be used.

Limitations
===========

Driver authors should be aware that a driver presence dependency for managed
device links (i.e. when ``DL_FLAG_STATELESS`` is not specified on link addition)
may cause probing of the consumer to be deferred indefinitely.  This can become
a problem if the consumer is required to probe before a certain initcall level
is reached.  Worse, if the supplier driver is blacklisted or missing, the
consumer will never be probed.

Moreover, managed device links cannot be deleted directly.  They are deleted
by the driver core when they are not necessary any more in accordance with the
``DL_FLAG_AUTOREMOVE_CONSUMER`` and ``DL_FLAG_AUTOREMOVE_SUPPLIER`` flags.
However, stateless device links (i.e. device links with ``DL_FLAG_STATELESS``
set) are expected to be removed by whoever called :c:func:`device_link_add()`
to add them with the help of either :c:func:`device_link_del()` or
:c:func:`device_link_remove()`.

Passing ``DL_FLAG_RPM_ACTIVE`` along with ``DL_FLAG_STATELESS`` to
:c:func:`device_link_add()` may cause the PM-runtime usage counter of the
supplier device to remain nonzero after a subsequent invocation of either
:c:func:`device_link_del()` or :c:func:`device_link_remove()` to remove the
device link returned by it.  This happens if :c:func:`device_link_add()` is
called twice in a row for the same consumer-supplier pair without removing the
link between these calls, in which case allowing the PM-runtime usage counter
of the supplier to drop on an attempt to remove the link may cause it to be
suspended while the consumer is still PM-runtime-active and that has to be
avoided.  [To work around this limitation it is sufficient to let the consumer
runtime suspend at least once, or call :c:func:`pm_runtime_set_suspended()` for
it with PM-runtime disabled, between the :c:func:`device_link_add()` and
:c:func:`device_link_del()` or :c:func:`device_link_remove()` calls.]

Sometimes drivers depend on optional resources.  They are able to operate
in a degraded mode (reduced feature set or performance) when those resources
are not present.  An example is an SPI controller that can use a DMA engine
or work in PIO mode.  The controller can determine presence of the optional
resources at probe time but on non-presence there is no way to know whether
they will become available in the near future (due to a supplier driver
probing) or never.  Consequently it cannot be determined whether to defer
probing or not.  It would be possible to notify drivers when optional
resources become available after probing, but it would come at a high cost
for drivers as switching between modes of operation at runtime based on the
availability of such resources would be much more complex than a mechanism
based on probe deferral.  In any case optional resources are beyond the
scope of device links.

Examples
========

* An MMU device exists alongside a busmaster device, both are in the same
  power domain.  The MMU implements DMA address translation for the busmaster
  device and shall be runtime resumed and kept active whenever and as long
  as the busmaster device is active.  The busmaster device's driver shall
  not bind before the MMU is bound.  To achieve this, a device link with
  runtime PM integration is added from the busmaster device (consumer)
  to the MMU device (supplier).  The effect with regards to runtime PM
  is the same as if the MMU was the parent of the master device.

  The fact that both devices share the same power domain would normally
  suggest usage of a struct dev_pm_domain or struct generic_pm_domain,
  however these are not independent devices that happen to share a power
  switch, but rather the MMU device serves the busmaster device and is
  useless without it.  A device link creates a synthetic hierarchical
  relationship between the devices and is thus more apt.

* A Thunderbolt host controller comprises a number of PCIe hotplug ports
  and an NHI device to manage the PCIe switch.  On resume from system sleep,
  the NHI device needs to re-establish PCI tunnels to attached devices
  before the hotplug ports can resume.  If the hotplug ports were children
  of the NHI, this resume order would automatically be enforced by the
  PM core, but unfortunately they're aunts.  The solution is to add
  device links from the hotplug ports (consumers) to the NHI device
  (supplier).  A driver presence dependency is not necessary for this
  use case.

* Discrete GPUs in hybrid graphics laptops often feature an HDA controller
  for HDMI/DP audio.  In the device hierarchy the HDA controller is a sibling
  of the VGA device, yet both share the same power domain and the HDA
  controller is only ever needed when an HDMI/DP display is attached to the
  VGA device.  A device link from the HDA controller (consumer) to the
  VGA device (supplier) aptly represents this relationship.

* ACPI allows definition of a device start order by way of _DEP objects.
  A classical example is when ACPI power management methods on one device
  are implemented in terms of I\ :sup:`2`\ C accesses and require a specific
  I\ :sup:`2`\ C controller to be present and functional for the power
  management of the device in question to work.

* In some SoCs a functional dependency exists from display, video codec and
  video processing IP cores on transparent memory access IP cores that handle
  burst access and compression/decompression.

Alternatives
============

* A struct dev_pm_domain can be used to override the bus,
  class or device type callbacks.  It is intended for devices sharing
  a single on/off switch, however it does not guarantee a specific
  suspend/resume ordering, this needs to be implemented separately.
  It also does not by itself track the runtime PM status of the involved
  devices and turn off the power switch only when all of them are runtime
  suspended.  Furthermore it cannot be used to enforce a specific shutdown
  ordering or a driver presence dependency.

* A struct generic_pm_domain is a lot more heavyweight than a
  device link and does not allow for shutdown ordering or driver presence
  dependencies.  It also cannot be used on ACPI systems.

Implementation
==============

The device hierarchy, which -- as the name implies -- is a tree,
becomes a directed acyclic graph once device links are added.

Ordering of these devices during suspend/resume is determined by the
dpm_list.  During shutdown it is determined by the devices_kset.  With
no device links present, the two lists are a flattened, one-dimensional
representations of the device tree such that a device is placed behind
all its ancestors.  That is achieved by traversing the ACPI namespace
or OpenFirmware device tree top-down and appending devices to the lists
as they are discovered.

Once device links are added, the lists need to satisfy the additional
constraint that a device is placed behind all its suppliers, recursively.
To ensure this, upon addition of the device link the consumer and the
entire sub-graph below it (all children and consumers of the consumer)
are moved to the end of the list.  (Call to :c:func:`device_reorder_to_tail()`
from :c:func:`device_link_add()`.)

To prevent introduction of dependency loops into the graph, it is
verified upon device link addition that the supplier is not dependent
on the consumer or any children or consumers of the consumer.
(Call to :c:func:`device_is_dependent()` from :c:func:`device_link_add()`.)
If that constraint is violated, :c:func:`device_link_add()` will return
``NULL`` and a ``WARNING`` will be logged.

Notably this also prevents the addition of a device link from a parent
device to a child.  However the converse is allowed, i.e. a device link
from a child to a parent.  Since the driver core already guarantees
correct suspend/resume and shutdown ordering between parent and child,
such a device link only makes sense if a driver presence dependency is
needed on top of that.  In this case driver authors should weigh
carefully if a device link is at all the right tool for the purpose.
A more suitable approach might be to simply use deferred probing or
add a device flag causing the parent driver to be probed before the
child one.

State machine
=============

.. kernel-doc:: include/linux/device.h
   :functions: device_link_state

::

                 .=============================.
                 |                             |
                 v                             |
 DORMANT <=> AVAILABLE <=> CONSUMER_PROBE => ACTIVE
    ^                                          |
    |                                          |
    '============ SUPPLIER_UNBIND <============'

* The initial state of a device link is automatically determined by
  :c:func:`device_link_add()` based on the driver presence on the supplier
  and consumer.  If the link is created before any devices are probed, it
  is set to ``DL_STATE_DORMANT``.

* When a supplier device is bound to a driver, links to its consumers
  progress to ``DL_STATE_AVAILABLE``.
  (Call to :c:func:`device_links_driver_bound()` from
  :c:func:`driver_bound()`.)

* Before a consumer device is probed, presence of supplier drivers is
  verified by checking the consumer device is not in the wait_for_suppliers
  list and by checking that links to suppliers are in ``DL_STATE_AVAILABLE``
  state.  The state of the links is updated to ``DL_STATE_CONSUMER_PROBE``.
  (Call to :c:func:`device_links_check_suppliers()` from
  :c:func:`really_probe()`.)
  This prevents the supplier from unbinding.
  (Call to :c:func:`wait_for_device_probe()` from
  :c:func:`device_links_unbind_consumers()`.)

* If the probe fails, links to suppliers revert back to ``DL_STATE_AVAILABLE``.
  (Call to :c:func:`device_links_no_driver()` from :c:func:`really_probe()`.)

* If the probe succeeds, links to suppliers progress to ``DL_STATE_ACTIVE``.
  (Call to :c:func:`device_links_driver_bound()` from :c:func:`driver_bound()`.)

* When the consumer's driver is later on removed, links to suppliers revert
  back to ``DL_STATE_AVAILABLE``.
  (Call to :c:func:`__device_links_no_driver()` from
  :c:func:`device_links_driver_cleanup()`, which in turn is called from
  :c:func:`__device_release_driver()`.)

* Before a supplier's driver is removed, links to consumers that are not
  bound to a driver are updated to ``DL_STATE_SUPPLIER_UNBIND``.
  (Call to :c:func:`device_links_busy()` from
  :c:func:`__device_release_driver()`.)
  This prevents the consumers from binding.
  (Call to :c:func:`device_links_check_suppliers()` from
  :c:func:`really_probe()`.)
  Consumers that are bound are freed from their driver; consumers that are
  probing are waited for until they are done.
  (Call to :c:func:`device_links_unbind_consumers()` from
  :c:func:`__device_release_driver()`.)
  Once all links to consumers are in ``DL_STATE_SUPPLIER_UNBIND`` state,
  the supplier driver is released and the links revert to ``DL_STATE_DORMANT``.
  (Call to :c:func:`device_links_driver_cleanup()` from
  :c:func:`__device_release_driver()`.)

API
===

See device_link_add(), device_link_del() and device_link_remove().


code:
Buffer Sharing and Synchronization
==================================

The dma-buf subsystem provides the framework for sharing buffers for
hardware (DMA) access across multiple device drivers and subsystems, and
for synchronizing asynchronous hardware access.

This is used, for example, by drm "prime" multi-GPU support, but is of
course not limited to GPU use cases.

The three main components of this are: (1) dma-buf, representing a
sg_table and exposed to userspace as a file descriptor to allow passing
between devices, (2) fence, which provides a mechanism to signal when
one device has finished access, and (3) reservation, which manages the
shared or exclusive fence(s) associated with the buffer.

Shared DMA Buffers
------------------

This document serves as a guide to device-driver writers on what is the dma-buf
buffer sharing API, how to use it for exporting and using shared buffers.

Any device driver which wishes to be a part of DMA buffer sharing, can do so as
either the 'exporter' of buffers, or the 'user' or 'importer' of buffers.

Say a driver A wants to use buffers created by driver B, then we call B as the
exporter, and A as buffer-user/importer.

The exporter

 - implements and manages operations in :c:type:`struct dma_buf_ops
   <dma_buf_ops>` for the buffer,
 - allows other users to share the buffer by using dma_buf sharing APIs,
 - manages the details of buffer allocation, wrapped in a :c:type:`struct
   dma_buf <dma_buf>`,
 - decides about the actual backing storage where this allocation happens,
 - and takes care of any migration of scatterlist - for all (shared) users of
   this buffer.

The buffer-user

 - is one of (many) sharing users of the buffer.
 - doesn't need to worry about how the buffer is allocated, or where.
 - and needs a mechanism to get access to the scatterlist that makes up this
   buffer in memory, mapped into its own address space, so it can access the
   same area of memory. This interface is provided by :c:type:`struct
   dma_buf_attachment <dma_buf_attachment>`.

Any exporters or users of the dma-buf buffer sharing framework must have a
'select DMA_SHARED_BUFFER' in their respective Kconfigs.

Userspace Interface Notes
~~~~~~~~~~~~~~~~~~~~~~~~~

Mostly a DMA buffer file descriptor is simply an opaque object for userspace,
and hence the generic interface exposed is very minimal. There's a few things to
consider though:

- Since kernel 3.12 the dma-buf FD supports the llseek system call, but only
  with offset=0 and whence=SEEK_END|SEEK_SET. SEEK_SET is supported to allow
  the usual size discover pattern size = SEEK_END(0); SEEK_SET(0). Every other
  llseek operation will report -EINVAL.

  If llseek on dma-buf FDs isn't support the kernel will report -ESPIPE for all
  cases. Userspace can use this to detect support for discovering the dma-buf
  size using llseek.

- In order to avoid fd leaks on exec, the FD_CLOEXEC flag must be set
  on the file descriptor.  This is not just a resource leak, but a
  potential security hole.  It could give the newly exec'd application
  access to buffers, via the leaked fd, to which it should otherwise
  not be permitted access.

  The problem with doing this via a separate fcntl() call, versus doing it
  atomically when the fd is created, is that this is inherently racy in a
  multi-threaded app[3].  The issue is made worse when it is library code
  opening/creating the file descriptor, as the application may not even be
  aware of the fd's.

  To avoid this problem, userspace must have a way to request O_CLOEXEC
  flag be set when the dma-buf fd is created.  So any API provided by
  the exporting driver to create a dmabuf fd must provide a way to let
  userspace control setting of O_CLOEXEC flag passed in to dma_buf_fd().

- Memory mapping the contents of the DMA buffer is also supported. See the
  discussion below on `CPU Access to DMA Buffer Objects`_ for the full details.

- The DMA buffer FD is also pollable, see `Implicit Fence Poll Support`_ below for
  details.

- The DMA buffer FD also supports a few dma-buf-specific ioctls, see
  `DMA Buffer ioctls`_ below for details.

Basic Operation and Device DMA Access
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. kernel-doc:: drivers/dma-buf/dma-buf.c
   :doc: dma buf device access

CPU Access to DMA Buffer Objects
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. kernel-doc:: drivers/dma-buf/dma-buf.c
   :doc: cpu access

Implicit Fence Poll Support
~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. kernel-doc:: drivers/dma-buf/dma-buf.c
   :doc: implicit fence polling

DMA-BUF statistics
~~~~~~~~~~~~~~~~~~
.. kernel-doc:: drivers/dma-buf/dma-buf-sysfs-stats.c
   :doc: overview

DMA Buffer ioctls
~~~~~~~~~~~~~~~~~

.. kernel-doc:: include/uapi/linux/dma-buf.h

DMA-BUF locking convention
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. kernel-doc:: drivers/dma-buf/dma-buf.c
   :doc: locking convention

Kernel Functions and Structures Reference
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. kernel-doc:: drivers/dma-buf/dma-buf.c
   :export:

.. kernel-doc:: include/linux/dma-buf.h
   :internal:

Reservation Objects
-------------------

.. kernel-doc:: drivers/dma-buf/dma-resv.c
   :doc: Reservation Object Overview

.. kernel-doc:: drivers/dma-buf/dma-resv.c
   :export:

.. kernel-doc:: include/linux/dma-resv.h
   :internal:

DMA Fences
----------

.. kernel-doc:: drivers/dma-buf/dma-fence.c
   :doc: DMA fences overview

DMA Fence Cross-Driver Contract
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. kernel-doc:: drivers/dma-buf/dma-fence.c
   :doc: fence cross-driver contract

DMA Fence Signalling Annotations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. kernel-doc:: drivers/dma-buf/dma-fence.c
   :doc: fence signalling annotation

DMA Fences Functions Reference
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. kernel-doc:: drivers/dma-buf/dma-fence.c
   :export:

.. kernel-doc:: include/linux/dma-fence.h
   :internal:

DMA Fence Array
~~~~~~~~~~~~~~~

.. kernel-doc:: drivers/dma-buf/dma-fence-array.c
   :export:

.. kernel-doc:: include/linux/dma-fence-array.h
   :internal:

DMA Fence Chain
~~~~~~~~~~~~~~~

.. kernel-doc:: drivers/dma-buf/dma-fence-chain.c
   :export:

.. kernel-doc:: include/linux/dma-fence-chain.h
   :internal:

DMA Fence unwrap
~~~~~~~~~~~~~~~~

.. kernel-doc:: include/linux/dma-fence-unwrap.h
   :internal:

DMA Fence uABI/Sync File
~~~~~~~~~~~~~~~~~~~~~~~~

.. kernel-doc:: drivers/dma-buf/sync_file.c
   :export:

.. kernel-doc:: include/linux/sync_file.h
   :internal:

Indefinite DMA Fences
~~~~~~~~~~~~~~~~~~~~~

At various times struct dma_fence with an indefinite time until dma_fence_wait()
finishes have been proposed. Examples include:

* Future fences, used in HWC1 to signal when a buffer isn't used by the display
  any longer, and created with the screen update that makes the buffer visible.
  The time this fence completes is entirely under userspace's control.

* Proxy fences, proposed to handle &drm_syncobj for which the fence has not yet
  been set. Used to asynchronously delay command submission.

* Userspace fences or gpu futexes, fine-grained locking within a command buffer
  that userspace uses for synchronization across engines or with the CPU, which
  are then imported as a DMA fence for integration into existing winsys
  protocols.

* Long-running compute command buffers, while still using traditional end of
  batch DMA fences for memory management instead of context preemption DMA
  fences which get reattached when the compute job is rescheduled.

Common to all these schemes is that userspace controls the dependencies of these
fences and controls when they fire. Mixing indefinite fences with normal
in-kernel DMA fences does not work, even when a fallback timeout is included to
protect against malicious userspace:

* Only the kernel knows about all DMA fence dependencies, userspace is not aware
  of dependencies injected due to memory management or scheduler decisions.

* Only userspace knows about all dependencies in indefinite fences and when
  exactly they will complete, the kernel has no visibility.

Furthermore the kernel has to be able to hold up userspace command submission
for memory management needs, which means we must support indefinite fences being
dependent upon DMA fences. If the kernel also support indefinite fences in the
kernel like a DMA fence, like any of the above proposal would, there is the
potential for deadlocks.

.. kernel-render:: DOT
   :alt: Indefinite Fencing Dependency Cycle
   :caption: Indefinite Fencing Dependency Cycle

   digraph "Fencing Cycle" {
      node [shape=box bgcolor=grey style=filled]
      kernel [label="Kernel DMA Fences"]
      userspace [label="userspace controlled fences"]
      kernel -> userspace [label="memory management"]
      userspace -> kernel [label="Future fence, fence proxy, ..."]

      { rank=same; kernel userspace }
   }

This means that the kernel might accidentally create deadlocks
through memory management dependencies which userspace is unaware of, which
randomly hangs workloads until the timeout kicks in. Workloads, which from
userspace's perspective, do not contain a deadlock.  In such a mixed fencing
architecture there is no single entity with knowledge of all dependencies.
Thefore preventing such deadlocks from within the kernel is not possible.

The only solution to avoid dependencies loops is by not allowing indefinite
fences in the kernel. This means:

* No future fences, proxy fences or userspace fences imported as DMA fences,
  with or without a timeout.

* No DMA fences that signal end of batchbuffer for command submission where
  userspace is allowed to use userspace fencing or long running compute
  workloads. This also means no implicit fencing for shared buffers in these
  cases.

Recoverable Hardware Page Faults Implications
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Modern hardware supports recoverable page faults, which has a lot of
implications for DMA fences.

First, a pending page fault obviously holds up the work that's running on the
accelerator and a memory allocation is usually required to resolve the fault.
But memory allocations are not allowed to gate completion of DMA fences, which
means any workload using recoverable page faults cannot use DMA fences for
synchronization. Synchronization fences controlled by userspace must be used
instead.

On GPUs this poses a problem, because current desktop compositor protocols on
Linux rely on DMA fences, which means without an entirely new userspace stack
built on top of userspace fences, they cannot benefit from recoverable page
faults. Specifically this means implicit synchronization will not be possible.
The exception is when page faults are only used as migration hints and never to
on-demand fill a memory request. For now this means recoverable page
faults on GPUs are limited to pure compute workloads.

Furthermore GPUs usually have shared resources between the 3D rendering and
compute side, like compute units or command submission engines. If both a 3D
job with a DMA fence and a compute workload using recoverable page faults are
pending they could deadlock:

- The 3D workload might need to wait for the compute job to finish and release
  hardware resources first.

- The compute workload might be stuck in a page fault, because the memory
  allocation is waiting for the DMA fence of the 3D workload to complete.

There are a few options to prevent this problem, one of which drivers need to
ensure:

- Compute workloads can always be preempted, even when a page fault is pending
  and not yet repaired. Not all hardware supports this.

- DMA fence workloads and workloads which need page fault handling have
  independent hardware resources to guarantee forward progress. This could be
  achieved through e.g. through dedicated engines and minimal compute unit
  reservations for DMA fence workloads.

- The reservation approach could be further refined by only reserving the
  hardware resources for DMA fence workloads when they are in-flight. This must
  cover the time from when the DMA fence is visible to other threads up to
  moment when fence is completed through dma_fence_signal().

- As a last resort, if the hardware provides no useful reservation mechanics,
  all workloads must be flushed from the GPU when switching between jobs
  requiring DMA fences or jobs requiring page fault handling: This means all DMA
  fences must complete before a compute job with page fault handling can be
  inserted into the scheduler queue. And vice versa, before a DMA fence can be
  made visible anywhere in the system, all compute workloads must be preempted
  to guarantee all pending GPU page faults are flushed.

- Only a fairly theoretical option would be to untangle these dependencies when
  allocating memory to repair hardware page faults, either through separate
  memory blocks or runtime tracking of the full dependency graph of all DMA
  fences. This results very wide impact on the kernel, since resolving the page
  on the CPU side can itself involve a page fault. It is much more feasible and
  robust to limit the impact of handling hardware page faults to the specific
  driver.

Note that workloads that run on independent hardware like copy engines or other
GPUs do not have any impact. This allows us to keep using DMA fences internally
in the kernel even for resolving hardware page faults, e.g. by using copy
engines to clear or copy memory needed to resolve the page fault.

In some ways this page fault problem is a special case of the `Infinite DMA
Fences` discussions: Infinite fences from compute workloads are allowed to
depend on DMA fences, but not the other way around. And not even the page fault
problem is new, because some other CPU thread in userspace might
hit a page fault which holds up a userspace fence - supporting page faults on
GPUs doesn't anything fundamentally new.

repo_name: angery/linux


code:
====================
DMA Engine API Guide
====================

Vinod Koul <vinod dot koul at intel.com>

.. note:: For DMA Engine usage in async_tx please see:
          ``Documentation/crypto/async-tx-api.rst``

Below is a guide to device driver writers on how to use the Slave-DMA API of the
DMA Engine. This is applicable only for slave DMA usage only.

DMA usage
=========

The slave DMA usage consists of following steps:

- Allocate a DMA slave channel

- Set slave and controller specific parameters

- Get a descriptor for transaction

- Submit the transaction

- Issue pending requests and wait for callback notification

The details of these operations are:

1. Allocate a DMA slave channel

   Channel allocation is slightly different in the slave DMA context,
   client drivers typically need a channel from a particular DMA
   controller only and even in some cases a specific channel is desired.
   To request a channel dma_request_chan() API is used.

   Interface:

   .. code-block:: c

      struct dma_chan *dma_request_chan(struct device *dev, const char *name);

   Which will find and return the ``name`` DMA channel associated with the 'dev'
   device. The association is done via DT, ACPI or board file based
   dma_slave_map matching table.

   A channel allocated via this interface is exclusive to the caller,
   until dma_release_channel() is called.

2. Set slave and controller specific parameters

   Next step is always to pass some specific information to the DMA
   driver. Most of the generic information which a slave DMA can use
   is in struct dma_slave_config. This allows the clients to specify
   DMA direction, DMA addresses, bus widths, DMA burst lengths etc
   for the peripheral.

   If some DMA controllers have more parameters to be sent then they
   should try to embed struct dma_slave_config in their controller
   specific structure. That gives flexibility to client to pass more
   parameters, if required.

   Interface:

   .. code-block:: c

      int dmaengine_slave_config(struct dma_chan *chan,
			struct dma_slave_config *config)

   Please see the dma_slave_config structure definition in dmaengine.h
   for a detailed explanation of the struct members. Please note
   that the 'direction' member will be going away as it duplicates the
   direction given in the prepare call.

3. Get a descriptor for transaction

  For slave usage the various modes of slave transfers supported by the
  DMA-engine are:

  - slave_sg: DMA a list of scatter gather buffers from/to a peripheral

  - dma_cyclic: Perform a cyclic DMA operation from/to a peripheral till the
    operation is explicitly stopped.

  - interleaved_dma: This is common to Slave as well as M2M clients. For slave
    address of devices' fifo could be already known to the driver.
    Various types of operations could be expressed by setting
    appropriate values to the 'dma_interleaved_template' members. Cyclic
    interleaved DMA transfers are also possible if supported by the channel by
    setting the DMA_PREP_REPEAT transfer flag.

  A non-NULL return of this transfer API represents a "descriptor" for
  the given transaction.

  Interface:

  .. code-block:: c

     struct dma_async_tx_descriptor *dmaengine_prep_slave_sg(
		struct dma_chan *chan, struct scatterlist *sgl,
		unsigned int sg_len, enum dma_data_direction direction,
		unsigned long flags);

     struct dma_async_tx_descriptor *dmaengine_prep_dma_cyclic(
		struct dma_chan *chan, dma_addr_t buf_addr, size_t buf_len,
		size_t period_len, enum dma_data_direction direction);

     struct dma_async_tx_descriptor *dmaengine_prep_interleaved_dma(
		struct dma_chan *chan, struct dma_interleaved_template *xt,
		unsigned long flags);

  The peripheral driver is expected to have mapped the scatterlist for
  the DMA operation prior to calling dmaengine_prep_slave_sg(), and must
  keep the scatterlist mapped until the DMA operation has completed.
  The scatterlist must be mapped using the DMA struct device.
  If a mapping needs to be synchronized later, dma_sync_*_for_*() must be
  called using the DMA struct device, too.
  So, normal setup should look like this:

  .. code-block:: c

     struct device *dma_dev = dmaengine_get_dma_device(chan);

     nr_sg = dma_map_sg(dma_dev, sgl, sg_len);
	if (nr_sg == 0)
		/* error */

	desc = dmaengine_prep_slave_sg(chan, sgl, nr_sg, direction, flags);

  Once a descriptor has been obtained, the callback information can be
  added and the descriptor must then be submitted. Some DMA engine
  drivers may hold a spinlock between a successful preparation and
  submission so it is important that these two operations are closely
  paired.

  .. note::

     Although the async_tx API specifies that completion callback
     routines cannot submit any new operations, this is not the
     case for slave/cyclic DMA.

     For slave DMA, the subsequent transaction may not be available
     for submission prior to callback function being invoked, so
     slave DMA callbacks are permitted to prepare and submit a new
     transaction.

     For cyclic DMA, a callback function may wish to terminate the
     DMA via dmaengine_terminate_async().

     Therefore, it is important that DMA engine drivers drop any
     locks before calling the callback function which may cause a
     deadlock.

     Note that callbacks will always be invoked from the DMA
     engines tasklet, never from interrupt context.

  **Optional: per descriptor metadata**

  DMAengine provides two ways for metadata support.

  DESC_METADATA_CLIENT

    The metadata buffer is allocated/provided by the client driver and it is
    attached to the descriptor.

  .. code-block:: c

     int dmaengine_desc_attach_metadata(struct dma_async_tx_descriptor *desc,
				   void *data, size_t len);

  DESC_METADATA_ENGINE

    The metadata buffer is allocated/managed by the DMA driver. The client
    driver can ask for the pointer, maximum size and the currently used size of
    the metadata and can directly update or read it.

    Becasue the DMA driver manages the memory area containing the metadata,
    clients must make sure that they do not try to access or get the pointer
    after their transfer completion callback has run for the descriptor.
    If no completion callback has been defined for the transfer, then the
    metadata must not be accessed after issue_pending.
    In other words: if the aim is to read back metadata after the transfer is
    completed, then the client must use completion callback.

  .. code-block:: c

     void *dmaengine_desc_get_metadata_ptr(struct dma_async_tx_descriptor *desc,
		size_t *payload_len, size_t *max_len);

     int dmaengine_desc_set_metadata_len(struct dma_async_tx_descriptor *desc,
		size_t payload_len);

  Client drivers can query if a given mode is supported with:

  .. code-block:: c

     bool dmaengine_is_metadata_mode_supported(struct dma_chan *chan,
		enum dma_desc_metadata_mode mode);

  Depending on the used mode client drivers must follow different flow.

  DESC_METADATA_CLIENT

    - DMA_MEM_TO_DEV / DEV_MEM_TO_MEM:

      1. prepare the descriptor (dmaengine_prep_*)
         construct the metadata in the client's buffer
      2. use dmaengine_desc_attach_metadata() to attach the buffer to the
         descriptor
      3. submit the transfer

    - DMA_DEV_TO_MEM:

      1. prepare the descriptor (dmaengine_prep_*)
      2. use dmaengine_desc_attach_metadata() to attach the buffer to the
         descriptor
      3. submit the transfer
      4. when the transfer is completed, the metadata should be available in the
         attached buffer

  DESC_METADATA_ENGINE

    - DMA_MEM_TO_DEV / DEV_MEM_TO_MEM:

      1. prepare the descriptor (dmaengine_prep_*)
      2. use dmaengine_desc_get_metadata_ptr() to get the pointer to the
         engine's metadata area
      3. update the metadata at the pointer
      4. use dmaengine_desc_set_metadata_len()  to tell the DMA engine the
         amount of data the client has placed into the metadata buffer
      5. submit the transfer

    - DMA_DEV_TO_MEM:

      1. prepare the descriptor (dmaengine_prep_*)
      2. submit the transfer
      3. on transfer completion, use dmaengine_desc_get_metadata_ptr() to get
         the pointer to the engine's metadata area
      4. read out the metadata from the pointer

  .. note::

     When DESC_METADATA_ENGINE mode is used the metadata area for the descriptor
     is no longer valid after the transfer has been completed (valid up to the
     point when the completion callback returns if used).

     Mixed use of DESC_METADATA_CLIENT / DESC_METADATA_ENGINE is not allowed,
     client drivers must use either of the modes per descriptor.

4. Submit the transaction

   Once the descriptor has been prepared and the callback information
   added, it must be placed on the DMA engine drivers pending queue.

   Interface:

   .. code-block:: c

      dma_cookie_t dmaengine_submit(struct dma_async_tx_descriptor *desc)

   This returns a cookie can be used to check the progress of DMA engine
   activity via other DMA engine calls not covered in this document.

   dmaengine_submit() will not start the DMA operation, it merely adds
   it to the pending queue. For this, see step 5, dma_async_issue_pending.

   .. note::

      After calling ``dmaengine_submit()`` the submitted transfer descriptor
      (``struct dma_async_tx_descriptor``) belongs to the DMA engine.
      Consequently, the client must consider invalid the pointer to that
      descriptor.

5. Issue pending DMA requests and wait for callback notification

   The transactions in the pending queue can be activated by calling the
   issue_pending API. If channel is idle then the first transaction in
   queue is started and subsequent ones queued up.

   On completion of each DMA operation, the next in queue is started and
   a tasklet triggered. The tasklet will then call the client driver
   completion callback routine for notification, if set.

   Interface:

   .. code-block:: c

      void dma_async_issue_pending(struct dma_chan *chan);

Further APIs
------------

1. Terminate APIs

   .. code-block:: c

      int dmaengine_terminate_sync(struct dma_chan *chan)
      int dmaengine_terminate_async(struct dma_chan *chan)
      int dmaengine_terminate_all(struct dma_chan *chan) /* DEPRECATED */

   This causes all activity for the DMA channel to be stopped, and may
   discard data in the DMA FIFO which hasn't been fully transferred.
   No callback functions will be called for any incomplete transfers.

   Two variants of this function are available.

   dmaengine_terminate_async() might not wait until the DMA has been fully
   stopped or until any running complete callbacks have finished. But it is
   possible to call dmaengine_terminate_async() from atomic context or from
   within a complete callback. dmaengine_synchronize() must be called before it
   is safe to free the memory accessed by the DMA transfer or free resources
   accessed from within the complete callback.

   dmaengine_terminate_sync() will wait for the transfer and any running
   complete callbacks to finish before it returns. But the function must not be
   called from atomic context or from within a complete callback.

   dmaengine_terminate_all() is deprecated and should not be used in new code.

2. Pause API

   .. code-block:: c

      int dmaengine_pause(struct dma_chan *chan)

   This pauses activity on the DMA channel without data loss.

3. Resume API

   .. code-block:: c

       int dmaengine_resume(struct dma_chan *chan)

   Resume a previously paused DMA channel. It is invalid to resume a
   channel which is not currently paused.

4. Check Txn complete

   .. code-block:: c

      enum dma_status dma_async_is_tx_complete(struct dma_chan *chan,
		dma_cookie_t cookie, dma_cookie_t *last, dma_cookie_t *used)

   This can be used to check the status of the channel. Please see
   the documentation in include/linux/dmaengine.h for a more complete
   description of this API.

   This can be used in conjunction with dma_async_is_complete() and
   the cookie returned from dmaengine_submit() to check for
   completion of a specific DMA transaction.

   .. note::

      Not all DMA engine drivers can return reliable information for
      a running DMA channel. It is recommended that DMA engine users
      pause or stop (via dmaengine_terminate_all()) the channel before
      using this API.

5. Synchronize termination API

   .. code-block:: c

      void dmaengine_synchronize(struct dma_chan *chan)

   Synchronize the termination of the DMA channel to the current context.

   This function should be used after dmaengine_terminate_async() to synchronize
   the termination of the DMA channel to the current context. The function will
   wait for the transfer and any running complete callbacks to finish before it
   returns.

   If dmaengine_terminate_async() is used to stop the DMA channel this function
   must be called before it is safe to free memory accessed by previously
   submitted descriptors or to free any resources accessed within the complete
   callback of previously submitted descriptors.

   The behavior of this function is undefined if dma_async_issue_pending() has
   been called between dmaengine_terminate_async() and this function.


code:
==============
DMA Test Guide
==============

Andy Shevchenko <andriy.shevchenko@linux.intel.com>

This small document introduces how to test DMA drivers using dmatest module.

The dmatest module tests DMA memcpy, memset, XOR and RAID6 P+Q operations using
various lengths and various offsets into the source and destination buffers. It
will initialize both buffers with a repeatable pattern and verify that the DMA
engine copies the requested region and nothing more. It will also verify that
the bytes aren't swapped around, and that the source buffer isn't modified.

The dmatest module can be configured to test a specific channel. It can also
test multiple channels at the same time, and it can start multiple threads
competing for the same channel.

.. note::
  The test suite works only on the channels that have at least one
  capability of the following: DMA_MEMCPY (memory-to-memory), DMA_MEMSET
  (const-to-memory or memory-to-memory, when emulated), DMA_XOR, DMA_PQ.

.. note::
  In case of any related questions use the official mailing list
  dmaengine@vger.kernel.org.

Part 1 - How to build the test module
=====================================

The menuconfig contains an option that could be found by following path:

	Device Drivers -> DMA Engine support -> DMA Test client

In the configuration file the option called CONFIG_DMATEST. The dmatest could
be built as module or inside kernel. Let's consider those cases.

Part 2 - When dmatest is built as a module
==========================================

Example of usage::

    % modprobe dmatest timeout=2000 iterations=1 channel=dma0chan0 run=1

...or::

    % modprobe dmatest
    % echo 2000 > /sys/module/dmatest/parameters/timeout
    % echo 1 > /sys/module/dmatest/parameters/iterations
    % echo dma0chan0 > /sys/module/dmatest/parameters/channel
    % echo 1 > /sys/module/dmatest/parameters/run

...or on the kernel command line::

    dmatest.timeout=2000 dmatest.iterations=1 dmatest.channel=dma0chan0 dmatest.run=1

Example of multi-channel test usage (new in the 5.0 kernel)::

    % modprobe dmatest
    % echo 2000 > /sys/module/dmatest/parameters/timeout
    % echo 1 > /sys/module/dmatest/parameters/iterations
    % echo dma0chan0 > /sys/module/dmatest/parameters/channel
    % echo dma0chan1 > /sys/module/dmatest/parameters/channel
    % echo dma0chan2 > /sys/module/dmatest/parameters/channel
    % echo 1 > /sys/module/dmatest/parameters/run

.. note::
  For all tests, starting in the 5.0 kernel, either single- or multi-channel,
  the channel parameter(s) must be set after all other parameters. It is at
  that time that the existing parameter values are acquired for use by the
  thread(s). All other parameters are shared. Therefore, if changes are made
  to any of the other parameters, and an additional channel specified, the
  (shared) parameters used for all threads will use the new values.
  After the channels are specified, each thread is set as pending. All threads
  begin execution when the run parameter is set to 1.

.. hint::
  A list of available channels can be found by running the following command::

    % ls -1 /sys/class/dma/

Once started a message like " dmatest: Added 1 threads using dma0chan0" is
emitted. A thread for that specific channel is created and is now pending, the
pending thread is started once run is to 1.

Note that running a new test will not stop any in progress test.

The following command returns the state of the test. ::

    % cat /sys/module/dmatest/parameters/run

To wait for test completion userpace can poll 'run' until it is false, or use
the wait parameter. Specifying 'wait=1' when loading the module causes module
initialization to pause until a test run has completed, while reading
/sys/module/dmatest/parameters/wait waits for any running test to complete
before returning. For example, the following scripts wait for 42 tests
to complete before exiting. Note that if 'iterations' is set to 'infinite' then
waiting is disabled.

Example::

    % modprobe dmatest run=1 iterations=42 wait=1
    % modprobe -r dmatest

...or::

    % modprobe dmatest run=1 iterations=42
    % cat /sys/module/dmatest/parameters/wait
    % modprobe -r dmatest

Part 3 - When built-in in the kernel
====================================

The module parameters that is supplied to the kernel command line will be used
for the first performed test. After user gets a control, the test could be
re-run with the same or different parameters. For the details see the above
section `Part 2 - When dmatest is built as a module`_.

In both cases the module parameters are used as the actual values for the test
case. You always could check them at run-time by running ::

    % grep -H . /sys/module/dmatest/parameters/*

Part 4 - Gathering the test results
===================================

Test results are printed to the kernel log buffer with the format::

    "dmatest: result <channel>: <test id>: '<error msg>' with src_off=<val> dst_off=<val> len=<val> (<err code>)"

Example of output::

    % dmesg | tail -n 1
    dmatest: result dma0chan0-copy0: #1: No errors with src_off=0x7bf dst_off=0x8ad len=0x3fea (0)

The message format is unified across the different types of errors. A
number in the parentheses represents additional information, e.g. error
code, error counter, or status. A test thread also emits a summary line at
completion listing the number of tests executed, number that failed, and a
result code.

Example::

    % dmesg | tail -n 1
    dmatest: dma0chan0-copy0: summary 1 test, 0 failures 1000 iops 100000 KB/s (0)

The details of a data miscompare error are also emitted, but do not follow the
above format.

Part 5 - Handling channel allocation
====================================

Allocating Channels
-------------------

Channels do not need to be configured prior to starting a test run. Attempting
to run the test without configuring the channels will result in testing any
channels that are available.

Example::

    % echo 1 > /sys/module/dmatest/parameters/run
    dmatest: No channels configured, continue with any

Channels are registered using the "channel" parameter. Channels can be requested by their
name, once requested, the channel is registered and a pending thread is added to the test list.

Example::

    % echo dma0chan2 > /sys/module/dmatest/parameters/channel
    dmatest: Added 1 threads using dma0chan2

More channels can be added by repeating the example above.
Reading back the channel parameter will return the name of last channel that was added successfully.

Example::

    % echo dma0chan1 > /sys/module/dmatest/parameters/channel
    dmatest: Added 1 threads using dma0chan1
    % echo dma0chan2 > /sys/module/dmatest/parameters/channel
    dmatest: Added 1 threads using dma0chan2
    % cat /sys/module/dmatest/parameters/channel
    dma0chan2

Another method of requesting channels is to request a channel with an empty string, Doing so
will request all channels available to be tested:

Example::

    % echo "" > /sys/module/dmatest/parameters/channel
    dmatest: Added 1 threads using dma0chan0
    dmatest: Added 1 threads using dma0chan3
    dmatest: Added 1 threads using dma0chan4
    dmatest: Added 1 threads using dma0chan5
    dmatest: Added 1 threads using dma0chan6
    dmatest: Added 1 threads using dma0chan7
    dmatest: Added 1 threads using dma0chan8

At any point during the test configuration, reading the "test_list" parameter will
print the list of currently pending tests.

Example::

    % cat /sys/module/dmatest/parameters/test_list
    dmatest: 1 threads using dma0chan0
    dmatest: 1 threads using dma0chan3
    dmatest: 1 threads using dma0chan4
    dmatest: 1 threads using dma0chan5
    dmatest: 1 threads using dma0chan6
    dmatest: 1 threads using dma0chan7
    dmatest: 1 threads using dma0chan8

Note: Channels will have to be configured for each test run as channel configurations do not
carry across to the next test run.

Releasing Channels
-------------------

Channels can be freed by setting run to 0.

Example::

    % echo dma0chan1 > /sys/module/dmatest/parameters/channel
    dmatest: Added 1 threads using dma0chan1
    % cat /sys/class/dma/dma0chan1/in_use
    1
    % echo 0 > /sys/module/dmatest/parameters/run
    % cat /sys/class/dma/dma0chan1/in_use
    0

Channels allocated by previous test runs are automatically freed when a new
channel is requested after completing a successful test run.


code:
=======================
DMAEngine documentation
=======================

DMAEngine documentation provides documents for various aspects of DMAEngine
framework.

DMAEngine development documentation
-----------------------------------

This book helps with DMAengine internal APIs and guide for DMAEngine device
driver writers.

.. toctree::
   :maxdepth: 1

   provider

DMAEngine client documentation
------------------------------

This book is a guide to device driver writers on how to use the Slave-DMA
API of the DMAEngine. This is applicable only for slave DMA usage only.

.. toctree::
   :maxdepth: 1

   client

DMA Test documentation
----------------------

This book introduces how to test DMA drivers using dmatest module.

.. toctree::
   :maxdepth: 1

   dmatest

PXA DMA documentation
----------------------

This book adds some notes about PXA DMA

.. toctree::
   :maxdepth: 1

   pxa_dma

.. only::  subproject and html

   Indices
   =======

   * :ref:`genindex`


code:
==================================
DMAengine controller documentation
==================================

Hardware Introduction
=====================

Most of the Slave DMA controllers have the same general principles of
operations.

They have a given number of channels to use for the DMA transfers, and
a given number of requests lines.

Requests and channels are pretty much orthogonal. Channels can be used
to serve several to any requests. To simplify, channels are the
entities that will be doing the copy, and requests what endpoints are
involved.

The request lines actually correspond to physical lines going from the
DMA-eligible devices to the controller itself. Whenever the device
will want to start a transfer, it will assert a DMA request (DRQ) by
asserting that request line.

A very simple DMA controller would only take into account a single
parameter: the transfer size. At each clock cycle, it would transfer a
byte of data from one buffer to another, until the transfer size has
been reached.

That wouldn't work well in the real world, since slave devices might
require a specific number of bits to be transferred in a single
cycle. For example, we may want to transfer as much data as the
physical bus allows to maximize performances when doing a simple
memory copy operation, but our audio device could have a narrower FIFO
that requires data to be written exactly 16 or 24 bits at a time. This
is why most if not all of the DMA controllers can adjust this, using a
parameter called the transfer width.

Moreover, some DMA controllers, whenever the RAM is used as a source
or destination, can group the reads or writes in memory into a buffer,
so instead of having a lot of small memory accesses, which is not
really efficient, you'll get several bigger transfers. This is done
using a parameter called the burst size, that defines how many single
reads/writes it's allowed to do without the controller splitting the
transfer into smaller sub-transfers.

Our theoretical DMA controller would then only be able to do transfers
that involve a single contiguous block of data. However, some of the
transfers we usually have are not, and want to copy data from
non-contiguous buffers to a contiguous buffer, which is called
scatter-gather.

DMAEngine, at least for mem2dev transfers, require support for
scatter-gather. So we're left with two cases here: either we have a
quite simple DMA controller that doesn't support it, and we'll have to
implement it in software, or we have a more advanced DMA controller,
that implements in hardware scatter-gather.

The latter are usually programmed using a collection of chunks to
transfer, and whenever the transfer is started, the controller will go
over that collection, doing whatever we programmed there.

This collection is usually either a table or a linked list. You will
then push either the address of the table and its number of elements,
or the first item of the list to one channel of the DMA controller,
and whenever a DRQ will be asserted, it will go through the collection
to know where to fetch the data from.

Either way, the format of this collection is completely dependent on
your hardware. Each DMA controller will require a different structure,
but all of them will require, for every chunk, at least the source and
destination addresses, whether it should increment these addresses or
not and the three parameters we saw earlier: the burst size, the
transfer width and the transfer size.

The one last thing is that usually, slave devices won't issue DRQ by
default, and you have to enable this in your slave device driver first
whenever you're willing to use DMA.

These were just the general memory-to-memory (also called mem2mem) or
memory-to-device (mem2dev) kind of transfers. Most devices often
support other kind of transfers or memory operations that dmaengine
support and will be detailed later in this document.

DMA Support in Linux
====================

Historically, DMA controller drivers have been implemented using the
async TX API, to offload operations such as memory copy, XOR,
cryptography, etc., basically any memory to memory operation.

Over time, the need for memory to device transfers arose, and
dmaengine was extended. Nowadays, the async TX API is written as a
layer on top of dmaengine, and acts as a client. Still, dmaengine
accommodates that API in some cases, and made some design choices to
ensure that it stayed compatible.

For more information on the Async TX API, please look the relevant
documentation file in Documentation/crypto/async-tx-api.rst.

DMAEngine APIs
==============

``struct dma_device`` Initialization
------------------------------------

Just like any other kernel framework, the whole DMAEngine registration
relies on the driver filling a structure and registering against the
framework. In our case, that structure is dma_device.

The first thing you need to do in your driver is to allocate this
structure. Any of the usual memory allocators will do, but you'll also
need to initialize a few fields in there:

- ``channels``: should be initialized as a list using the
  INIT_LIST_HEAD macro for example

- ``src_addr_widths``:
  should contain a bitmask of the supported source transfer width

- ``dst_addr_widths``:
  should contain a bitmask of the supported destination transfer width

- ``directions``:
  should contain a bitmask of the supported slave directions
  (i.e. excluding mem2mem transfers)

- ``residue_granularity``:
  granularity of the transfer residue reported to dma_set_residue.
  This can be either:

  - Descriptor:
    your device doesn't support any kind of residue
    reporting. The framework will only know that a particular
    transaction descriptor is done.

  - Segment:
    your device is able to report which chunks have been transferred

  - Burst:
    your device is able to report which burst have been transferred

- ``dev``: should hold the pointer to the ``struct device`` associated
  to your current driver instance.

Supported transaction types
---------------------------

The next thing you need is to set which transaction types your device
(and driver) supports.

Our ``dma_device structure`` has a field called cap_mask that holds the
various types of transaction supported, and you need to modify this
mask using the dma_cap_set function, with various flags depending on
transaction types you support as an argument.

All those capabilities are defined in the ``dma_transaction_type enum``,
in ``include/linux/dmaengine.h``

Currently, the types available are:

- DMA_MEMCPY

  - The device is able to do memory to memory copies

  - No matter what the overall size of the combined chunks for source and
    destination is, only as many bytes as the smallest of the two will be
    transmitted. That means the number and size of the scatter-gather buffers in
    both lists need not be the same, and that the operation functionally is
    equivalent to a ``strncpy`` where the ``count`` argument equals the smallest
    total size of the two scatter-gather list buffers.

  - It's usually used for copying pixel data between host memory and
    memory-mapped GPU device memory, such as found on modern PCI video graphics
    cards. The most immediate example is the OpenGL API function
    ``glReadPielx()``, which might require a verbatim copy of a huge framebuffer
    from local device memory onto host memory.

- DMA_XOR

  - The device is able to perform XOR operations on memory areas

  - Used to accelerate XOR intensive tasks, such as RAID5

- DMA_XOR_VAL

  - The device is able to perform parity check using the XOR
    algorithm against a memory buffer.

- DMA_PQ

  - The device is able to perform RAID6 P+Q computations, P being a
    simple XOR, and Q being a Reed-Solomon algorithm.

- DMA_PQ_VAL

  - The device is able to perform parity check using RAID6 P+Q
    algorithm against a memory buffer.

- DMA_MEMSET

  - The device is able to fill memory with the provided pattern

  - The pattern is treated as a single byte signed value.

- DMA_INTERRUPT

  - The device is able to trigger a dummy transfer that will
    generate periodic interrupts

  - Used by the client drivers to register a callback that will be
    called on a regular basis through the DMA controller interrupt

- DMA_PRIVATE

  - The devices only supports slave transfers, and as such isn't
    available for async transfers.

- DMA_ASYNC_TX

  - Must not be set by the device, and will be set by the framework
    if needed

  - TODO: What is it about?

- DMA_SLAVE

  - The device can handle device to memory transfers, including
    scatter-gather transfers.

  - While in the mem2mem case we were having two distinct types to
    deal with a single chunk to copy or a collection of them, here,
    we just have a single transaction type that is supposed to
    handle both.

  - If you want to transfer a single contiguous memory buffer,
    simply build a scatter list with only one item.

- DMA_CYCLIC

  - The device can handle cyclic transfers.

  - A cyclic transfer is a transfer where the chunk collection will
    loop over itself, with the last item pointing to the first.

  - It's usually used for audio transfers, where you want to operate
    on a single ring buffer that you will fill with your audio data.

- DMA_INTERLEAVE

  - The device supports interleaved transfer.

  - These transfers can transfer data from a non-contiguous buffer
    to a non-contiguous buffer, opposed to DMA_SLAVE that can
    transfer data from a non-contiguous data set to a continuous
    destination buffer.

  - It's usually used for 2d content transfers, in which case you
    want to transfer a portion of uncompressed data directly to the
    display to print it

- DMA_COMPLETION_NO_ORDER

  - The device does not support in order completion.

  - The driver should return DMA_OUT_OF_ORDER for device_tx_status if
    the device is setting this capability.

  - All cookie tracking and checking API should be treated as invalid if
    the device exports this capability.

  - At this point, this is incompatible with polling option for dmatest.

  - If this cap is set, the user is recommended to provide an unique
    identifier for each descriptor sent to the DMA device in order to
    properly track the completion.

- DMA_REPEAT

  - The device supports repeated transfers. A repeated transfer, indicated by
    the DMA_PREP_REPEAT transfer flag, is similar to a cyclic transfer in that
    it gets automatically repeated when it ends, but can additionally be
    replaced by the client.

  - This feature is limited to interleaved transfers, this flag should thus not
    be set if the DMA_INTERLEAVE flag isn't set. This limitation is based on
    the current needs of DMA clients, support for additional transfer types
    should be added in the future if and when the need arises.

- DMA_LOAD_EOT

  - The device supports replacing repeated transfers at end of transfer (EOT)
    by queuing a new transfer with the DMA_PREP_LOAD_EOT flag set.

  - Support for replacing a currently running transfer at another point (such
    as end of burst instead of end of transfer) will be added in the future
    based on DMA clients needs, if and when the need arises.

These various types will also affect how the source and destination
addresses change over time.

Addresses pointing to RAM are typically incremented (or decremented)
after each transfer. In case of a ring buffer, they may loop
(DMA_CYCLIC). Addresses pointing to a device's register (e.g. a FIFO)
are typically fixed.

Per descriptor metadata support
-------------------------------
Some data movement architecture (DMA controller and peripherals) uses metadata
associated with a transaction. The DMA controller role is to transfer the
payload and the metadata alongside.
The metadata itself is not used by the DMA engine itself, but it contains
parameters, keys, vectors, etc for peripheral or from the peripheral.

The DMAengine framework provides a generic ways to facilitate the metadata for
descriptors. Depending on the architecture the DMA driver can implement either
or both of the methods and it is up to the client driver to choose which one
to use.

- DESC_METADATA_CLIENT

  The metadata buffer is allocated/provided by the client driver and it is
  attached (via the dmaengine_desc_attach_metadata() helper to the descriptor.

  From the DMA driver the following is expected for this mode:

  - DMA_MEM_TO_DEV / DEV_MEM_TO_MEM

    The data from the provided metadata buffer should be prepared for the DMA
    controller to be sent alongside of the payload data. Either by copying to a
    hardware descriptor, or highly coupled packet.

  - DMA_DEV_TO_MEM

    On transfer completion the DMA driver must copy the metadata to the client
    provided metadata buffer before notifying the client about the completion.
    After the transfer completion, DMA drivers must not touch the metadata
    buffer provided by the client.

- DESC_METADATA_ENGINE

  The metadata buffer is allocated/managed by the DMA driver. The client driver
  can ask for the pointer, maximum size and the currently used size of the
  metadata and can directly update or read it. dmaengine_desc_get_metadata_ptr()
  and dmaengine_desc_set_metadata_len() is provided as helper functions.

  From the DMA driver the following is expected for this mode:

  - get_metadata_ptr()

    Should return a pointer for the metadata buffer, the maximum size of the
    metadata buffer and the currently used / valid (if any) bytes in the buffer.

  - set_metadata_len()

    It is called by the clients after it have placed the metadata to the buffer
    to let the DMA driver know the number of valid bytes provided.

  Note: since the client will ask for the metadata pointer in the completion
  callback (in DMA_DEV_TO_MEM case) the DMA driver must ensure that the
  descriptor is not freed up prior the callback is called.

Device operations
-----------------

Our dma_device structure also requires a few function pointers in
order to implement the actual logic, now that we described what
operations we were able to perform.

The functions that we have to fill in there, and hence have to
implement, obviously depend on the transaction types you reported as
supported.

- ``device_alloc_chan_resources``

- ``device_free_chan_resources``

  - These functions will be called whenever a driver will call
    ``dma_request_channel`` or ``dma_release_channel`` for the first/last
    time on the channel associated to that driver.

  - They are in charge of allocating/freeing all the needed
    resources in order for that channel to be useful for your driver.

  - These functions can sleep.

- ``device_prep_dma_*``

  - These functions are matching the capabilities you registered
    previously.

  - These functions all take the buffer or the scatterlist relevant
    for the transfer being prepared, and should create a hardware
    descriptor or a list of hardware descriptors from it

  - These functions can be called from an interrupt context

  - Any allocation you might do should be using the GFP_NOWAIT
    flag, in order not to potentially sleep, but without depleting
    the emergency pool either.

  - Drivers should try to pre-allocate any memory they might need
    during the transfer setup at probe time to avoid putting to
    much pressure on the nowait allocator.

  - It should return a unique instance of the
    ``dma_async_tx_descriptor structure``, that further represents this
    particular transfer.

  - This structure can be initialized using the function
    ``dma_async_tx_descriptor_init``.

  - You'll also need to set two fields in this structure:

    - flags:
      TODO: Can it be modified by the driver itself, or
      should it be always the flags passed in the arguments

    - tx_submit: A pointer to a function you have to implement,
      that is supposed to push the current transaction descriptor to a
      pending queue, waiting for issue_pending to be called.

  - In this structure the function pointer callback_result can be
    initialized in order for the submitter to be notified that a
    transaction has completed. In the earlier code the function pointer
    callback has been used. However it does not provide any status to the
    transaction and will be deprecated. The result structure defined as
    ``dmaengine_result`` that is passed in to callback_result
    has two fields:

    - result: This provides the transfer result defined by
      ``dmaengine_tx_result``. Either success or some error condition.

    - residue: Provides the residue bytes of the transfer for those that
      support residue.

- ``device_issue_pending``

  - Takes the first transaction descriptor in the pending queue,
    and starts the transfer. Whenever that transfer is done, it
    should move to the next transaction in the list.

  - This function can be called in an interrupt context

- ``device_tx_status``

  - Should report the bytes left to go over on the given channel

  - Should only care about the transaction descriptor passed as
    argument, not the currently active one on a given channel

  - The tx_state argument might be NULL

  - Should use dma_set_residue to report it

  - In the case of a cyclic transfer, it should only take into
    account the total size of the cyclic buffer.

  - Should return DMA_OUT_OF_ORDER if the device does not support in order
    completion and is completing the operation out of order.

  - This function can be called in an interrupt context.

- device_config

  - Reconfigures the channel with the configuration given as argument

  - This command should NOT perform synchronously, or on any
    currently queued transfers, but only on subsequent ones

  - In this case, the function will receive a ``dma_slave_config``
    structure pointer as an argument, that will detail which
    configuration to use.

  - Even though that structure contains a direction field, this
    field is deprecated in favor of the direction argument given to
    the prep_* functions

  - This call is mandatory for slave operations only. This should NOT be
    set or expected to be set for memcpy operations.
    If a driver support both, it should use this call for slave
    operations only and not for memcpy ones.

- device_pause

  - Pauses a transfer on the channel

  - This command should operate synchronously on the channel,
    pausing right away the work of the given channel

- device_resume

  - Resumes a transfer on the channel

  - This command should operate synchronously on the channel,
    resuming right away the work of the given channel

- device_terminate_all

  - Aborts all the pending and ongoing transfers on the channel

  - For aborted transfers the complete callback should not be called

  - Can be called from atomic context or from within a complete
    callback of a descriptor. Must not sleep. Drivers must be able
    to handle this correctly.

  - Termination may be asynchronous. The driver does not have to
    wait until the currently active transfer has completely stopped.
    See device_synchronize.

- device_synchronize

  - Must synchronize the termination of a channel to the current
    context.

  - Must make sure that memory for previously submitted
    descriptors is no longer accessed by the DMA controller.

  - Must make sure that all complete callbacks for previously
    submitted descriptors have finished running and none are
    scheduled to run.

  - May sleep.

Misc notes
==========

(stuff that should be documented, but don't really know
where to put them)

``dma_run_dependencies``

- Should be called at the end of an async TX transfer, and can be
  ignored in the slave transfers case.

- Makes sure that dependent operations are run before marking it
  as complete.

dma_cookie_t

- it's a DMA transaction ID that will increment over time.

- Not really relevant any more since the introduction of ``virt-dma``
  that abstracts it away.

DMA_CTRL_ACK

- If clear, the descriptor cannot be reused by provider until the
  client acknowledges receipt, i.e. has a chance to establish any
  dependency chains

- This can be acked by invoking async_tx_ack()

- If set, does not mean descriptor can be reused

DMA_CTRL_REUSE

- If set, the descriptor can be reused after being completed. It should
  not be freed by provider if this flag is set.

- The descriptor should be prepared for reuse by invoking
  ``dmaengine_desc_set_reuse()`` which will set DMA_CTRL_REUSE.

- ``dmaengine_desc_set_reuse()`` will succeed only when channel support
  reusable descriptor as exhibited by capabilities

- As a consequence, if a device driver wants to skip the
  ``dma_map_sg()`` and ``dma_unmap_sg()`` in between 2 transfers,
  because the DMA'd data wasn't used, it can resubmit the transfer right after
  its completion.

- Descriptor can be freed in few ways

  - Clearing DMA_CTRL_REUSE by invoking
    ``dmaengine_desc_clear_reuse()`` and submitting for last txn

  - Explicitly invoking ``dmaengine_desc_free()``, this can succeed only
    when DMA_CTRL_REUSE is already set

  - Terminating the channel

- DMA_PREP_CMD

  - If set, the client driver tells DMA controller that passed data in DMA
    API is command data.

  - Interpretation of command data is DMA controller specific. It can be
    used for issuing commands to other peripherals/register reads/register
    writes for which the descriptor should be in different format from
    normal data descriptors.

- DMA_PREP_REPEAT

  - If set, the transfer will be automatically repeated when it ends until a
    new transfer is queued on the same channel with the DMA_PREP_LOAD_EOT flag.
    If the next transfer to be queued on the channel does not have the
    DMA_PREP_LOAD_EOT flag set, the current transfer will be repeated until the
    client terminates all transfers.

  - This flag is only supported if the channel reports the DMA_REPEAT
    capability.

- DMA_PREP_LOAD_EOT

  - If set, the transfer will replace the transfer currently being executed at
    the end of the transfer.

  - This is the default behaviour for non-repeated transfers, specifying
    DMA_PREP_LOAD_EOT for non-repeated transfers will thus make no difference.

  - When using repeated transfers, DMA clients will usually need to set the
    DMA_PREP_LOAD_EOT flag on all transfers, otherwise the channel will keep
    repeating the last repeated transfer and ignore the new transfers being
    queued. Failure to set DMA_PREP_LOAD_EOT will appear as if the channel was
    stuck on the previous transfer.

  - This flag is only supported if the channel reports the DMA_LOAD_EOT
    capability.

General Design Notes
====================

Most of the DMAEngine drivers you'll see are based on a similar design
that handles the end of transfer interrupts in the handler, but defer
most work to a tasklet, including the start of a new transfer whenever
the previous transfer ended.

This is a rather inefficient design though, because the inter-transfer
latency will be not only the interrupt latency, but also the
scheduling latency of the tasklet, which will leave the channel idle
in between, which will slow down the global transfer rate.

You should avoid this kind of practice, and instead of electing a new
transfer in your tasklet, move that part to the interrupt handler in
order to have a shorter idle window (that we can't really avoid
anyway).

Glossary
========

- Burst: A number of consecutive read or write operations that
  can be queued to buffers before being flushed to memory.

- Chunk: A contiguous collection of bursts

- Transfer: A collection of chunks (be it contiguous or not)


code:
==============================
PXA/MMP - DMA Slave controller
==============================

Constraints
===========

a) Transfers hot queuing
A driver submitting a transfer and issuing it should be granted the transfer
is queued even on a running DMA channel.
This implies that the queuing doesn't wait for the previous transfer end,
and that the descriptor chaining is not only done in the irq/tasklet code
triggered by the end of the transfer.
A transfer which is submitted and issued on a phy doesn't wait for a phy to
stop and restart, but is submitted on a "running channel". The other
drivers, especially mmp_pdma waited for the phy to stop before relaunching
a new transfer.

b) All transfers having asked for confirmation should be signaled
Any issued transfer with DMA_PREP_INTERRUPT should trigger a callback call.
This implies that even if an irq/tasklet is triggered by end of tx1, but
at the time of irq/dma tx2 is already finished, tx1->complete() and
tx2->complete() should be called.

c) Channel running state
A driver should be able to query if a channel is running or not. For the
multimedia case, such as video capture, if a transfer is submitted and then
a check of the DMA channel reports a "stopped channel", the transfer should
not be issued until the next "start of frame interrupt", hence the need to
know if a channel is in running or stopped state.

d) Bandwidth guarantee
The PXA architecture has 4 levels of DMAs priorities : high, normal, low.
The high priorities get twice as much bandwidth as the normal, which get twice
as much as the low priorities.
A driver should be able to request a priority, especially the real-time
ones such as pxa_camera with (big) throughputs.

Design
======
a) Virtual channels
Same concept as in sa11x0 driver, ie. a driver was assigned a "virtual
channel" linked to the requestor line, and the physical DMA channel is
assigned on the fly when the transfer is issued.

b) Transfer anatomy for a scatter-gather transfer

::

   +------------+-----+---------------+----------------+-----------------+
   | desc-sg[0] | ... | desc-sg[last] | status updater | finisher/linker |
   +------------+-----+---------------+----------------+-----------------+

This structure is pointed by dma->sg_cpu.
The descriptors are used as follows :

    - desc-sg[i]: i-th descriptor, transferring the i-th sg
      element to the video buffer scatter gather

    - status updater
      Transfers a single u32 to a well known dma coherent memory to leave
      a trace that this transfer is done. The "well known" is unique per
      physical channel, meaning that a read of this value will tell which
      is the last finished transfer at that point in time.

    - finisher: has ddadr=DADDR_STOP, dcmd=ENDIRQEN

    - linker: has ddadr= desc-sg[0] of next transfer, dcmd=0

c) Transfers hot-chaining
Suppose the running chain is:

::

   Buffer 1              Buffer 2
   +---------+----+---+  +----+----+----+---+
   | d0 | .. | dN | l |  | d0 | .. | dN | f |
   +---------+----+-|-+  ^----+----+----+---+
                    |    |
                    +----+

After a call to dmaengine_submit(b3), the chain will look like:

::

   Buffer 1              Buffer 2              Buffer 3
   +---------+----+---+  +----+----+----+---+  +----+----+----+---+
   | d0 | .. | dN | l |  | d0 | .. | dN | l |  | d0 | .. | dN | f |
   +---------+----+-|-+  ^----+----+----+-|-+  ^----+----+----+---+
                    |    |                |    |
                    +----+                +----+
                                         new_link

If while new_link was created the DMA channel stopped, it is _not_
restarted. Hot-chaining doesn't break the assumption that
dma_async_issue_pending() is to be used to ensure the transfer is actually started.

One exception to this rule :

- if Buffer1 and Buffer2 had all their addresses 8 bytes aligned

- and if Buffer3 has at least one address not 4 bytes aligned

- then hot-chaining cannot happen, as the channel must be stopped, the
  "align bit" must be set, and the channel restarted As a consequence,
  such a transfer tx_submit() will be queued on the submitted queue, and
  this specific case if the DMA is already running in aligned mode.

d) Transfers completion updater
Each time a transfer is completed on a channel, an interrupt might be
generated or not, up to the client's request. But in each case, the last
descriptor of a transfer, the "status updater", will write the latest
transfer being completed into the physical channel's completion mark.

This will speed up residue calculation, for large transfers such as video
buffers which hold around 6k descriptors or more. This also allows without
any lock to find out what is the latest completed transfer in a running
DMA chain.

e) Transfers completion, irq and tasklet
When a transfer flagged as "DMA_PREP_INTERRUPT" is finished, the dma irq
is raised. Upon this interrupt, a tasklet is scheduled for the physical
channel.

The tasklet is responsible for :

- reading the physical channel last updater mark

- calling all the transfer callbacks of finished transfers, based on
  that mark, and each transfer flags.

If a transfer is completed while this handling is done, a dma irq will
be raised, and the tasklet will be scheduled once again, having a new
updater mark.

f) Residue
Residue granularity will be descriptor based. The issued but not completed
transfers will be scanned for all of their descriptors against the
currently running descriptor.

g) Most complicated case of driver's tx queues
The most tricky situation is when :

 - there are not "acked" transfers (tx0)

 - a driver submitted an aligned tx1, not chained

 - a driver submitted an aligned tx2 => tx2 is cold chained to tx1

 - a driver issued tx1+tx2 => channel is running in aligned mode

 - a driver submitted an aligned tx3 => tx3 is hot-chained

 - a driver submitted an unaligned tx4 => tx4 is put in submitted queue,
   not chained

 - a driver issued tx4 => tx4 is put in issued queue, not chained

 - a driver submitted an aligned tx5 => tx5 is put in submitted queue, not
   chained

 - a driver submitted an aligned tx6 => tx6 is put in submitted queue,
   cold chained to tx5

 This translates into (after tx4 is issued) :

 - issued queue

 ::

      +-----+ +-----+ +-----+ +-----+
      | tx1 | | tx2 | | tx3 | | tx4 |
      +---|-+ ^---|-+ ^-----+ +-----+
          |   |   |   |
          +---+   +---+
        - submitted queue
      +-----+ +-----+
      | tx5 | | tx6 |
      +---|-+ ^-----+
          |   |
          +---+

- completed queue : empty

- allocated queue : tx0

It should be noted that after tx3 is completed, the channel is stopped, and
restarted in "unaligned mode" to handle tx4.

Author: Robert Jarzmik <robert.jarzmik@free.fr>


code:
==============
Driver Binding
==============

Driver binding is the process of associating a device with a device
driver that can control it. Bus drivers have typically handled this
because there have been bus-specific structures to represent the
devices and the drivers. With generic device and device driver
structures, most of the binding can take place using common code.

Bus
~~~

The bus type structure contains a list of all devices that are on that bus
type in the system. When device_register is called for a device, it is
inserted into the end of this list. The bus object also contains a
list of all drivers of that bus type. When driver_register is called
for a driver, it is inserted at the end of this list. These are the
two events which trigger driver binding.

device_register
~~~~~~~~~~~~~~~

When a new device is added, the bus's list of drivers is iterated over
to find one that supports it. In order to determine that, the device
ID of the device must match one of the device IDs that the driver
supports. The format and semantics for comparing IDs is bus-specific.
Instead of trying to derive a complex state machine and matching
algorithm, it is up to the bus driver to provide a callback to compare
a device against the IDs of a driver. The bus returns 1 if a match was
found; 0 otherwise.

int match(struct device * dev, struct device_driver * drv);

If a match is found, the device's driver field is set to the driver
and the driver's probe callback is called. This gives the driver a
chance to verify that it really does support the hardware, and that
it's in a working state.

Device Class
~~~~~~~~~~~~

Upon the successful completion of probe, the device is registered with
the class to which it belongs. Device drivers belong to one and only one
class, and that is set in the driver's devclass field.
devclass_add_device is called to enumerate the device within the class
and actually register it with the class, which happens with the
class's register_dev callback.

Driver
~~~~~~

When a driver is attached to a device, the device is inserted into the
driver's list of devices.

sysfs
~~~~~

A symlink is created in the bus's 'devices' directory that points to
the device's directory in the physical hierarchy.

A symlink is created in the driver's 'devices' directory that points
to the device's directory in the physical hierarchy.

A directory for the device is created in the class's directory. A
symlink is created in that directory that points to the device's
physical location in the sysfs tree.

A symlink can be created (though this isn't done yet) in the device's
physical directory to either its class directory, or the class's
top-level directory. One can also be created to point to its driver's
directory also.

driver_register
~~~~~~~~~~~~~~~

The process is almost identical for when a new driver is added.
The bus's list of devices is iterated over to find a match. Devices
that already have a driver are skipped. All the devices are iterated
over, to bind as many devices as possible to the driver.

Removal
~~~~~~~

When a device is removed, the reference count for it will eventually
go to 0. When it does, the remove callback of the driver is called. It
is removed from the driver's list of devices and the reference count
of the driver is decremented. All symlinks between the two are removed.

When a driver is removed, the list of devices that it supports is
iterated over, and the driver's remove callback is called for each
one. The device is removed from that list and the symlinks removed.


code:
=========
Bus Types
=========

Definition
~~~~~~~~~~
See the kerneldoc for the struct bus_type.

int bus_register(struct bus_type * bus);

Declaration
~~~~~~~~~~~

Each bus type in the kernel (PCI, USB, etc) should declare one static
object of this type. They must initialize the name field, and may
optionally initialize the match callback::

   struct bus_type pci_bus_type = {
          .name	= "pci",
          .match	= pci_bus_match,
   };

The structure should be exported to drivers in a header file:

extern struct bus_type pci_bus_type;

Registration
~~~~~~~~~~~~

When a bus driver is initialized, it calls bus_register. This
initializes the rest of the fields in the bus object and inserts it
into a global list of bus types. Once the bus object is registered,
the fields in it are usable by the bus driver.

Callbacks
~~~~~~~~~

match(): Attaching Drivers to Devices
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The format of device ID structures and the semantics for comparing
them are inherently bus-specific. Drivers typically declare an array
of device IDs of devices they support that reside in a bus-specific
driver structure.

The purpose of the match callback is to give the bus an opportunity to
determine if a particular driver supports a particular device by
comparing the device IDs the driver supports with the device ID of a
particular device, without sacrificing bus-specific functionality or
type-safety.

When a driver is registered with the bus, the bus's list of devices is
iterated over, and the match callback is called for each device that
does not have a driver associated with it.

Device and Driver Lists
~~~~~~~~~~~~~~~~~~~~~~~

The lists of devices and drivers are intended to replace the local
lists that many buses keep. They are lists of struct devices and
struct device_drivers, respectively. Bus drivers are free to use the
lists as they please, but conversion to the bus-specific type may be
necessary.

The LDM core provides helper functions for iterating over each list::

  int bus_for_each_dev(struct bus_type * bus, struct device * start,
		       void * data,
		       int (*fn)(struct device *, void *));

  int bus_for_each_drv(struct bus_type * bus, struct device_driver * start,
		       void * data, int (*fn)(struct device_driver *, void *));

These helpers iterate over the respective list, and call the callback
for each device or driver in the list. All list accesses are
synchronized by taking the bus's lock (read currently). The reference
count on each object in the list is incremented before the callback is
called; it is decremented after the next object has been obtained. The
lock is not held when calling the callback.

sysfs
~~~~~~~~
There is a top-level directory named 'bus'.

Each bus gets a directory in the bus directory, along with two default
directories::

	/sys/bus/pci/
	|-- devices
	`-- drivers

Drivers registered with the bus get a directory in the bus's drivers
directory::

	/sys/bus/pci/
	|-- devices
	`-- drivers
	    |-- Intel ICH
	    |-- Intel ICH Joystick
	    |-- agpgart
	    `-- e100

Each device that is discovered on a bus of that type gets a symlink in
the bus's devices directory to the device's directory in the physical
hierarchy::

	/sys/bus/pci/
	|-- devices
	|   |-- 00:00.0 -> ../../../root/pci0/00:00.0
	|   |-- 00:01.0 -> ../../../root/pci0/00:01.0
	|   `-- 00:02.0 -> ../../../root/pci0/00:02.0
	`-- drivers

Exporting Attributes
~~~~~~~~~~~~~~~~~~~~

::

  struct bus_attribute {
	struct attribute	attr;
	ssize_t (*show)(struct bus_type *, char * buf);
	ssize_t (*store)(struct bus_type *, const char * buf, size_t count);
  };

Bus drivers can export attributes using the BUS_ATTR_RW macro that works
similarly to the DEVICE_ATTR_RW macro for devices. For example, a
definition like this::

	static BUS_ATTR_RW(debug);

is equivalent to declaring::

	static bus_attribute bus_attr_debug;

This can then be used to add and remove the attribute from the bus's
sysfs directory using::

	int bus_create_file(struct bus_type *, struct bus_attribute *);
	void bus_remove_file(struct bus_type *, struct bus_attribute *);


code:
=============================
Device Driver Design Patterns
=============================

This document describes a few common design patterns found in device drivers.
It is likely that subsystem maintainers will ask driver developers to
conform to these design patterns.

1. State Container
2. container_of()

1. State Container
~~~~~~~~~~~~~~~~~~

While the kernel contains a few device drivers that assume that they will
only be probed() once on a certain system (singletons), it is custom to assume
that the device the driver binds to will appear in several instances. This
means that the probe() function and all callbacks need to be reentrant.

The most common way to achieve this is to use the state container design
pattern. It usually has this form::

  struct foo {
      spinlock_t lock; /* Example member */
      (...)
  };

  static int foo_probe(...)
  {
      struct foo *foo;

      foo = devm_kzalloc(dev, sizeof(*foo), GFP_KERNEL);
      if (!foo)
          return -ENOMEM;
      spin_lock_init(&foo->lock);
      (...)
  }

This will create an instance of struct foo in memory every time probe() is
called. This is our state container for this instance of the device driver.
Of course it is then necessary to always pass this instance of the
state around to all functions that need access to the state and its members.

For example, if the driver is registering an interrupt handler, you would
pass around a pointer to struct foo like this::

  static irqreturn_t foo_handler(int irq, void *arg)
  {
      struct foo *foo = arg;
      (...)
  }

  static int foo_probe(...)
  {
      struct foo *foo;

      (...)
      ret = request_irq(irq, foo_handler, 0, "foo", foo);
  }

This way you always get a pointer back to the correct instance of foo in
your interrupt handler.

2. container_of()
~~~~~~~~~~~~~~~~~

Continuing on the above example we add an offloaded work::

  struct foo {
      spinlock_t lock;
      struct workqueue_struct *wq;
      struct work_struct offload;
      (...)
  };

  static void foo_work(struct work_struct *work)
  {
      struct foo *foo = container_of(work, struct foo, offload);

      (...)
  }

  static irqreturn_t foo_handler(int irq, void *arg)
  {
      struct foo *foo = arg;

      queue_work(foo->wq, &foo->offload);
      (...)
  }

  static int foo_probe(...)
  {
      struct foo *foo;

      foo->wq = create_singlethread_workqueue("foo-wq");
      INIT_WORK(&foo->offload, foo_work);
      (...)
  }

The design pattern is the same for an hrtimer or something similar that will
return a single argument which is a pointer to a struct member in the
callback.

container_of() is a macro defined in <linux/kernel.h>

What container_of() does is to obtain a pointer to the containing struct from
a pointer to a member by a simple subtraction using the offsetof() macro from
standard C, which allows something similar to object oriented behaviours.
Notice that the contained member must not be a pointer, but an actual member
for this to work.

We can see here that we avoid having global pointers to our struct foo *
instance this way, while still keeping the number of parameters passed to the
work function to a single pointer.


code:
==========================
The Basic Device Structure
==========================

See the kerneldoc for the struct device.

Programming Interface
~~~~~~~~~~~~~~~~~~~~~
The bus driver that discovers the device uses this to register the
device with the core::

  int device_register(struct device * dev);

The bus should initialize the following fields:

    - parent
    - name
    - bus_id
    - bus

A device is removed from the core when its reference count goes to
0. The reference count can be adjusted using::

  struct device * get_device(struct device * dev);
  void put_device(struct device * dev);

get_device() will return a pointer to the struct device passed to it
if the reference is not already 0 (if it's in the process of being
removed already).

A driver can access the lock in the device structure using::

  void lock_device(struct device * dev);
  void unlock_device(struct device * dev);

Attributes
~~~~~~~~~~

::

  struct device_attribute {
	struct attribute	attr;
	ssize_t (*show)(struct device *dev, struct device_attribute *attr,
			char *buf);
	ssize_t (*store)(struct device *dev, struct device_attribute *attr,
			 const char *buf, size_t count);
  };

Attributes of devices can be exported by a device driver through sysfs.

Please see Documentation/filesystems/sysfs.rst for more information
on how sysfs works.

As explained in Documentation/core-api/kobject.rst, device attributes must be
created before the KOBJ_ADD uevent is generated. The only way to realize
that is by defining an attribute group.

Attributes are declared using a macro called DEVICE_ATTR::

  #define DEVICE_ATTR(name,mode,show,store)

Example:::

  static DEVICE_ATTR(type, 0444, type_show, NULL);
  static DEVICE_ATTR(power, 0644, power_show, power_store);

Helper macros are available for common values of mode, so the above examples
can be simplified to:::

  static DEVICE_ATTR_RO(type);
  static DEVICE_ATTR_RW(power);

This declares two structures of type struct device_attribute with respective
names 'dev_attr_type' and 'dev_attr_power'. These two attributes can be
organized as follows into a group::

  static struct attribute *dev_attrs[] = {
	&dev_attr_type.attr,
	&dev_attr_power.attr,
	NULL,
  };

  static struct attribute_group dev_group = {
	.attrs = dev_attrs,
  };

  static const struct attribute_group *dev_groups[] = {
	&dev_group,
	NULL,
  };

A helper macro is available for the common case of a single group, so the
above two structures can be declared using:::

  ATTRIBUTE_GROUPS(dev);

This array of groups can then be associated with a device by setting the
group pointer in struct device before device_register() is invoked::

        dev->groups = dev_groups;
        device_register(dev);

The device_register() function will use the 'groups' pointer to create the
device attributes and the device_unregister() function will use this pointer
to remove the device attributes.

Word of warning:  While the kernel allows device_create_file() and
device_remove_file() to be called on a device at any time, userspace has
strict expectations on when attributes get created.  When a new device is
registered in the kernel, a uevent is generated to notify userspace (like
udev) that a new device is available.  If attributes are added after the
device is registered, then userspace won't get notified and userspace will
not know about the new attributes.

This is important for device driver that need to publish additional
attributes for a device at driver probe time.  If the device driver simply
calls device_create_file() on the device structure passed to it, then
userspace will never be notified of the new attributes.