code:
=========================
Writing a MUSB Glue Layer
=========================

:Author: Apelete Seketeli

Introduction
============

The Linux MUSB subsystem is part of the larger Linux USB subsystem. It
provides support for embedded USB Device Controllers (UDC) that do not
use Universal Host Controller Interface (UHCI) or Open Host Controller
Interface (OHCI).

Instead, these embedded UDC rely on the USB On-the-Go (OTG)
specification which they implement at least partially. The silicon
reference design used in most cases is the Multipoint USB Highspeed
Dual-Role Controller (MUSB HDRC) found in the Mentor Graphics Inventra™
design.

As a self-taught exercise I have written an MUSB glue layer for the
Ingenic JZ4740 SoC, modelled after the many MUSB glue layers in the
kernel source tree. This layer can be found at
``drivers/usb/musb/jz4740.c``. In this documentation I will walk through the
basics of the ``jz4740.c`` glue layer, explaining the different pieces and
what needs to be done in order to write your own device glue layer.

.. _musb-basics:

Linux MUSB Basics
=================

To get started on the topic, please read USB On-the-Go Basics (see
Resources) which provides an introduction of USB OTG operation at the
hardware level. A couple of wiki pages by Texas Instruments and Analog
Devices also provide an overview of the Linux kernel MUSB configuration,
albeit focused on some specific devices provided by these companies.
Finally, getting acquainted with the USB specification at USB home page
may come in handy, with practical instance provided through the Writing
USB Device Drivers documentation (again, see Resources).

Linux USB stack is a layered architecture in which the MUSB controller
hardware sits at the lowest. The MUSB controller driver abstract the
MUSB controller hardware to the Linux USB stack::

	  ------------------------
	  |                      | <------- drivers/usb/gadget
	  | Linux USB Core Stack | <------- drivers/usb/host
	  |                      | <------- drivers/usb/core
	  ------------------------
		     ⬍
	 --------------------------
	 |                        | <------ drivers/usb/musb/musb_gadget.c
	 | MUSB Controller driver | <------ drivers/usb/musb/musb_host.c
	 |                        | <------ drivers/usb/musb/musb_core.c
	 --------------------------
		     ⬍
      ---------------------------------
      | MUSB Platform Specific Driver |
      |                               | <-- drivers/usb/musb/jz4740.c
      |       aka "Glue Layer"        |
      ---------------------------------
		     ⬍
      ---------------------------------
      |   MUSB Controller Hardware    |
      ---------------------------------

As outlined above, the glue layer is actually the platform specific code
sitting in between the controller driver and the controller hardware.

Just like a Linux USB driver needs to register itself with the Linux USB
subsystem, the MUSB glue layer needs first to register itself with the
MUSB controller driver. This will allow the controller driver to know
about which device the glue layer supports and which functions to call
when a supported device is detected or released; remember we are talking
about an embedded controller chip here, so no insertion or removal at
run-time.

All of this information is passed to the MUSB controller driver through
a :c:type:`platform_driver` structure defined in the glue layer as::

    static struct platform_driver jz4740_driver = {
	.probe      = jz4740_probe,
	.remove     = jz4740_remove,
	.driver     = {
	    .name   = "musb-jz4740",
	},
    };

The probe and remove function pointers are called when a matching device
is detected and, respectively, released. The name string describes the
device supported by this glue layer. In the current case it matches a
platform_device structure declared in ``arch/mips/jz4740/platform.c``. Note
that we are not using device tree bindings here.

In order to register itself to the controller driver, the glue layer
goes through a few steps, basically allocating the controller hardware
resources and initialising a couple of circuits. To do so, it needs to
keep track of the information used throughout these steps. This is done
by defining a private ``jz4740_glue`` structure::

    struct jz4740_glue {
	struct device           *dev;
	struct platform_device  *musb;
	struct clk      *clk;
    };

The dev and musb members are both device structure variables. The first
one holds generic information about the device, since it's the basic
device structure, and the latter holds information more closely related
to the subsystem the device is registered to. The clk variable keeps
information related to the device clock operation.

Let's go through the steps of the probe function that leads the glue
layer to register itself to the controller driver.

.. note::

   For the sake of readability each function will be split in logical
   parts, each part being shown as if it was independent from the others.

.. code-block:: c
    :emphasize-lines: 8,12,18

    static int jz4740_probe(struct platform_device *pdev)
    {
	struct platform_device      *musb;
	struct jz4740_glue      *glue;
	struct clk                      *clk;
	int             ret;

	glue = devm_kzalloc(&pdev->dev, sizeof(*glue), GFP_KERNEL);
	if (!glue)
	    return -ENOMEM;

	musb = platform_device_alloc("musb-hdrc", PLATFORM_DEVID_AUTO);
	if (!musb) {
	    dev_err(&pdev->dev, "failed to allocate musb device\n");
	    return -ENOMEM;
	}

	clk = devm_clk_get(&pdev->dev, "udc");
	if (IS_ERR(clk)) {
	    dev_err(&pdev->dev, "failed to get clock\n");
	    ret = PTR_ERR(clk);
	    goto err_platform_device_put;
	}

	ret = clk_prepare_enable(clk);
	if (ret) {
	    dev_err(&pdev->dev, "failed to enable clock\n");
	    goto err_platform_device_put;
	}

	musb->dev.parent        = &pdev->dev;

	glue->dev           = &pdev->dev;
	glue->musb          = musb;
	glue->clk           = clk;

	return 0;

    err_platform_device_put:
	platform_device_put(musb);
	return ret;
    }

The first few lines of the probe function allocate and assign the glue,
musb and clk variables. The ``GFP_KERNEL`` flag (line 8) allows the
allocation process to sleep and wait for memory, thus being usable in a
locking situation. The ``PLATFORM_DEVID_AUTO`` flag (line 12) allows
automatic allocation and management of device IDs in order to avoid
device namespace collisions with explicit IDs. With :c:func:`devm_clk_get`
(line 18) the glue layer allocates the clock -- the ``devm_`` prefix
indicates that :c:func:`clk_get` is managed: it automatically frees the
allocated clock resource data when the device is released -- and enable
it.

Then comes the registration steps:

.. code-block:: c
    :emphasize-lines: 3,5,7,9,16

    static int jz4740_probe(struct platform_device *pdev)
    {
	struct musb_hdrc_platform_data  *pdata = &jz4740_musb_platform_data;

	pdata->platform_ops     = &jz4740_musb_ops;

	platform_set_drvdata(pdev, glue);

	ret = platform_device_add_resources(musb, pdev->resource,
			    pdev->num_resources);
	if (ret) {
	    dev_err(&pdev->dev, "failed to add resources\n");
	    goto err_clk_disable;
	}

	ret = platform_device_add_data(musb, pdata, sizeof(*pdata));
	if (ret) {
	    dev_err(&pdev->dev, "failed to add platform_data\n");
	    goto err_clk_disable;
	}

	return 0;

    err_clk_disable:
	clk_disable_unprepare(clk);
    err_platform_device_put:
	platform_device_put(musb);
	return ret;
    }

The first step is to pass the device data privately held by the glue
layer on to the controller driver through :c:func:`platform_set_drvdata`
(line 7). Next is passing on the device resources information, also privately
held at that point, through :c:func:`platform_device_add_resources` (line 9).

Finally comes passing on the platform specific data to the controller
driver (line 16). Platform data will be discussed in
:ref:`musb-dev-platform-data`, but here we are looking at the
``platform_ops`` function pointer (line 5) in ``musb_hdrc_platform_data``
structure (line 3). This function pointer allows the MUSB controller
driver to know which function to call for device operation::

    static const struct musb_platform_ops jz4740_musb_ops = {
	.init       = jz4740_musb_init,
	.exit       = jz4740_musb_exit,
    };

Here we have the minimal case where only init and exit functions are
called by the controller driver when needed. Fact is the JZ4740 MUSB
controller is a basic controller, lacking some features found in other
controllers, otherwise we may also have pointers to a few other
functions like a power management function or a function to switch
between OTG and non-OTG modes, for instance.

At that point of the registration process, the controller driver
actually calls the init function:

   .. code-block:: c
    :emphasize-lines: 12,14

    static int jz4740_musb_init(struct musb *musb)
    {
	musb->xceiv = usb_get_phy(USB_PHY_TYPE_USB2);
	if (!musb->xceiv) {
	    pr_err("HS UDC: no transceiver configured\n");
	    return -ENODEV;
	}

	/* Silicon does not implement ConfigData register.
	 * Set dyn_fifo to avoid reading EP config from hardware.
	 */
	musb->dyn_fifo = true;

	musb->isr = jz4740_musb_interrupt;

	return 0;
    }

The goal of ``jz4740_musb_init()`` is to get hold of the transceiver
driver data of the MUSB controller hardware and pass it on to the MUSB
controller driver, as usual. The transceiver is the circuitry inside the
controller hardware responsible for sending/receiving the USB data.
Since it is an implementation of the physical layer of the OSI model,
the transceiver is also referred to as PHY.

Getting hold of the ``MUSB PHY`` driver data is done with ``usb_get_phy()``
which returns a pointer to the structure containing the driver instance
data. The next couple of instructions (line 12 and 14) are used as a
quirk and to setup IRQ handling respectively. Quirks and IRQ handling
will be discussed later in :ref:`musb-dev-quirks` and
:ref:`musb-handling-irqs`\ ::

    static int jz4740_musb_exit(struct musb *musb)
    {
	usb_put_phy(musb->xceiv);

	return 0;
    }

Acting as the counterpart of init, the exit function releases the MUSB
PHY driver when the controller hardware itself is about to be released.

Again, note that init and exit are fairly simple in this case due to the
basic set of features of the JZ4740 controller hardware. When writing an
musb glue layer for a more complex controller hardware, you might need
to take care of more processing in those two functions.

Returning from the init function, the MUSB controller driver jumps back
into the probe function::

    static int jz4740_probe(struct platform_device *pdev)
    {
	ret = platform_device_add(musb);
	if (ret) {
	    dev_err(&pdev->dev, "failed to register musb device\n");
	    goto err_clk_disable;
	}

	return 0;

    err_clk_disable:
	clk_disable_unprepare(clk);
    err_platform_device_put:
	platform_device_put(musb);
	return ret;
    }

This is the last part of the device registration process where the glue
layer adds the controller hardware device to Linux kernel device
hierarchy: at this stage, all known information about the device is
passed on to the Linux USB core stack:

   .. code-block:: c
    :emphasize-lines: 5,6

    static int jz4740_remove(struct platform_device *pdev)
    {
	struct jz4740_glue  *glue = platform_get_drvdata(pdev);

	platform_device_unregister(glue->musb);
	clk_disable_unprepare(glue->clk);

	return 0;
    }

Acting as the counterpart of probe, the remove function unregister the
MUSB controller hardware (line 5) and disable the clock (line 6),
allowing it to be gated.

.. _musb-handling-irqs:

Handling IRQs
=============

Additionally to the MUSB controller hardware basic setup and
registration, the glue layer is also responsible for handling the IRQs:

   .. code-block:: c
    :emphasize-lines: 7,9-11,14,24

    static irqreturn_t jz4740_musb_interrupt(int irq, void *__hci)
    {
	unsigned long   flags;
	irqreturn_t     retval = IRQ_NONE;
	struct musb     *musb = __hci;

	spin_lock_irqsave(&musb->lock, flags);

	musb->int_usb = musb_readb(musb->mregs, MUSB_INTRUSB);
	musb->int_tx = musb_readw(musb->mregs, MUSB_INTRTX);
	musb->int_rx = musb_readw(musb->mregs, MUSB_INTRRX);

	/*
	 * The controller is gadget only, the state of the host mode IRQ bits is
	 * undefined. Mask them to make sure that the musb driver core will
	 * never see them set
	 */
	musb->int_usb &= MUSB_INTR_SUSPEND | MUSB_INTR_RESUME |
	    MUSB_INTR_RESET | MUSB_INTR_SOF;

	if (musb->int_usb || musb->int_tx || musb->int_rx)
	    retval = musb_interrupt(musb);

	spin_unlock_irqrestore(&musb->lock, flags);

	return retval;
    }

Here the glue layer mostly has to read the relevant hardware registers
and pass their values on to the controller driver which will handle the
actual event that triggered the IRQ.

The interrupt handler critical section is protected by the
:c:func:`spin_lock_irqsave` and counterpart :c:func:`spin_unlock_irqrestore`
functions (line 7 and 24 respectively), which prevent the interrupt
handler code to be run by two different threads at the same time.

Then the relevant interrupt registers are read (line 9 to 11):

-  ``MUSB_INTRUSB``: indicates which USB interrupts are currently active,

-  ``MUSB_INTRTX``: indicates which of the interrupts for TX endpoints are
   currently active,

-  ``MUSB_INTRRX``: indicates which of the interrupts for TX endpoints are
   currently active.

Note that :c:func:`musb_readb` is used to read 8-bit registers at most, while
:c:func:`musb_readw` allows us to read at most 16-bit registers. There are
other functions that can be used depending on the size of your device
registers. See ``musb_io.h`` for more information.

Instruction on line 18 is another quirk specific to the JZ4740 USB
device controller, which will be discussed later in :ref:`musb-dev-quirks`.

The glue layer still needs to register the IRQ handler though. Remember
the instruction on line 14 of the init function::

    static int jz4740_musb_init(struct musb *musb)
    {
	musb->isr = jz4740_musb_interrupt;

	return 0;
    }

This instruction sets a pointer to the glue layer IRQ handler function,
in order for the controller hardware to call the handler back when an
IRQ comes from the controller hardware. The interrupt handler is now
implemented and registered.

.. _musb-dev-platform-data:

Device Platform Data
====================

In order to write an MUSB glue layer, you need to have some data
describing the hardware capabilities of your controller hardware, which
is called the platform data.

Platform data is specific to your hardware, though it may cover a broad
range of devices, and is generally found somewhere in the ``arch/``
directory, depending on your device architecture.

For instance, platform data for the JZ4740 SoC is found in
``arch/mips/jz4740/platform.c``. In the ``platform.c`` file each device of the
JZ4740 SoC is described through a set of structures.

Here is the part of ``arch/mips/jz4740/platform.c`` that covers the USB
Device Controller (UDC):

   .. code-block:: c
    :emphasize-lines: 2,7,14-17,21,22,25,26,28,29

    /* USB Device Controller */
    struct platform_device jz4740_udc_xceiv_device = {
	.name = "usb_phy_gen_xceiv",
	.id   = 0,
    };

    static struct resource jz4740_udc_resources[] = {
	[0] = {
	    .start = JZ4740_UDC_BASE_ADDR,
	    .end   = JZ4740_UDC_BASE_ADDR + 0x10000 - 1,
	    .flags = IORESOURCE_MEM,
	},
	[1] = {
	    .start = JZ4740_IRQ_UDC,
	    .end   = JZ4740_IRQ_UDC,
	    .flags = IORESOURCE_IRQ,
	    .name  = "mc",
	},
    };

    struct platform_device jz4740_udc_device = {
	.name = "musb-jz4740",
	.id   = -1,
	.dev  = {
	    .dma_mask          = &jz4740_udc_device.dev.coherent_dma_mask,
	    .coherent_dma_mask = DMA_BIT_MASK(32),
	},
	.num_resources = ARRAY_SIZE(jz4740_udc_resources),
	.resource      = jz4740_udc_resources,
    };

The ``jz4740_udc_xceiv_device`` platform device structure (line 2)
describes the UDC transceiver with a name and id number.

At the time of this writing, note that ``usb_phy_gen_xceiv`` is the
specific name to be used for all transceivers that are either built-in
with reference USB IP or autonomous and doesn't require any PHY
programming. You will need to set ``CONFIG_NOP_USB_XCEIV=y`` in the
kernel configuration to make use of the corresponding transceiver
driver. The id field could be set to -1 (equivalent to
``PLATFORM_DEVID_NONE``), -2 (equivalent to ``PLATFORM_DEVID_AUTO``) or
start with 0 for the first device of this kind if we want a specific id
number.

The ``jz4740_udc_resources`` resource structure (line 7) defines the UDC
registers base addresses.

The first array (line 9 to 11) defines the UDC registers base memory
addresses: start points to the first register memory address, end points
to the last register memory address and the flags member defines the
type of resource we are dealing with. So ``IORESOURCE_MEM`` is used to
define the registers memory addresses. The second array (line 14 to 17)
defines the UDC IRQ registers addresses. Since there is only one IRQ
register available for the JZ4740 UDC, start and end point at the same
address. The ``IORESOURCE_IRQ`` flag tells that we are dealing with IRQ
resources, and the name ``mc`` is in fact hard-coded in the MUSB core in
order for the controller driver to retrieve this IRQ resource by
querying it by its name.

Finally, the ``jz4740_udc_device`` platform device structure (line 21)
describes the UDC itself.

The ``musb-jz4740`` name (line 22) defines the MUSB driver that is used
for this device; remember this is in fact the name that we used in the
``jz4740_driver`` platform driver structure in :ref:`musb-basics`.
The id field (line 23) is set to -1 (equivalent to ``PLATFORM_DEVID_NONE``)
since we do not need an id for the device: the MUSB controller driver was
already set to allocate an automatic id in :ref:`musb-basics`. In the dev field
we care for DMA related information here. The ``dma_mask`` field (line 25)
defines the width of the DMA mask that is going to be used, and
``coherent_dma_mask`` (line 26) has the same purpose but for the
``alloc_coherent`` DMA mappings: in both cases we are using a 32 bits mask.
Then the resource field (line 29) is simply a pointer to the resource
structure defined before, while the ``num_resources`` field (line 28) keeps
track of the number of arrays defined in the resource structure (in this
case there were two resource arrays defined before).

With this quick overview of the UDC platform data at the ``arch/`` level now
done, let's get back to the MUSB glue layer specific platform data in
``drivers/usb/musb/jz4740.c``:

   .. code-block:: c
    :emphasize-lines: 3,5,7-9,11

    static struct musb_hdrc_config jz4740_musb_config = {
	/* Silicon does not implement USB OTG. */
	.multipoint = 0,
	/* Max EPs scanned, driver will decide which EP can be used. */
	.num_eps    = 4,
	/* RAMbits needed to configure EPs from table */
	.ram_bits   = 9,
	.fifo_cfg = jz4740_musb_fifo_cfg,
	.fifo_cfg_size = ARRAY_SIZE(jz4740_musb_fifo_cfg),
    };

    static struct musb_hdrc_platform_data jz4740_musb_platform_data = {
	.mode   = MUSB_PERIPHERAL,
	.config = &jz4740_musb_config,
    };

First the glue layer configures some aspects of the controller driver
operation related to the controller hardware specifics. This is done
through the ``jz4740_musb_config`` :c:type:`musb_hdrc_config` structure.

Defining the OTG capability of the controller hardware, the multipoint
member (line 3) is set to 0 (equivalent to false) since the JZ4740 UDC
is not OTG compatible. Then ``num_eps`` (line 5) defines the number of USB
endpoints of the controller hardware, including endpoint 0: here we have
3 endpoints + endpoint 0. Next is ``ram_bits`` (line 7) which is the width
of the RAM address bus for the MUSB controller hardware. This
information is needed when the controller driver cannot automatically
configure endpoints by reading the relevant controller hardware
registers. This issue will be discussed when we get to device quirks in
:ref:`musb-dev-quirks`. Last two fields (line 8 and 9) are also
about device quirks: ``fifo_cfg`` points to the USB endpoints configuration
table and ``fifo_cfg_size`` keeps track of the size of the number of
entries in that configuration table. More on that later in
:ref:`musb-dev-quirks`.

Then this configuration is embedded inside ``jz4740_musb_platform_data``
:c:type:`musb_hdrc_platform_data` structure (line 11): config is a pointer to
the configuration structure itself, and mode tells the controller driver
if the controller hardware may be used as ``MUSB_HOST`` only,
``MUSB_PERIPHERAL`` only or ``MUSB_OTG`` which is a dual mode.

Remember that ``jz4740_musb_platform_data`` is then used to convey
platform data information as we have seen in the probe function in
:ref:`musb-basics`.

.. _musb-dev-quirks:

Device Quirks
=============

Completing the platform data specific to your device, you may also need
to write some code in the glue layer to work around some device specific
limitations. These quirks may be due to some hardware bugs, or simply be
the result of an incomplete implementation of the USB On-the-Go
specification.

The JZ4740 UDC exhibits such quirks, some of which we will discuss here
for the sake of insight even though these might not be found in the
controller hardware you are working on.

Let's get back to the init function first:

   .. code-block:: c
    :emphasize-lines: 12

    static int jz4740_musb_init(struct musb *musb)
    {
	musb->xceiv = usb_get_phy(USB_PHY_TYPE_USB2);
	if (!musb->xceiv) {
	    pr_err("HS UDC: no transceiver configured\n");
	    return -ENODEV;
	}

	/* Silicon does not implement ConfigData register.
	 * Set dyn_fifo to avoid reading EP config from hardware.
	 */
	musb->dyn_fifo = true;

	musb->isr = jz4740_musb_interrupt;

	return 0;
    }

Instruction on line 12 helps the MUSB controller driver to work around
the fact that the controller hardware is missing registers that are used
for USB endpoints configuration.

Without these registers, the controller driver is unable to read the
endpoints configuration from the hardware, so we use line 12 instruction
to bypass reading the configuration from silicon, and rely on a
hard-coded table that describes the endpoints configuration instead::

    static struct musb_fifo_cfg jz4740_musb_fifo_cfg[] = {
	{ .hw_ep_num = 1, .style = FIFO_TX, .maxpacket = 512, },
	{ .hw_ep_num = 1, .style = FIFO_RX, .maxpacket = 512, },
	{ .hw_ep_num = 2, .style = FIFO_TX, .maxpacket = 64, },
    };

Looking at the configuration table above, we see that each endpoints is
described by three fields: ``hw_ep_num`` is the endpoint number, style is
its direction (either ``FIFO_TX`` for the controller driver to send packets
in the controller hardware, or ``FIFO_RX`` to receive packets from
hardware), and maxpacket defines the maximum size of each data packet
that can be transmitted over that endpoint. Reading from the table, the
controller driver knows that endpoint 1 can be used to send and receive
USB data packets of 512 bytes at once (this is in fact a bulk in/out
endpoint), and endpoint 2 can be used to send data packets of 64 bytes
at once (this is in fact an interrupt endpoint).

Note that there is no information about endpoint 0 here: that one is
implemented by default in every silicon design, with a predefined
configuration according to the USB specification. For more examples of
endpoint configuration tables, see ``musb_core.c``.

Let's now get back to the interrupt handler function:

   .. code-block:: c
    :emphasize-lines: 18-19

    static irqreturn_t jz4740_musb_interrupt(int irq, void *__hci)
    {
	unsigned long   flags;
	irqreturn_t     retval = IRQ_NONE;
	struct musb     *musb = __hci;

	spin_lock_irqsave(&musb->lock, flags);

	musb->int_usb = musb_readb(musb->mregs, MUSB_INTRUSB);
	musb->int_tx = musb_readw(musb->mregs, MUSB_INTRTX);
	musb->int_rx = musb_readw(musb->mregs, MUSB_INTRRX);

	/*
	 * The controller is gadget only, the state of the host mode IRQ bits is
	 * undefined. Mask them to make sure that the musb driver core will
	 * never see them set
	 */
	musb->int_usb &= MUSB_INTR_SUSPEND | MUSB_INTR_RESUME |
	    MUSB_INTR_RESET | MUSB_INTR_SOF;

	if (musb->int_usb || musb->int_tx || musb->int_rx)
	    retval = musb_interrupt(musb);

	spin_unlock_irqrestore(&musb->lock, flags);

	return retval;
    }

Instruction on line 18 above is a way for the controller driver to work
around the fact that some interrupt bits used for USB host mode
operation are missing in the ``MUSB_INTRUSB`` register, thus left in an
undefined hardware state, since this MUSB controller hardware is used in
peripheral mode only. As a consequence, the glue layer masks these
missing bits out to avoid parasite interrupts by doing a logical AND
operation between the value read from ``MUSB_INTRUSB`` and the bits that
are actually implemented in the register.

These are only a couple of the quirks found in the JZ4740 USB device
controller. Some others were directly addressed in the MUSB core since
the fixes were generic enough to provide a better handling of the issues
for others controller hardware eventually.

Conclusion
==========

Writing a Linux MUSB glue layer should be a more accessible task, as
this documentation tries to show the ins and outs of this exercise.

The JZ4740 USB device controller being fairly simple, I hope its glue
layer serves as a good example for the curious mind. Used with the
current MUSB glue layers, this documentation should provide enough
guidance to get started; should anything gets out of hand, the linux-usb
mailing list archive is another helpful resource to browse through.

Acknowledgements
================

Many thanks to Lars-Peter Clausen and Maarten ter Huurne for answering
my questions while I was writing the JZ4740 glue layer and for helping
me out getting the code in good shape.

I would also like to thank the Qi-Hardware community at large for its
cheerful guidance and support.

Resources
=========

USB Home Page: https://www.usb.org

linux-usb Mailing List Archives: https://marc.info/?l=linux-usb

USB On-the-Go Basics:
https://www.maximintegrated.com/app-notes/index.mvp/id/1822

:ref:`Writing USB Device Drivers <writing-usb-driver>`

Texas Instruments USB Configuration Wiki Page:
http://processors.wiki.ti.com/index.php/Usbgeneralpage


code:
.. _writing-usb-driver:

==========================
Writing USB Device Drivers
==========================

:Author: Greg Kroah-Hartman

Introduction
============

The Linux USB subsystem has grown from supporting only two different
types of devices in the 2.2.7 kernel (mice and keyboards), to over 20
different types of devices in the 2.4 kernel. Linux currently supports
almost all USB class devices (standard types of devices like keyboards,
mice, modems, printers and speakers) and an ever-growing number of
vendor-specific devices (such as USB to serial converters, digital
cameras, Ethernet devices and MP3 players). For a full list of the
different USB devices currently supported, see Resources.

The remaining kinds of USB devices that do not have support on Linux are
almost all vendor-specific devices. Each vendor decides to implement a
custom protocol to talk to their device, so a custom driver usually
needs to be created. Some vendors are open with their USB protocols and
help with the creation of Linux drivers, while others do not publish
them, and developers are forced to reverse-engineer. See Resources for
some links to handy reverse-engineering tools.

Because each different protocol causes a new driver to be created, I
have written a generic USB driver skeleton, modelled after the
pci-skeleton.c file in the kernel source tree upon which many PCI
network drivers have been based. This USB skeleton can be found at
drivers/usb/usb-skeleton.c in the kernel source tree. In this article I
will walk through the basics of the skeleton driver, explaining the
different pieces and what needs to be done to customize it to your
specific device.

Linux USB Basics
================

If you are going to write a Linux USB driver, please become familiar
with the USB protocol specification. It can be found, along with many
other useful documents, at the USB home page (see Resources). An
excellent introduction to the Linux USB subsystem can be found at the
USB Working Devices List (see Resources). It explains how the Linux USB
subsystem is structured and introduces the reader to the concept of USB
urbs (USB Request Blocks), which are essential to USB drivers.

The first thing a Linux USB driver needs to do is register itself with
the Linux USB subsystem, giving it some information about which devices
the driver supports and which functions to call when a device supported
by the driver is inserted or removed from the system. All of this
information is passed to the USB subsystem in the :c:type:`usb_driver`
structure. The skeleton driver declares a :c:type:`usb_driver` as::

    static struct usb_driver skel_driver = {
	    .name        = "skeleton",
	    .probe       = skel_probe,
	    .disconnect  = skel_disconnect,
	    .suspend     = skel_suspend,
	    .resume      = skel_resume,
	    .pre_reset   = skel_pre_reset,
	    .post_reset  = skel_post_reset,
	    .id_table    = skel_table,
	    .supports_autosuspend = 1,
    };

The variable name is a string that describes the driver. It is used in
informational messages printed to the system log. The probe and
disconnect function pointers are called when a device that matches the
information provided in the ``id_table`` variable is either seen or
removed.

The fops and minor variables are optional. Most USB drivers hook into
another kernel subsystem, such as the SCSI, network or TTY subsystem.
These types of drivers register themselves with the other kernel
subsystem, and any user-space interactions are provided through that
interface. But for drivers that do not have a matching kernel subsystem,
such as MP3 players or scanners, a method of interacting with user space
is needed. The USB subsystem provides a way to register a minor device
number and a set of :c:type:`file_operations` function pointers that enable
this user-space interaction. The skeleton driver needs this kind of
interface, so it provides a minor starting number and a pointer to its
:c:type:`file_operations` functions.

The USB driver is then registered with a call to usb_register(),
usually in the driver's init function, as shown here::

    static int __init usb_skel_init(void)
    {
	    int result;

	    /* register this driver with the USB subsystem */
	    result = usb_register(&skel_driver);
	    if (result < 0) {
		    pr_err("usb_register failed for the %s driver. Error number %d\n",
		           skel_driver.name, result);
		    return -1;
	    }

	    return 0;
    }
    module_init(usb_skel_init);

When the driver is unloaded from the system, it needs to deregister
itself with the USB subsystem. This is done with usb_deregister()
function::

    static void __exit usb_skel_exit(void)
    {
	    /* deregister this driver with the USB subsystem */
	    usb_deregister(&skel_driver);
    }
    module_exit(usb_skel_exit);

To enable the linux-hotplug system to load the driver automatically when
the device is plugged in, you need to create a ``MODULE_DEVICE_TABLE``.
The following code tells the hotplug scripts that this module supports a
single device with a specific vendor and product ID::

    /* table of devices that work with this driver */
    static struct usb_device_id skel_table [] = {
	    { USB_DEVICE(USB_SKEL_VENDOR_ID, USB_SKEL_PRODUCT_ID) },
	    { }                      /* Terminating entry */
    };
    MODULE_DEVICE_TABLE (usb, skel_table);

There are other macros that can be used in describing a struct
:c:type:`usb_device_id` for drivers that support a whole class of USB
drivers. See :ref:`usb.h <usb_header>` for more information on this.

Device operation
================

When a device is plugged into the USB bus that matches the device ID
pattern that your driver registered with the USB core, the probe
function is called. The :c:type:`usb_device` structure, interface number and
the interface ID are passed to the function::

    static int skel_probe(struct usb_interface *interface,
	const struct usb_device_id *id)

The driver now needs to verify that this device is actually one that it
can accept. If so, it returns 0. If not, or if any error occurs during
initialization, an errorcode (such as ``-ENOMEM`` or ``-ENODEV``) is
returned from the probe function.

In the skeleton driver, we determine what end points are marked as
bulk-in and bulk-out. We create buffers to hold the data that will be
sent and received from the device, and a USB urb to write data to the
device is initialized.

Conversely, when the device is removed from the USB bus, the disconnect
function is called with the device pointer. The driver needs to clean
any private data that has been allocated at this time and to shut down
any pending urbs that are in the USB system.

Now that the device is plugged into the system and the driver is bound
to the device, any of the functions in the :c:type:`file_operations` structure
that were passed to the USB subsystem will be called from a user program
trying to talk to the device. The first function called will be open, as
the program tries to open the device for I/O. We increment our private
usage count and save a pointer to our internal structure in the file
structure. This is done so that future calls to file operations will
enable the driver to determine which device the user is addressing. All
of this is done with the following code::

    /* increment our usage count for the device */
    kref_get(&dev->kref);

    /* save our object in the file's private structure */
    file->private_data = dev;

After the open function is called, the read and write functions are
called to receive and send data to the device. In the ``skel_write``
function, we receive a pointer to some data that the user wants to send
to the device and the size of the data. The function determines how much
data it can send to the device based on the size of the write urb it has
created (this size depends on the size of the bulk out end point that
the device has). Then it copies the data from user space to kernel
space, points the urb to the data and submits the urb to the USB
subsystem. This can be seen in the following code::

    /* we can only write as much as 1 urb will hold */
    size_t writesize = min_t(size_t, count, MAX_TRANSFER);

    /* copy the data from user space into our urb */
    copy_from_user(buf, user_buffer, writesize);

    /* set up our urb */
    usb_fill_bulk_urb(urb,
		      dev->udev,
		      usb_sndbulkpipe(dev->udev, dev->bulk_out_endpointAddr),
		      buf,
		      writesize,
		      skel_write_bulk_callback,
		      dev);

    /* send the data out the bulk port */
    retval = usb_submit_urb(urb, GFP_KERNEL);
    if (retval) {
	    dev_err(&dev->interface->dev,
                "%s - failed submitting write urb, error %d\n",
                __func__, retval);
    }

When the write urb is filled up with the proper information using the
:c:func:`usb_fill_bulk_urb` function, we point the urb's completion callback
to call our own ``skel_write_bulk_callback`` function. This function is
called when the urb is finished by the USB subsystem. The callback
function is called in interrupt context, so caution must be taken not to
do very much processing at that time. Our implementation of
``skel_write_bulk_callback`` merely reports if the urb was completed
successfully or not and then returns.

The read function works a bit differently from the write function in
that we do not use an urb to transfer data from the device to the
driver. Instead we call the :c:func:`usb_bulk_msg` function, which can be used
to send or receive data from a device without having to create urbs and
handle urb completion callback functions. We call the :c:func:`usb_bulk_msg`
function, giving it a buffer into which to place any data received from
the device and a timeout value. If the timeout period expires without
receiving any data from the device, the function will fail and return an
error message. This can be shown with the following code::

    /* do an immediate bulk read to get data from the device */
    retval = usb_bulk_msg (skel->dev,
			   usb_rcvbulkpipe (skel->dev,
			   skel->bulk_in_endpointAddr),
			   skel->bulk_in_buffer,
			   skel->bulk_in_size,
			   &count, 5000);
    /* if the read was successful, copy the data to user space */
    if (!retval) {
	    if (copy_to_user (buffer, skel->bulk_in_buffer, count))
		    retval = -EFAULT;
	    else
		    retval = count;
    }

The :c:func:`usb_bulk_msg` function can be very useful for doing single reads
or writes to a device; however, if you need to read or write constantly to
a device, it is recommended to set up your own urbs and submit them to
the USB subsystem.

When the user program releases the file handle that it has been using to
talk to the device, the release function in the driver is called. In
this function we decrement our private usage count and wait for possible
pending writes::

    /* decrement our usage count for the device */
    --skel->open_count;

One of the more difficult problems that USB drivers must be able to
handle smoothly is the fact that the USB device may be removed from the
system at any point in time, even if a program is currently talking to
it. It needs to be able to shut down any current reads and writes and
notify the user-space programs that the device is no longer there. The
following code (function ``skel_delete``) is an example of how to do
this::

    static inline void skel_delete (struct usb_skel *dev)
    {
	kfree (dev->bulk_in_buffer);
	if (dev->bulk_out_buffer != NULL)
	    usb_free_coherent (dev->udev, dev->bulk_out_size,
		dev->bulk_out_buffer,
		dev->write_urb->transfer_dma);
	usb_free_urb (dev->write_urb);
	kfree (dev);
    }

If a program currently has an open handle to the device, we reset the
flag ``device_present``. For every read, write, release and other
functions that expect a device to be present, the driver first checks
this flag to see if the device is still present. If not, it releases
that the device has disappeared, and a ``-ENODEV`` error is returned to the
user-space program. When the release function is eventually called, it
determines if there is no device and if not, it does the cleanup that
the ``skel_disconnect`` function normally does if there are no open files
on the device (see Listing 5).

Isochronous Data
================

This usb-skeleton driver does not have any examples of interrupt or
isochronous data being sent to or from the device. Interrupt data is
sent almost exactly as bulk data is, with a few minor exceptions.
Isochronous data works differently with continuous streams of data being
sent to or from the device. The audio and video camera drivers are very
good examples of drivers that handle isochronous data and will be useful
if you also need to do this.

Conclusion
==========

Writing Linux USB device drivers is not a difficult task as the
usb-skeleton driver shows. This driver, combined with the other current
USB drivers, should provide enough examples to help a beginning author
create a working driver in a minimal amount of time. The linux-usb-devel
mailing list archives also contain a lot of helpful information.

Resources
=========

The Linux USB Project:
http://www.linux-usb.org/

Linux Hotplug Project:
http://linux-hotplug.sourceforge.net/

linux-usb Mailing List Archives:
https://lore.kernel.org/linux-usb/

Programming Guide for Linux USB Device Drivers:
https://lmu.web.psi.ch/docu/manuals/software_manuals/linux_sl/usb_linux_programming_guide.pdf

USB Home Page: https://www.usb.org

repo_name: angery/linux


code:
.. SPDX-License-Identifier: GPL-2.0-only
.. include:: <isonum.txt>

=====================
VFIO Mediated devices
=====================

:Copyright: |copy| 2016, NVIDIA CORPORATION. All rights reserved.
:Author: Neo Jia <cjia@nvidia.com>
:Author: Kirti Wankhede <kwankhede@nvidia.com>

Virtual Function I/O (VFIO) Mediated devices[1]
===============================================

The number of use cases for virtualizing DMA devices that do not have built-in
SR_IOV capability is increasing. Previously, to virtualize such devices,
developers had to create their own management interfaces and APIs, and then
integrate them with user space software. To simplify integration with user space
software, we have identified common requirements and a unified management
interface for such devices.

The VFIO driver framework provides unified APIs for direct device access. It is
an IOMMU/device-agnostic framework for exposing direct device access to user
space in a secure, IOMMU-protected environment. This framework is used for
multiple devices, such as GPUs, network adapters, and compute accelerators. With
direct device access, virtual machines or user space applications have direct
access to the physical device. This framework is reused for mediated devices.

The mediated core driver provides a common interface for mediated device
management that can be used by drivers of different devices. This module
provides a generic interface to perform these operations:

* Create and destroy a mediated device
* Add a mediated device to and remove it from a mediated bus driver
* Add a mediated device to and remove it from an IOMMU group

The mediated core driver also provides an interface to register a bus driver.
For example, the mediated VFIO mdev driver is designed for mediated devices and
supports VFIO APIs. The mediated bus driver adds a mediated device to and
removes it from a VFIO group.

The following high-level block diagram shows the main components and interfaces
in the VFIO mediated driver framework. The diagram shows NVIDIA, Intel, and IBM
devices as examples, as these devices are the first devices to use this module::

     +---------------+
     |               |
     | +-----------+ |  mdev_register_driver() +--------------+
     | |           | +<------------------------+              |
     | |  mdev     | |                         |              |
     | |  bus      | +------------------------>+ vfio_mdev.ko |<-> VFIO user
     | |  driver   | |     probe()/remove()    |              |    APIs
     | |           | |                         +--------------+
     | +-----------+ |
     |               |
     |  MDEV CORE    |
     |   MODULE      |
     |   mdev.ko     |
     | +-----------+ |  mdev_register_parent() +--------------+
     | |           | +<------------------------+              |
     | |           | |                         |  nvidia.ko   |<-> physical
     | |           | +------------------------>+              |    device
     | |           | |        callbacks        +--------------+
     | | Physical  | |
     | |  device   | |  mdev_register_parent() +--------------+
     | | interface | |<------------------------+              |
     | |           | |                         |  i915.ko     |<-> physical
     | |           | +------------------------>+              |    device
     | |           | |        callbacks        +--------------+
     | |           | |
     | |           | |  mdev_register_parent() +--------------+
     | |           | +<------------------------+              |
     | |           | |                         | ccw_device.ko|<-> physical
     | |           | +------------------------>+              |    device
     | |           | |        callbacks        +--------------+
     | +-----------+ |
     +---------------+

Registration Interfaces
=======================

The mediated core driver provides the following types of registration
interfaces:

* Registration interface for a mediated bus driver
* Physical device driver interface

Registration Interface for a Mediated Bus Driver
------------------------------------------------

The registration interface for a mediated device driver provides the following
structure to represent a mediated device's driver::

     /*
      * struct mdev_driver [2] - Mediated device's driver
      * @probe: called when new device created
      * @remove: called when device removed
      * @driver: device driver structure
      */
     struct mdev_driver {
	     int  (*probe)  (struct mdev_device *dev);
	     void (*remove) (struct mdev_device *dev);
	     unsigned int (*get_available)(struct mdev_type *mtype);
	     ssize_t (*show_description)(struct mdev_type *mtype, char *buf);
	     struct device_driver    driver;
     };

A mediated bus driver for mdev should use this structure in the function calls
to register and unregister itself with the core driver:

* Register::

    int mdev_register_driver(struct mdev_driver *drv);

* Unregister::

    void mdev_unregister_driver(struct mdev_driver *drv);

The mediated bus driver's probe function should create a vfio_device on top of
the mdev_device and connect it to an appropriate implementation of
vfio_device_ops.

When a driver wants to add the GUID creation sysfs to an existing device it has
probe'd to then it should call::

    int mdev_register_parent(struct mdev_parent *parent, struct device *dev,
			struct mdev_driver *mdev_driver);

This will provide the 'mdev_supported_types/XX/create' files which can then be
used to trigger the creation of a mdev_device. The created mdev_device will be
attached to the specified driver.

When the driver needs to remove itself it calls::

    void mdev_unregister_parent(struct mdev_parent *parent);

Which will unbind and destroy all the created mdevs and remove the sysfs files.

Mediated Device Management Interface Through sysfs
==================================================

The management interface through sysfs enables user space software, such as
libvirt, to query and configure mediated devices in a hardware-agnostic fashion.
This management interface provides flexibility to the underlying physical
device's driver to support features such as:

* Mediated device hot plug
* Multiple mediated devices in a single virtual machine
* Multiple mediated devices from different physical devices

Links in the mdev_bus Class Directory
-------------------------------------
The /sys/class/mdev_bus/ directory contains links to devices that are registered
with the mdev core driver.

Directories and files under the sysfs for Each Physical Device
--------------------------------------------------------------

::

  |- [parent physical device]
  |--- Vendor-specific-attributes [optional]
  |--- [mdev_supported_types]
  |     |--- [<type-id>]
  |     |   |--- create
  |     |   |--- name
  |     |   |--- available_instances
  |     |   |--- device_api
  |     |   |--- description
  |     |   |--- [devices]
  |     |--- [<type-id>]
  |     |   |--- create
  |     |   |--- name
  |     |   |--- available_instances
  |     |   |--- device_api
  |     |   |--- description
  |     |   |--- [devices]
  |     |--- [<type-id>]
  |          |--- create
  |          |--- name
  |          |--- available_instances
  |          |--- device_api
  |          |--- description
  |          |--- [devices]

* [mdev_supported_types]

  The list of currently supported mediated device types and their details.

  [<type-id>], device_api, and available_instances are mandatory attributes
  that should be provided by vendor driver.

* [<type-id>]

  The [<type-id>] name is created by adding the device driver string as a prefix
  to the string provided by the vendor driver. This format of this name is as
  follows::

	sprintf(buf, "%s-%s", dev_driver_string(parent->dev), group->name);

* device_api

  This attribute shows which device API is being created, for example,
  "vfio-pci" for a PCI device.

* available_instances

  This attribute shows the number of devices of type <type-id> that can be
  created.

* [device]

  This directory contains links to the devices of type <type-id> that have been
  created.

* name

  This attribute shows a human readable name.

* description

  This attribute can show brief features/description of the type. This is an
  optional attribute.

Directories and Files Under the sysfs for Each mdev Device
----------------------------------------------------------

::

  |- [parent phy device]
  |--- [$MDEV_UUID]
         |--- remove
         |--- mdev_type {link to its type}
         |--- vendor-specific-attributes [optional]

* remove (write only)

Writing '1' to the 'remove' file destroys the mdev device. The vendor driver can
fail the remove() callback if that device is active and the vendor driver
doesn't support hot unplug.

Example::

	# echo 1 > /sys/bus/mdev/devices/$mdev_UUID/remove

Mediated device Hot plug
------------------------

Mediated devices can be created and assigned at runtime. The procedure to hot
plug a mediated device is the same as the procedure to hot plug a PCI device.

Translation APIs for Mediated Devices
=====================================

The following APIs are provided for translating user pfn to host pfn in a VFIO
driver::

	int vfio_pin_pages(struct vfio_device *device, dma_addr_t iova,
				  int npage, int prot, struct page **pages);

	void vfio_unpin_pages(struct vfio_device *device, dma_addr_t iova,
				    int npage);

These functions call back into the back-end IOMMU module by using the pin_pages
and unpin_pages callbacks of the struct vfio_iommu_driver_ops[4]. Currently
these callbacks are supported in the TYPE1 IOMMU module. To enable them for
other IOMMU backend modules, such as PPC64 sPAPR module, they need to provide
these two callback functions.

Using the Sample Code
=====================

mtty.c in samples/vfio-mdev/ directory is a sample driver program to
demonstrate how to use the mediated device framework.

The sample driver creates an mdev device that simulates a serial port over a PCI
card.

1. Build and load the mtty.ko module.

   This step creates a dummy device, /sys/devices/virtual/mtty/mtty/

   Files in this device directory in sysfs are similar to the following::

     # tree /sys/devices/virtual/mtty/mtty/
        /sys/devices/virtual/mtty/mtty/
        |-- mdev_supported_types
        |   |-- mtty-1
        |   |   |-- available_instances
        |   |   |-- create
        |   |   |-- device_api
        |   |   |-- devices
        |   |   `-- name
        |   `-- mtty-2
        |       |-- available_instances
        |       |-- create
        |       |-- device_api
        |       |-- devices
        |       `-- name
        |-- mtty_dev
        |   `-- sample_mtty_dev
        |-- power
        |   |-- autosuspend_delay_ms
        |   |-- control
        |   |-- runtime_active_time
        |   |-- runtime_status
        |   `-- runtime_suspended_time
        |-- subsystem -> ../../../../class/mtty
        `-- uevent

2. Create a mediated device by using the dummy device that you created in the
   previous step::

     # echo "83b8f4f2-509f-382f-3c1e-e6bfe0fa1001" >	\
              /sys/devices/virtual/mtty/mtty/mdev_supported_types/mtty-2/create

3. Add parameters to qemu-kvm::

     -device vfio-pci,\
      sysfsdev=/sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1001

4. Boot the VM.

   In the Linux guest VM, with no hardware on the host, the device appears
   as  follows::

     # lspci -s 00:05.0 -xxvv
     00:05.0 Serial controller: Device 4348:3253 (rev 10) (prog-if 02 [16550])
             Subsystem: Device 4348:3253
             Physical Slot: 5
             Control: I/O+ Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
     Stepping- SERR- FastB2B- DisINTx-
             Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
     <TAbort- <MAbort- >SERR- <PERR- INTx-
             Interrupt: pin A routed to IRQ 10
             Region 0: I/O ports at c150 [size=8]
             Region 1: I/O ports at c158 [size=8]
             Kernel driver in use: serial
     00: 48 43 53 32 01 00 00 02 10 02 00 07 00 00 00 00
     10: 51 c1 00 00 59 c1 00 00 00 00 00 00 00 00 00 00
     20: 00 00 00 00 00 00 00 00 00 00 00 00 48 43 53 32
     30: 00 00 00 00 00 00 00 00 00 00 00 00 0a 01 00 00

     In the Linux guest VM, dmesg output for the device is as follows:

     serial 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ 10
     0000:00:05.0: ttyS1 at I/O 0xc150 (irq = 10) is a 16550A
     0000:00:05.0: ttyS2 at I/O 0xc158 (irq = 10) is a 16550A

5. In the Linux guest VM, check the serial ports::

     # setserial -g /dev/ttyS*
     /dev/ttyS0, UART: 16550A, Port: 0x03f8, IRQ: 4
     /dev/ttyS1, UART: 16550A, Port: 0xc150, IRQ: 10
     /dev/ttyS2, UART: 16550A, Port: 0xc158, IRQ: 10

6. Using minicom or any terminal emulation program, open port /dev/ttyS1 or
   /dev/ttyS2 with hardware flow control disabled.

7. Type data on the minicom terminal or send data to the terminal emulation
   program and read the data.

   Data is loop backed from hosts mtty driver.

8. Destroy the mediated device that you created::

     # echo 1 > /sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1001/remove

References
==========

1. See Documentation/driver-api/vfio.rst for more information on VFIO.
2. struct mdev_driver in include/linux/mdev.h
3. struct mdev_parent_ops in include/linux/mdev.h
4. struct vfio_iommu_driver_ops in include/linux/vfio.h


code:
.. SPDX-License-Identifier: GPL-2.0

Acceptance criteria for vfio-pci device specific driver variants
================================================================

Overview
--------
The vfio-pci driver exists as a device agnostic driver using the
system IOMMU and relying on the robustness of platform fault
handling to provide isolated device access to userspace.  While the
vfio-pci driver does include some device specific support, further
extensions for yet more advanced device specific features are not
sustainable.  The vfio-pci driver has therefore split out
vfio-pci-core as a library that may be reused to implement features
requiring device specific knowledge, ex. saving and loading device
state for the purposes of supporting migration.

In support of such features, it's expected that some device specific
variants may interact with parent devices (ex. SR-IOV PF in support of
a user assigned VF) or other extensions that may not be otherwise
accessible via the vfio-pci base driver.  Authors of such drivers
should be diligent not to create exploitable interfaces via these
interactions or allow unchecked userspace data to have an effect
beyond the scope of the assigned device.

New driver submissions are therefore requested to have approval via
sign-off/ack/review/etc for any interactions with parent drivers.
Additionally, drivers should make an attempt to provide sufficient
documentation for reviewers to understand the device specific
extensions, for example in the case of migration data, how is the
device state composed and consumed, which portions are not otherwise
available to the user via vfio-pci, what safeguards exist to validate
the data, etc.  To that extent, authors should additionally expect to
require reviews from at least one of the listed reviewers, in addition
to the overall vfio maintainer.


code:
==================================
VFIO - "Virtual Function I/O" [1]_
==================================

Many modern systems now provide DMA and interrupt remapping facilities
to help ensure I/O devices behave within the boundaries they've been
allotted.  This includes x86 hardware with AMD-Vi and Intel VT-d,
POWER systems with Partitionable Endpoints (PEs) and embedded PowerPC
systems such as Freescale PAMU.  The VFIO driver is an IOMMU/device
agnostic framework for exposing direct device access to userspace, in
a secure, IOMMU protected environment.  In other words, this allows
safe [2]_, non-privileged, userspace drivers.

Why do we want that?  Virtual machines often make use of direct device
access ("device assignment") when configured for the highest possible
I/O performance.  From a device and host perspective, this simply
turns the VM into a userspace driver, with the benefits of
significantly reduced latency, higher bandwidth, and direct use of
bare-metal device drivers [3]_.

Some applications, particularly in the high performance computing
field, also benefit from low-overhead, direct device access from
userspace.  Examples include network adapters (often non-TCP/IP based)
and compute accelerators.  Prior to VFIO, these drivers had to either
go through the full development cycle to become proper upstream
driver, be maintained out of tree, or make use of the UIO framework,
which has no notion of IOMMU protection, limited interrupt support,
and requires root privileges to access things like PCI configuration
space.

The VFIO driver framework intends to unify these, replacing both the
KVM PCI specific device assignment code as well as provide a more
secure, more featureful userspace driver environment than UIO.

Groups, Devices, and IOMMUs
---------------------------

Devices are the main target of any I/O driver.  Devices typically
create a programming interface made up of I/O access, interrupts,
and DMA.  Without going into the details of each of these, DMA is
by far the most critical aspect for maintaining a secure environment
as allowing a device read-write access to system memory imposes the
greatest risk to the overall system integrity.

To help mitigate this risk, many modern IOMMUs now incorporate
isolation properties into what was, in many cases, an interface only
meant for translation (ie. solving the addressing problems of devices
with limited address spaces).  With this, devices can now be isolated
from each other and from arbitrary memory access, thus allowing
things like secure direct assignment of devices into virtual machines.

This isolation is not always at the granularity of a single device
though.  Even when an IOMMU is capable of this, properties of devices,
interconnects, and IOMMU topologies can each reduce this isolation.
For instance, an individual device may be part of a larger multi-
function enclosure.  While the IOMMU may be able to distinguish
between devices within the enclosure, the enclosure may not require
transactions between devices to reach the IOMMU.  Examples of this
could be anything from a multi-function PCI device with backdoors
between functions to a non-PCI-ACS (Access Control Services) capable
bridge allowing redirection without reaching the IOMMU.  Topology
can also play a factor in terms of hiding devices.  A PCIe-to-PCI
bridge masks the devices behind it, making transaction appear as if
from the bridge itself.  Obviously IOMMU design plays a major factor
as well.

Therefore, while for the most part an IOMMU may have device level
granularity, any system is susceptible to reduced granularity.  The
IOMMU API therefore supports a notion of IOMMU groups.  A group is
a set of devices which is isolatable from all other devices in the
system.  Groups are therefore the unit of ownership used by VFIO.

While the group is the minimum granularity that must be used to
ensure secure user access, it's not necessarily the preferred
granularity.  In IOMMUs which make use of page tables, it may be
possible to share a set of page tables between different groups,
reducing the overhead both to the platform (reduced TLB thrashing,
reduced duplicate page tables), and to the user (programming only
a single set of translations).  For this reason, VFIO makes use of
a container class, which may hold one or more groups.  A container
is created by simply opening the /dev/vfio/vfio character device.

On its own, the container provides little functionality, with all
but a couple version and extension query interfaces locked away.
The user needs to add a group into the container for the next level
of functionality.  To do this, the user first needs to identify the
group associated with the desired device.  This can be done using
the sysfs links described in the example below.  By unbinding the
device from the host driver and binding it to a VFIO driver, a new
VFIO group will appear for the group as /dev/vfio/$GROUP, where
$GROUP is the IOMMU group number of which the device is a member.
If the IOMMU group contains multiple devices, each will need to
be bound to a VFIO driver before operations on the VFIO group
are allowed (it's also sufficient to only unbind the device from
host drivers if a VFIO driver is unavailable; this will make the
group available, but not that particular device).  TBD - interface
for disabling driver probing/locking a device.

Once the group is ready, it may be added to the container by opening
the VFIO group character device (/dev/vfio/$GROUP) and using the
VFIO_GROUP_SET_CONTAINER ioctl, passing the file descriptor of the
previously opened container file.  If desired and if the IOMMU driver
supports sharing the IOMMU context between groups, multiple groups may
be set to the same container.  If a group fails to set to a container
with existing groups, a new empty container will need to be used
instead.

With a group (or groups) attached to a container, the remaining
ioctls become available, enabling access to the VFIO IOMMU interfaces.
Additionally, it now becomes possible to get file descriptors for each
device within a group using an ioctl on the VFIO group file descriptor.

The VFIO device API includes ioctls for describing the device, the I/O
regions and their read/write/mmap offsets on the device descriptor, as
well as mechanisms for describing and registering interrupt
notifications.

VFIO Usage Example
------------------

Assume user wants to access PCI device 0000:06:0d.0::

	$ readlink /sys/bus/pci/devices/0000:06:0d.0/iommu_group
	../../../../kernel/iommu_groups/26

This device is therefore in IOMMU group 26.  This device is on the
pci bus, therefore the user will make use of vfio-pci to manage the
group::

	# modprobe vfio-pci

Binding this device to the vfio-pci driver creates the VFIO group
character devices for this group::

	$ lspci -n -s 0000:06:0d.0
	06:0d.0 0401: 1102:0002 (rev 08)
	# echo 0000:06:0d.0 > /sys/bus/pci/devices/0000:06:0d.0/driver/unbind
	# echo 1102 0002 > /sys/bus/pci/drivers/vfio-pci/new_id

Now we need to look at what other devices are in the group to free
it for use by VFIO::

	$ ls -l /sys/bus/pci/devices/0000:06:0d.0/iommu_group/devices
	total 0
	lrwxrwxrwx. 1 root root 0 Apr 23 16:13 0000:00:1e.0 ->
		../../../../devices/pci0000:00/0000:00:1e.0
	lrwxrwxrwx. 1 root root 0 Apr 23 16:13 0000:06:0d.0 ->
		../../../../devices/pci0000:00/0000:00:1e.0/0000:06:0d.0
	lrwxrwxrwx. 1 root root 0 Apr 23 16:13 0000:06:0d.1 ->
		../../../../devices/pci0000:00/0000:00:1e.0/0000:06:0d.1

This device is behind a PCIe-to-PCI bridge [4]_, therefore we also
need to add device 0000:06:0d.1 to the group following the same
procedure as above.  Device 0000:00:1e.0 is a bridge that does
not currently have a host driver, therefore it's not required to
bind this device to the vfio-pci driver (vfio-pci does not currently
support PCI bridges).

The final step is to provide the user with access to the group if
unprivileged operation is desired (note that /dev/vfio/vfio provides
no capabilities on its own and is therefore expected to be set to
mode 0666 by the system)::

	# chown user:user /dev/vfio/26

The user now has full access to all the devices and the iommu for this
group and can access them as follows::

	int container, group, device, i;
	struct vfio_group_status group_status =
					{ .argsz = sizeof(group_status) };
	struct vfio_iommu_type1_info iommu_info = { .argsz = sizeof(iommu_info) };
	struct vfio_iommu_type1_dma_map dma_map = { .argsz = sizeof(dma_map) };
	struct vfio_device_info device_info = { .argsz = sizeof(device_info) };

	/* Create a new container */
	container = open("/dev/vfio/vfio", O_RDWR);

	if (ioctl(container, VFIO_GET_API_VERSION) != VFIO_API_VERSION)
		/* Unknown API version */

	if (!ioctl(container, VFIO_CHECK_EXTENSION, VFIO_TYPE1_IOMMU))
		/* Doesn't support the IOMMU driver we want. */

	/* Open the group */
	group = open("/dev/vfio/26", O_RDWR);

	/* Test the group is viable and available */
	ioctl(group, VFIO_GROUP_GET_STATUS, &group_status);

	if (!(group_status.flags & VFIO_GROUP_FLAGS_VIABLE))
		/* Group is not viable (ie, not all devices bound for vfio) */

	/* Add the group to the container */
	ioctl(group, VFIO_GROUP_SET_CONTAINER, &container);

	/* Enable the IOMMU model we want */
	ioctl(container, VFIO_SET_IOMMU, VFIO_TYPE1_IOMMU);

	/* Get addition IOMMU info */
	ioctl(container, VFIO_IOMMU_GET_INFO, &iommu_info);

	/* Allocate some space and setup a DMA mapping */
	dma_map.vaddr = mmap(0, 1024 * 1024, PROT_READ | PROT_WRITE,
			     MAP_PRIVATE | MAP_ANONYMOUS, 0, 0);
	dma_map.size = 1024 * 1024;
	dma_map.iova = 0; /* 1MB starting at 0x0 from device view */
	dma_map.flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE;

	ioctl(container, VFIO_IOMMU_MAP_DMA, &dma_map);

	/* Get a file descriptor for the device */
	device = ioctl(group, VFIO_GROUP_GET_DEVICE_FD, "0000:06:0d.0");

	/* Test and setup the device */
	ioctl(device, VFIO_DEVICE_GET_INFO, &device_info);

	for (i = 0; i < device_info.num_regions; i++) {
		struct vfio_region_info reg = { .argsz = sizeof(reg) };

		reg.index = i;

		ioctl(device, VFIO_DEVICE_GET_REGION_INFO, &reg);

		/* Setup mappings... read/write offsets, mmaps
		 * For PCI devices, config space is a region */
	}

	for (i = 0; i < device_info.num_irqs; i++) {
		struct vfio_irq_info irq = { .argsz = sizeof(irq) };

		irq.index = i;

		ioctl(device, VFIO_DEVICE_GET_IRQ_INFO, &irq);

		/* Setup IRQs... eventfds, VFIO_DEVICE_SET_IRQS */
	}

	/* Gratuitous device reset and go... */
	ioctl(device, VFIO_DEVICE_RESET);

VFIO User API
-------------------------------------------------------------------------------

Please see include/linux/vfio.h for complete API documentation.

VFIO bus driver API
-------------------------------------------------------------------------------

VFIO bus drivers, such as vfio-pci make use of only a few interfaces
into VFIO core.  When devices are bound and unbound to the driver,
the driver should call vfio_register_group_dev() and
vfio_unregister_group_dev() respectively::

	void vfio_init_group_dev(struct vfio_device *device,
				struct device *dev,
				const struct vfio_device_ops *ops);
	void vfio_uninit_group_dev(struct vfio_device *device);
	int vfio_register_group_dev(struct vfio_device *device);
	void vfio_unregister_group_dev(struct vfio_device *device);

The driver should embed the vfio_device in its own structure and call
vfio_init_group_dev() to pre-configure it before going to registration
and call vfio_uninit_group_dev() after completing the un-registration.
vfio_register_group_dev() indicates to the core to begin tracking the
iommu_group of the specified dev and register the dev as owned by a VFIO bus
driver. Once vfio_register_group_dev() returns it is possible for userspace to
start accessing the driver, thus the driver should ensure it is completely
ready before calling it. The driver provides an ops structure for callbacks
similar to a file operations structure::

	struct vfio_device_ops {
		int	(*open)(struct vfio_device *vdev);
		void	(*release)(struct vfio_device *vdev);
		ssize_t	(*read)(struct vfio_device *vdev, char __user *buf,
				size_t count, loff_t *ppos);
		ssize_t	(*write)(struct vfio_device *vdev,
				 const char __user *buf,
				 size_t size, loff_t *ppos);
		long	(*ioctl)(struct vfio_device *vdev, unsigned int cmd,
				 unsigned long arg);
		int	(*mmap)(struct vfio_device *vdev,
				struct vm_area_struct *vma);
	};

Each function is passed the vdev that was originally registered
in the vfio_register_group_dev() call above.  This allows the bus driver
to obtain its private data using container_of().  The open/release
callbacks are issued when a new file descriptor is created for a
device (via VFIO_GROUP_GET_DEVICE_FD).  The ioctl interface provides
a direct pass through for VFIO_DEVICE_* ioctls.  The read/write/mmap
interfaces implement the device region access defined by the device's
own VFIO_DEVICE_GET_REGION_INFO ioctl.

PPC64 sPAPR implementation note
-------------------------------

This implementation has some specifics:

1) On older systems (POWER7 with P5IOC2/IODA1) only one IOMMU group per
   container is supported as an IOMMU table is allocated at the boot time,
   one table per a IOMMU group which is a Partitionable Endpoint (PE)
   (PE is often a PCI domain but not always).

   Newer systems (POWER8 with IODA2) have improved hardware design which allows
   to remove this limitation and have multiple IOMMU groups per a VFIO
   container.

2) The hardware supports so called DMA windows - the PCI address range
   within which DMA transfer is allowed, any attempt to access address space
   out of the window leads to the whole PE isolation.

3) PPC64 guests are paravirtualized but not fully emulated. There is an API
   to map/unmap pages for DMA, and it normally maps 1..32 pages per call and
   currently there is no way to reduce the number of calls. In order to make
   things faster, the map/unmap handling has been implemented in real mode
   which provides an excellent performance which has limitations such as
   inability to do locked pages accounting in real time.

4) According to sPAPR specification, A Partitionable Endpoint (PE) is an I/O
   subtree that can be treated as a unit for the purposes of partitioning and
   error recovery. A PE may be a single or multi-function IOA (IO Adapter), a
   function of a multi-function IOA, or multiple IOAs (possibly including
   switch and bridge structures above the multiple IOAs). PPC64 guests detect
   PCI errors and recover from them via EEH RTAS services, which works on the
   basis of additional ioctl commands.

   So 4 additional ioctls have been added:

	VFIO_IOMMU_SPAPR_TCE_GET_INFO
		returns the size and the start of the DMA window on the PCI bus.

	VFIO_IOMMU_ENABLE
		enables the container. The locked pages accounting
		is done at this point. This lets user first to know what
		the DMA window is and adjust rlimit before doing any real job.

	VFIO_IOMMU_DISABLE
		disables the container.

	VFIO_EEH_PE_OP
		provides an API for EEH setup, error detection and recovery.

   The code flow from the example above should be slightly changed::

	struct vfio_eeh_pe_op pe_op = { .argsz = sizeof(pe_op), .flags = 0 };

	.....
	/* Add the group to the container */
	ioctl(group, VFIO_GROUP_SET_CONTAINER, &container);

	/* Enable the IOMMU model we want */
	ioctl(container, VFIO_SET_IOMMU, VFIO_SPAPR_TCE_IOMMU)

	/* Get addition sPAPR IOMMU info */
	vfio_iommu_spapr_tce_info spapr_iommu_info;
	ioctl(container, VFIO_IOMMU_SPAPR_TCE_GET_INFO, &spapr_iommu_info);

	if (ioctl(container, VFIO_IOMMU_ENABLE))
		/* Cannot enable container, may be low rlimit */

	/* Allocate some space and setup a DMA mapping */
	dma_map.vaddr = mmap(0, 1024 * 1024, PROT_READ | PROT_WRITE,
			     MAP_PRIVATE | MAP_ANONYMOUS, 0, 0);

	dma_map.size = 1024 * 1024;
	dma_map.iova = 0; /* 1MB starting at 0x0 from device view */
	dma_map.flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE;

	/* Check here is .iova/.size are within DMA window from spapr_iommu_info */
	ioctl(container, VFIO_IOMMU_MAP_DMA, &dma_map);

	/* Get a file descriptor for the device */
	device = ioctl(group, VFIO_GROUP_GET_DEVICE_FD, "0000:06:0d.0");

	....

	/* Gratuitous device reset and go... */
	ioctl(device, VFIO_DEVICE_RESET);

	/* Make sure EEH is supported */
	ioctl(container, VFIO_CHECK_EXTENSION, VFIO_EEH);

	/* Enable the EEH functionality on the device */
	pe_op.op = VFIO_EEH_PE_ENABLE;
	ioctl(container, VFIO_EEH_PE_OP, &pe_op);

	/* You're suggested to create additional data struct to represent
	 * PE, and put child devices belonging to same IOMMU group to the
	 * PE instance for later reference.
	 */

	/* Check the PE's state and make sure it's in functional state */
	pe_op.op = VFIO_EEH_PE_GET_STATE;
	ioctl(container, VFIO_EEH_PE_OP, &pe_op);

	/* Save device state using pci_save_state().
	 * EEH should be enabled on the specified device.
	 */

	....

	/* Inject EEH error, which is expected to be caused by 32-bits
	 * config load.
	 */
	pe_op.op = VFIO_EEH_PE_INJECT_ERR;
	pe_op.err.type = EEH_ERR_TYPE_32;
	pe_op.err.func = EEH_ERR_FUNC_LD_CFG_ADDR;
	pe_op.err.addr = 0ul;
	pe_op.err.mask = 0ul;
	ioctl(container, VFIO_EEH_PE_OP, &pe_op);

	....

	/* When 0xFF's returned from reading PCI config space or IO BARs
	 * of the PCI device. Check the PE's state to see if that has been
	 * frozen.
	 */
	ioctl(container, VFIO_EEH_PE_OP, &pe_op);

	/* Waiting for pending PCI transactions to be completed and don't
	 * produce any more PCI traffic from/to the affected PE until
	 * recovery is finished.
	 */

	/* Enable IO for the affected PE and collect logs. Usually, the
	 * standard part of PCI config space, AER registers are dumped
	 * as logs for further analysis.
	 */
	pe_op.op = VFIO_EEH_PE_UNFREEZE_IO;
	ioctl(container, VFIO_EEH_PE_OP, &pe_op);

	/*
	 * Issue PE reset: hot or fundamental reset. Usually, hot reset
	 * is enough. However, the firmware of some PCI adapters would
	 * require fundamental reset.
	 */
	pe_op.op = VFIO_EEH_PE_RESET_HOT;
	ioctl(container, VFIO_EEH_PE_OP, &pe_op);
	pe_op.op = VFIO_EEH_PE_RESET_DEACTIVATE;
	ioctl(container, VFIO_EEH_PE_OP, &pe_op);

	/* Configure the PCI bridges for the affected PE */
	pe_op.op = VFIO_EEH_PE_CONFIGURE;
	ioctl(container, VFIO_EEH_PE_OP, &pe_op);

	/* Restored state we saved at initialization time. pci_restore_state()
	 * is good enough as an example.
	 */

	/* Hopefully, error is recovered successfully. Now, you can resume to
	 * start PCI traffic to/from the affected PE.
	 */

	....

5) There is v2 of SPAPR TCE IOMMU. It deprecates VFIO_IOMMU_ENABLE/
   VFIO_IOMMU_DISABLE and implements 2 new ioctls:
   VFIO_IOMMU_SPAPR_REGISTER_MEMORY and VFIO_IOMMU_SPAPR_UNREGISTER_MEMORY
   (which are unsupported in v1 IOMMU).

   PPC64 paravirtualized guests generate a lot of map/unmap requests,
   and the handling of those includes pinning/unpinning pages and updating
   mm::locked_vm counter to make sure we do not exceed the rlimit.
   The v2 IOMMU splits accounting and pinning into separate operations:

   - VFIO_IOMMU_SPAPR_REGISTER_MEMORY/VFIO_IOMMU_SPAPR_UNREGISTER_MEMORY ioctls
     receive a user space address and size of the block to be pinned.
     Bisecting is not supported and VFIO_IOMMU_UNREGISTER_MEMORY is expected to
     be called with the exact address and size used for registering
     the memory block. The userspace is not expected to call these often.
     The ranges are stored in a linked list in a VFIO container.

   - VFIO_IOMMU_MAP_DMA/VFIO_IOMMU_UNMAP_DMA ioctls only update the actual
     IOMMU table and do not do pinning; instead these check that the userspace
     address is from pre-registered range.

   This separation helps in optimizing DMA for guests.

6) sPAPR specification allows guests to have an additional DMA window(s) on
   a PCI bus with a variable page size. Two ioctls have been added to support
   this: VFIO_IOMMU_SPAPR_TCE_CREATE and VFIO_IOMMU_SPAPR_TCE_REMOVE.
   The platform has to support the functionality or error will be returned to
   the userspace. The existing hardware supports up to 2 DMA windows, one is
   2GB long, uses 4K pages and called "default 32bit window"; the other can
   be as big as entire RAM, use different page size, it is optional - guests
   create those in run-time if the guest driver supports 64bit DMA.

   VFIO_IOMMU_SPAPR_TCE_CREATE receives a page shift, a DMA window size and
   a number of TCE table levels (if a TCE table is going to be big enough and
   the kernel may not be able to allocate enough of physically contiguous
   memory). It creates a new window in the available slot and returns the bus
   address where the new window starts. Due to hardware limitation, the user
   space cannot choose the location of DMA windows.

   VFIO_IOMMU_SPAPR_TCE_REMOVE receives the bus start address of the window
   and removes it.

-------------------------------------------------------------------------------

.. [1] VFIO was originally an acronym for "Virtual Function I/O" in its
   initial implementation by Tom Lyon while as Cisco.  We've since
   outgrown the acronym, but it's catchy.

.. [2] "safe" also depends upon a device being "well behaved".  It's
   possible for multi-function devices to have backdoors between
   functions and even for single function devices to have alternative
   access to things like PCI config space through MMIO registers.  To
   guard against the former we can include additional precautions in the
   IOMMU driver to group multi-function PCI devices together
   (iommu=group_mf).  The latter we can't prevent, but the IOMMU should
   still provide isolation.  For PCI, SR-IOV Virtual Functions are the
   best indicator of "well behaved", as these are designed for
   virtualization usage models.

.. [3] As always there are trade-offs to virtual machine device
   assignment that are beyond the scope of VFIO.  It's expected that
   future IOMMU technologies will reduce some, but maybe not all, of
   these trade-offs.

.. [4] In this case the device is below a PCI bridge, so transactions
   from either function of the device are indistinguishable to the iommu::

	-[0000:00]-+-1e.0-[06]--+-0d.0
				\-0d.1

	00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)


code:
VME Device Drivers
==================

Driver registration
-------------------

As with other subsystems within the Linux kernel, VME device drivers register
with the VME subsystem, typically called from the devices init routine.  This is
achieved via a call to :c:func:`vme_register_driver`.

A pointer to a structure of type :c:type:`struct vme_driver <vme_driver>` must
be provided to the registration function. Along with the maximum number of
devices your driver is able to support.

At the minimum, the '.name', '.match' and '.probe' elements of
:c:type:`struct vme_driver <vme_driver>` should be correctly set. The '.name'
element is a pointer to a string holding the device driver's name.

The '.match' function allows control over which VME devices should be registered
with the driver. The match function should return 1 if a device should be
probed and 0 otherwise. This example match function (from vme_user.c) limits
the number of devices probed to one:

.. code-block:: c

	#define USER_BUS_MAX	1
	...
	static int vme_user_match(struct vme_dev *vdev)
	{
		if (vdev->id.num >= USER_BUS_MAX)
			return 0;
		return 1;
	}

The '.probe' element should contain a pointer to the probe routine. The
probe routine is passed a :c:type:`struct vme_dev <vme_dev>` pointer as an
argument.

Here, the 'num' field refers to the sequential device ID for this specific
driver. The bridge number (or bus number) can be accessed using
dev->bridge->num.

A function is also provided to unregister the driver from the VME core called
:c:func:`vme_unregister_driver` and should usually be called from the device
driver's exit routine.

Resource management
-------------------

Once a driver has registered with the VME core the provided match routine will
be called the number of times specified during the registration. If a match
succeeds, a non-zero value should be returned. A zero return value indicates
failure. For all successful matches, the probe routine of the corresponding
driver is called. The probe routine is passed a pointer to the devices
device structure. This pointer should be saved, it will be required for
requesting VME resources.

The driver can request ownership of one or more master windows
(:c:func:`vme_master_request`), slave windows (:c:func:`vme_slave_request`)
and/or dma channels (:c:func:`vme_dma_request`). Rather than allowing the device
driver to request a specific window or DMA channel (which may be used by a
different driver) the API allows a resource to be assigned based on the required
attributes of the driver in question. For slave windows these attributes are
split into the VME address spaces that need to be accessed in 'aspace' and VME
bus cycle types required in 'cycle'. Master windows add a further set of
attributes in 'width' specifying the required data transfer widths. These
attributes are defined as bitmasks and as such any combination of the
attributes can be requested for a single window, the core will assign a window
that meets the requirements, returning a pointer of type vme_resource that
should be used to identify the allocated resource when it is used. For DMA
controllers, the request function requires the potential direction of any
transfers to be provided in the route attributes. This is typically VME-to-MEM
and/or MEM-to-VME, though some hardware can support VME-to-VME and MEM-to-MEM
transfers as well as test pattern generation. If an unallocated window fitting
the requirements can not be found a NULL pointer will be returned.

Functions are also provided to free window allocations once they are no longer
required. These functions (:c:func:`vme_master_free`, :c:func:`vme_slave_free`
and :c:func:`vme_dma_free`) should be passed the pointer to the resource
provided during resource allocation.

Master windows
--------------

Master windows provide access from the local processor[s] out onto the VME bus.
The number of windows available and the available access modes is dependent on
the underlying chipset. A window must be configured before it can be used.

Master window configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~

Once a master window has been assigned :c:func:`vme_master_set` can be used to
configure it and :c:func:`vme_master_get` to retrieve the current settings. The
address spaces, transfer widths and cycle types are the same as described
under resource management, however some of the options are mutually exclusive.
For example, only one address space may be specified.

Master window access
~~~~~~~~~~~~~~~~~~~~

The function :c:func:`vme_master_read` can be used to read from and
:c:func:`vme_master_write` used to write to configured master windows.

In addition to simple reads and writes, :c:func:`vme_master_rmw` is provided to
do a read-modify-write transaction. Parts of a VME window can also be mapped
into user space memory using :c:func:`vme_master_mmap`.

Slave windows
-------------

Slave windows provide devices on the VME bus access into mapped portions of the
local memory. The number of windows available and the access modes that can be
used is dependent on the underlying chipset. A window must be configured before
it can be used.

Slave window configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~

Once a slave window has been assigned :c:func:`vme_slave_set` can be used to
configure it and :c:func:`vme_slave_get` to retrieve the current settings.

The address spaces, transfer widths and cycle types are the same as described
under resource management, however some of the options are mutually exclusive.
For example, only one address space may be specified.

Slave window buffer allocation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Functions are provided to allow the user to allocate
(:c:func:`vme_alloc_consistent`) and free (:c:func:`vme_free_consistent`)
contiguous buffers which will be accessible by the VME bridge. These functions
do not have to be used, other methods can be used to allocate a buffer, though
care must be taken to ensure that they are contiguous and accessible by the VME
bridge.

Slave window access
~~~~~~~~~~~~~~~~~~~

Slave windows map local memory onto the VME bus, the standard methods for
accessing memory should be used.

DMA channels
------------

The VME DMA transfer provides the ability to run link-list DMA transfers. The
API introduces the concept of DMA lists. Each DMA list is a link-list which can
be passed to a DMA controller. Multiple lists can be created, extended,
executed, reused and destroyed.

List Management
~~~~~~~~~~~~~~~

The function :c:func:`vme_new_dma_list` is provided to create and
:c:func:`vme_dma_list_free` to destroy DMA lists. Execution of a list will not
automatically destroy the list, thus enabling a list to be reused for repetitive
tasks.

List Population
~~~~~~~~~~~~~~~

An item can be added to a list using :c:func:`vme_dma_list_add` (the source and
destination attributes need to be created before calling this function, this is
covered under "Transfer Attributes").

.. note::

	The detailed attributes of the transfers source and destination
	are not checked until an entry is added to a DMA list, the request
	for a DMA channel purely checks the directions in which the
	controller is expected to transfer data. As a result it is
	possible for this call to return an error, for example if the
	source or destination is in an unsupported VME address space.

Transfer Attributes
~~~~~~~~~~~~~~~~~~~

The attributes for the source and destination are handled separately from adding
an item to a list. This is due to the diverse attributes required for each type
of source and destination. There are functions to create attributes for PCI, VME
and pattern sources and destinations (where appropriate):

 - PCI source or destination: :c:func:`vme_dma_pci_attribute`
 - VME source or destination: :c:func:`vme_dma_vme_attribute`
 - Pattern source: :c:func:`vme_dma_pattern_attribute`

The function :c:func:`vme_dma_free_attribute` should be used to free an
attribute.

List Execution
~~~~~~~~~~~~~~

The function :c:func:`vme_dma_list_exec` queues a list for execution and will
return once the list has been executed.

Interrupts
----------

The VME API provides functions to attach and detach callbacks to specific VME
level and status ID combinations and for the generation of VME interrupts with
specific VME level and status IDs.

Attaching Interrupt Handlers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The function :c:func:`vme_irq_request` can be used to attach and
:c:func:`vme_irq_free` to free a specific VME level and status ID combination.
Any given combination can only be assigned a single callback function. A void
pointer parameter is provided, the value of which is passed to the callback
function, the use of this pointer is user undefined. The callback parameters are
as follows. Care must be taken in writing a callback function, callback
functions run in interrupt context:

.. code-block:: c

	void callback(int level, int statid, void *priv);

Interrupt Generation
~~~~~~~~~~~~~~~~~~~~

The function :c:func:`vme_irq_generate` can be used to generate a VME interrupt
at a given VME level and VME status ID.

Location monitors
-----------------

The VME API provides the following functionality to configure the location
monitor.

Location Monitor Management
~~~~~~~~~~~~~~~~~~~~~~~~~~~

The function :c:func:`vme_lm_request` is provided to request the use of a block
of location monitors and :c:func:`vme_lm_free` to free them after they are no
longer required. Each block may provide a number of location monitors,
monitoring adjacent locations. The function :c:func:`vme_lm_count` can be used
to determine how many locations are provided.

Location Monitor Configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Once a bank of location monitors has been allocated, the function
:c:func:`vme_lm_set` is provided to configure the location and mode of the
location monitor. The function :c:func:`vme_lm_get` can be used to retrieve
existing settings.

Location Monitor Use
~~~~~~~~~~~~~~~~~~~~

The function :c:func:`vme_lm_attach` enables a callback to be attached and
:c:func:`vme_lm_detach` allows on to be detached from each location monitor
location. Each location monitor can monitor a number of adjacent locations. The
callback function is declared as follows.

.. code-block:: c

	void callback(void *data);

Slot Detection
--------------

The function :c:func:`vme_slot_num` returns the slot ID of the provided bridge.

Bus Detection
-------------

The function :c:func:`vme_bus_num` returns the bus ID of the provided bridge.

VME API
-------

.. kernel-doc:: drivers/staging/vme_user/vme.h
   :internal:

.. kernel-doc:: drivers/staging/vme_user/vme.c
   :export:


code:
======================
W1: Dallas' 1-wire bus
======================

:Author: David Fries

W1 API internal to the kernel
=============================

include/linux/w1.h
~~~~~~~~~~~~~~~~~~

W1 kernel API functions.

.. kernel-doc:: include/linux/w1.h
   :internal:

drivers/w1/w1.c
~~~~~~~~~~~~~~~

W1 core functions.

.. kernel-doc:: drivers/w1/w1.c
   :internal:

drivers/w1/w1_family.c
~~~~~~~~~~~~~~~~~~~~~~~

Allows registering device family operations.

.. kernel-doc:: drivers/w1/w1_family.c
   :export:

drivers/w1/w1_internal.h
~~~~~~~~~~~~~~~~~~~~~~~~

W1 internal initialization for master devices.

.. kernel-doc:: drivers/w1/w1_internal.h
   :internal:

drivers/w1/w1_int.c
~~~~~~~~~~~~~~~~~~~~

W1 internal initialization for master devices.

.. kernel-doc:: drivers/w1/w1_int.c
   :export:

drivers/w1/w1_netlink.h
~~~~~~~~~~~~~~~~~~~~~~~~

W1 external netlink API structures and commands.

.. kernel-doc:: drivers/w1/w1_netlink.h
   :internal:

drivers/w1/w1_io.c
~~~~~~~~~~~~~~~~~~~

W1 input/output.

.. kernel-doc:: drivers/w1/w1_io.c
   :export:

.. kernel-doc:: drivers/w1/w1_io.c
   :internal:


code:
====================================
Xilinx Zynq MPSoC EEMI Documentation
====================================

Xilinx Zynq MPSoC Firmware Interface
-------------------------------------
The zynqmp-firmware node describes the interface to platform firmware.
ZynqMP has an interface to communicate with secure firmware. Firmware
driver provides an interface to firmware APIs. Interface APIs can be
used by any driver to communicate with PMC(Platform Management Controller).

Embedded Energy Management Interface (EEMI)
----------------------------------------------
The embedded energy management interface is used to allow software
components running across different processing clusters on a chip or
device to communicate with a power management controller (PMC) on a
device to issue or respond to power management requests.

Any driver who wants to communicate with PMC using EEMI APIs use the
functions provided for each function.

IOCTL
------
IOCTL API is for device control and configuration. It is not a system
IOCTL but it is an EEMI API. This API can be used by master to control
any device specific configuration. IOCTL definitions can be platform
specific. This API also manage shared device configuration.

The following IOCTL IDs are valid for device control:
- IOCTL_SET_PLL_FRAC_MODE	8
- IOCTL_GET_PLL_FRAC_MODE	9
- IOCTL_SET_PLL_FRAC_DATA	10
- IOCTL_GET_PLL_FRAC_DATA	11

Refer EEMI API guide [0] for IOCTL specific parameters and other EEMI APIs.

References
----------
[0] Embedded Energy Management Interface (EEMI) API guide:
    https://www.xilinx.com/support/documentation/user_guides/ug1200-eemi-api.pdf


code:
===========
Xilinx FPGA
===========

.. toctree::
    :maxdepth: 1

    eemi

.. only::  subproject and html

   Indices
   =======

   * :ref:`genindex`