code:
.. SPDX-License-Identifier: GPL-2.0
.. _imc:

===================================
IMC (In-Memory Collection Counters)
===================================

Anju T Sudhakar, 10 May 2019

.. contents::
    :depth: 3

Basic overview
==============

IMC (In-Memory collection counters) is a hardware monitoring facility that
collects large numbers of hardware performance events at Nest level (these are
on-chip but off-core), Core level and Thread level.

The Nest PMU counters are handled by a Nest IMC microcode which runs in the OCC
(On-Chip Controller) complex. The microcode collects the counter data and moves
the nest IMC counter data to memory.

The Core and Thread IMC PMU counters are handled in the core. Core level PMU
counters give us the IMC counters' data per core and thread level PMU counters
give us the IMC counters' data per CPU thread.

OPAL obtains the IMC PMU and supported events information from the IMC Catalog
and passes on to the kernel via the device tree. The event's information
contains:

- Event name
- Event Offset
- Event description

and possibly also:

- Event scale
- Event unit

Some PMUs may have a common scale and unit values for all their supported
events. For those cases, the scale and unit properties for those events must be
inherited from the PMU.

The event offset in the memory is where the counter data gets accumulated.

IMC catalog is available at:
	https://github.com/open-power/ima-catalog

The kernel discovers the IMC counters information in the device tree at the
`imc-counters` device node which has a compatible field
`ibm,opal-in-memory-counters`. From the device tree, the kernel parses the PMUs
and their event's information and register the PMU and its attributes in the
kernel.

IMC example usage
=================

.. code-block:: sh

  # perf list
  [...]
  nest_mcs01/PM_MCS01_64B_RD_DISP_PORT01/            [Kernel PMU event]
  nest_mcs01/PM_MCS01_64B_RD_DISP_PORT23/            [Kernel PMU event]
  [...]
  core_imc/CPM_0THRD_NON_IDLE_PCYC/                  [Kernel PMU event]
  core_imc/CPM_1THRD_NON_IDLE_INST/                  [Kernel PMU event]
  [...]
  thread_imc/CPM_0THRD_NON_IDLE_PCYC/                [Kernel PMU event]
  thread_imc/CPM_1THRD_NON_IDLE_INST/                [Kernel PMU event]

To see per chip data for nest_mcs0/PM_MCS_DOWN_128B_DATA_XFER_MC0/:

.. code-block:: sh

  # ./perf stat -e "nest_mcs01/PM_MCS01_64B_WR_DISP_PORT01/" -a --per-socket

To see non-idle instructions for core 0:

.. code-block:: sh

  # ./perf stat -e "core_imc/CPM_NON_IDLE_INST/" -C 0 -I 1000

To see non-idle instructions for a "make":

.. code-block:: sh

  # ./perf stat -e "thread_imc/CPM_NON_IDLE_PCYC/" make

IMC Trace-mode
===============

POWER9 supports two modes for IMC which are the Accumulation mode and Trace
mode. In Accumulation mode, event counts are accumulated in system Memory.
Hypervisor then reads the posted counts periodically or when requested. In IMC
Trace mode, the 64 bit trace SCOM value is initialized with the event
information. The CPMCxSEL and CPMC_LOAD in the trace SCOM, specifies the event
to be monitored and the sampling duration. On each overflow in the CPMCxSEL,
hardware snapshots the program counter along with event counts and writes into
memory pointed by LDBAR.

LDBAR is a 64 bit special purpose per thread register, it has bits to indicate
whether hardware is configured for accumulation or trace mode.

LDBAR Register Layout
---------------------

  +-------+----------------------+
  | 0     | Enable/Disable       |
  +-------+----------------------+
  | 1     | 0: Accumulation Mode |
  |       +----------------------+
  |       | 1: Trace Mode        |
  +-------+----------------------+
  | 2:3   | Reserved             |
  +-------+----------------------+
  | 4-6   | PB scope             |
  +-------+----------------------+
  | 7     | Reserved             |
  +-------+----------------------+
  | 8:50  | Counter Address      |
  +-------+----------------------+
  | 51:63 | Reserved             |
  +-------+----------------------+

TRACE_IMC_SCOM bit representation
---------------------------------

  +-------+------------+
  | 0:1   | SAMPSEL    |
  +-------+------------+
  | 2:33  | CPMC_LOAD  |
  +-------+------------+
  | 34:40 | CPMC1SEL   |
  +-------+------------+
  | 41:47 | CPMC2SEL   |
  +-------+------------+
  | 48:50 | BUFFERSIZE |
  +-------+------------+
  | 51:63 | RESERVED   |
  +-------+------------+

CPMC_LOAD contains the sampling duration. SAMPSEL and CPMCxSEL determines the
event to count. BUFFERSIZE indicates the memory range. On each overflow,
hardware snapshots the program counter along with event counts and updates the
memory and reloads the CMPC_LOAD value for the next sampling duration. IMC
hardware does not support exceptions, so it quietly wraps around if memory
buffer reaches the end.

*Currently the event monitored for trace-mode is fixed as cycle.*

Trace IMC example usage
=======================

.. code-block:: sh

  # perf list
  [....]
  trace_imc/trace_cycles/                            [Kernel PMU event]

To record an application/process with trace-imc event:

.. code-block:: sh

  # perf record -e trace_imc/trace_cycles/ yes > /dev/null
  [ perf record: Woken up 1 times to write data ]
  [ perf record: Captured and wrote 0.012 MB perf.data (21 samples) ]

The `perf.data` generated, can be read using perf report.

Benefits of using IMC trace-mode
================================

PMI (Performance Monitoring Interrupts) interrupt handling is avoided, since IMC
trace mode snapshots the program counter and updates to the memory. And this
also provide a way for the operating system to do instruction sampling in real
time without PMI processing overhead.

Performance data using `perf top` with and without trace-imc event.

PMI interrupts count when `perf top` command is executed without trace-imc event.

.. code-block:: sh

  # grep PMI /proc/interrupts
  PMI:          0          0          0          0   Performance monitoring interrupts
  # ./perf top
  ...
  # grep PMI /proc/interrupts
  PMI:      39735       8710      17338      17801   Performance monitoring interrupts
  # ./perf top -e trace_imc/trace_cycles/
  ...
  # grep PMI /proc/interrupts
  PMI:      39735       8710      17338      17801   Performance monitoring interrupts

That is, the PMI interrupt counts do not increment when using the `trace_imc` event.


code:
.. SPDX-License-Identifier: GPL-2.0

=======
powerpc
=======

.. toctree::
    :maxdepth: 1

    associativity
    booting
    bootwrapper
    cpu_families
    cpu_features
    cxl
    cxlflash
    dawr-power9
    dscr
    eeh-pci-error-recovery
    elf_hwcaps
    elfnote
    firmware-assisted-dump
    hvcs
    imc
    isa-versions
    kaslr-booke32
    mpc52xx
    papr_hcalls
    pci_iov_resource_on_powernv
    pmu-ebb
    ptrace
    qe_firmware
    syscall64-abi
    transactional_memory
    ultravisor
    vas-api
    vcpudispatch_stats

    features

.. only::  subproject and html

   Indices
   =======

   * :ref:`genindex`

repo_name: angery/linux


code:
==========================
CPU to ISA Version Mapping
==========================

Mapping of some CPU versions to relevant ISA versions.

Note Power4 and Power4+ are not supported.

========= ====================================================================
CPU       Architecture version
========= ====================================================================
Power10   Power ISA v3.1
Power9    Power ISA v3.0B
Power8    Power ISA v2.07
e6500     Power ISA v2.06 with some exceptions
e5500     Power ISA v2.06 with some exceptions, no Altivec
Power7    Power ISA v2.06
Power6    Power ISA v2.05
PA6T      Power ISA v2.04
Cell PPU  - Power ISA v2.02 with some minor exceptions
          - Plus Altivec/VMX ~= 2.03
Power5++  Power ISA v2.04 (no VMX)
Power5+   Power ISA v2.03
Power5    - PowerPC User Instruction Set Architecture Book I v2.02
          - PowerPC Virtual Environment Architecture Book II v2.02
          - PowerPC Operating Environment Architecture Book III v2.02
PPC970    - PowerPC User Instruction Set Architecture Book I v2.01
          - PowerPC Virtual Environment Architecture Book II v2.01
          - PowerPC Operating Environment Architecture Book III v2.01
          - Plus Altivec/VMX ~= 2.03
Power4+   - PowerPC User Instruction Set Architecture Book I v2.01
          - PowerPC Virtual Environment Architecture Book II v2.01
          - PowerPC Operating Environment Architecture Book III v2.01
Power4    - PowerPC User Instruction Set Architecture Book I v2.00
          - PowerPC Virtual Environment Architecture Book II v2.00
          - PowerPC Operating Environment Architecture Book III v2.00
========= ====================================================================

Key Features
------------

========== ==================
CPU        VMX (aka. Altivec)
========== ==================
Power10    Yes
Power9     Yes
Power8     Yes
e6500      Yes
e5500      No
Power7     Yes
Power6     Yes
PA6T       Yes
Cell PPU   Yes
Power5++   No
Power5+    No
Power5     No
PPC970     Yes
Power4+    No
Power4     No
========== ==================

========== ====
CPU        VSX
========== ====
Power10    Yes
Power9     Yes
Power8     Yes
e6500      No
e5500      No
Power7     Yes
Power6     No
PA6T       No
Cell PPU   No
Power5++   No
Power5+    No
Power5     No
PPC970     No
Power4+    No
Power4     No
========== ====

========== ====================================
CPU        Transactional Memory
========== ====================================
Power10    No  (* see Power ISA v3.1, "Appendix A. Notes on the Removal of Transactional Memory from the Architecture")
Power9     Yes (* see transactional_memory.txt)
Power8     Yes
e6500      No
e5500      No
Power7     No
Power6     No
PA6T       No
Cell PPU   No
Power5++   No
Power5+    No
Power5     No
PPC970     No
Power4+    No
Power4     No
========== ====================================


code:
KASAN is supported on powerpc on 32-bit and Radix 64-bit only.

32 bit support
==============

KASAN is supported on both hash and nohash MMUs on 32-bit.

The shadow area sits at the top of the kernel virtual memory space above the
fixmap area and occupies one eighth of the total kernel virtual memory space.

Instrumentation of the vmalloc area is optional, unless built with modules,
in which case it is required.

64 bit support
==============

Currently, only the radix MMU is supported. There have been versions for hash
and Book3E processors floating around on the mailing list, but nothing has been
merged.

KASAN support on Book3S is a bit tricky to get right:

 - It would be good to support inline instrumentation so as to be able to catch
   stack issues that cannot be caught with outline mode.

 - Inline instrumentation requires a fixed offset.

 - Book3S runs code with translations off ("real mode") during boot, including a
   lot of generic device-tree parsing code which is used to determine MMU
   features.

 - Some code - most notably a lot of KVM code - also runs with translations off
   after boot.

 - Therefore any offset has to point to memory that is valid with
   translations on or off.

One approach is just to give up on inline instrumentation. This way boot-time
checks can be delayed until after the MMU is set is up, and we can just not
instrument any code that runs with translations off after booting. This is the
current approach.

To avoid this limitiation, the KASAN shadow would have to be placed inside the
linear mapping, using the same high-bits trick we use for the rest of the linear
mapping. This is tricky:

 - We'd like to place it near the start of physical memory. In theory we can do
   this at run-time based on how much physical memory we have, but this requires
   being able to arbitrarily relocate the kernel, which is basically the tricky
   part of KASLR. Not being game to implement both tricky things at once, this
   is hopefully something we can revisit once we get KASLR for Book3S.

 - Alternatively, we can place the shadow at the _end_ of memory, but this
   requires knowing how much contiguous physical memory a system has _at compile
   time_. This is a big hammer, and has some unfortunate consequences: inablity
   to handle discontiguous physical memory, total failure to boot on machines
   with less memory than specified, and that machines with more memory than
   specified can't use it. This was deemed unacceptable.


code:
.. SPDX-License-Identifier: GPL-2.0

===========================
KASLR for Freescale BookE32
===========================

The word KASLR stands for Kernel Address Space Layout Randomization.

This document tries to explain the implementation of the KASLR for
Freescale BookE32. KASLR is a security feature that deters exploit
attempts relying on knowledge of the location of kernel internals.

Since CONFIG_RELOCATABLE has already supported, what we need to do is
map or copy kernel to a proper place and relocate. Freescale Book-E
parts expect lowmem to be mapped by fixed TLB entries(TLB1). The TLB1
entries are not suitable to map the kernel directly in a randomized
region, so we chose to copy the kernel to a proper place and restart to
relocate.

Entropy is derived from the banner and timer base, which will change every
build and boot. This not so much safe so additionally the bootloader may
pass entropy via the /chosen/kaslr-seed node in device tree.

We will use the first 512M of the low memory to randomize the kernel
image. The memory will be split in 64M zones. We will use the lower 8
bit of the entropy to decide the index of the 64M zone. Then we chose a
16K aligned offset inside the 64M zone to put the kernel in::

    KERNELBASE

        |-->   64M   <--|
        |               |
        +---------------+    +----------------+---------------+
        |               |....|    |kernel|    |               |
        +---------------+    +----------------+---------------+
        |                         |
        |----->   offset    <-----|

                              kernstart_virt_addr

To enable KASLR, set CONFIG_RANDOMIZE_BASE = y. If KASLR is enabled and you
want to disable it at runtime, add "nokaslr" to the kernel cmdline.


code:
=============================
Linux 2.6.x on MPC52xx family
=============================

For the latest info, go to https://www.246tNt.com/mpc52xx/

To compile/use :

  - U-Boot::

     # <edit Makefile to set ARCH=ppc & CROSS_COMPILE=... ( also EXTRAVERSION
        if you wish to ).
     # make lite5200_defconfig
     # make uImage

     then, on U-boot:
     => tftpboot 200000 uImage
     => tftpboot 400000 pRamdisk
     => bootm 200000 400000

  - DBug::

     # <edit Makefile to set ARCH=ppc & CROSS_COMPILE=... ( also EXTRAVERSION
        if you wish to ).
     # make lite5200_defconfig
     # cp your_initrd.gz arch/ppc/boot/images/ramdisk.image.gz
     # make zImage.initrd
     # make

     then in DBug:
     DBug> dn -i zImage.initrd.lite5200

Some remarks:

 - The port is named mpc52xxx, and config options are PPC_MPC52xx. The MGT5100
   is not supported, and I'm not sure anyone is interested in working on it
   so. I didn't took 5xxx because there's apparently a lot of 5xxx that have
   nothing to do with the MPC5200. I also included the 'MPC' for the same
   reason.
 - Of course, I inspired myself from the 2.4 port. If you think I forgot to
   mention you/your company in the copyright of some code, I'll correct it
   ASAP.


code:
.. SPDX-License-Identifier: GPL-2.0

===========================
Hypercall Op-codes (hcalls)
===========================

Overview
=========

Virtualization on 64-bit Power Book3S Platforms is based on the PAPR
specification [1]_ which describes the run-time environment for a guest
operating system and how it should interact with the hypervisor for
privileged operations. Currently there are two PAPR compliant hypervisors:

- **IBM PowerVM (PHYP)**: IBM's proprietary hypervisor that supports AIX,
  IBM-i and  Linux as supported guests (termed as Logical Partitions
  or LPARS). It supports the full PAPR specification.

- **Qemu/KVM**: Supports PPC64 linux guests running on a PPC64 linux host.
  Though it only implements a subset of PAPR specification called LoPAPR [2]_.

On PPC64 arch a guest kernel running on top of a PAPR hypervisor is called
a *pSeries guest*. A pseries guest runs in a supervisor mode (HV=0) and must
issue hypercalls to the hypervisor whenever it needs to perform an action
that is hypervisor priviledged [3]_ or for other services managed by the
hypervisor.

Hence a Hypercall (hcall) is essentially a request by the pseries guest
asking hypervisor to perform a privileged operation on behalf of the guest. The
guest issues a with necessary input operands. The hypervisor after performing
the privilege operation returns a status code and output operands back to the
guest.

HCALL ABI
=========
The ABI specification for a hcall between a pseries guest and PAPR hypervisor
is covered in section 14.5.3 of ref [2]_. Switch to the  Hypervisor context is
done via the instruction **HVCS** that expects the Opcode for hcall is set in *r3*
and any in-arguments for the hcall are provided in registers *r4-r12*. If values
have to be passed through a memory buffer, the data stored in that buffer should be
in Big-endian byte order.

Once control returns back to the guest after hypervisor has serviced the
'HVCS' instruction the return value of the hcall is available in *r3* and any
out values are returned in registers *r4-r12*. Again like in case of in-arguments,
any out values stored in a memory buffer will be in Big-endian byte order.

Powerpc arch code provides convenient wrappers named **plpar_hcall_xxx** defined
in a arch specific header [4]_ to issue hcalls from the linux kernel
running as pseries guest.

Register Conventions
====================

Any hcall should follow same register convention as described in section 2.2.1.1
of "64-Bit ELF V2 ABI Specification: Power Architecture"[5]_. Table below
summarizes these conventions:

+----------+----------+-------------------------------------------+
| Register |Volatile  |  Purpose                                  |
| Range    |(Y/N)     |                                           |
+==========+==========+===========================================+
|   r0     |    Y     |  Optional-usage                           |
+----------+----------+-------------------------------------------+
|   r1     |    N     |  Stack Pointer                            |
+----------+----------+-------------------------------------------+
|   r2     |    N     |  TOC                                      |
+----------+----------+-------------------------------------------+
|   r3     |    Y     |  hcall opcode/return value                |
+----------+----------+-------------------------------------------+
|  r4-r10  |    Y     |  in and out values                        |
+----------+----------+-------------------------------------------+
|   r11    |    Y     |  Optional-usage/Environmental pointer     |
+----------+----------+-------------------------------------------+
|   r12    |    Y     |  Optional-usage/Function entry address at |
|          |          |  global entry point                       |
+----------+----------+-------------------------------------------+
|   r13    |    N     |  Thread-Pointer                           |
+----------+----------+-------------------------------------------+
|  r14-r31 |    N     |  Local Variables                          |
+----------+----------+-------------------------------------------+
|    LR    |    Y     |  Link Register                            |
+----------+----------+-------------------------------------------+
|   CTR    |    Y     |  Loop Counter                             |
+----------+----------+-------------------------------------------+
|   XER    |    Y     |  Fixed-point exception register.          |
+----------+----------+-------------------------------------------+
|  CR0-1   |    Y     |  Condition register fields.               |
+----------+----------+-------------------------------------------+
|  CR2-4   |    N     |  Condition register fields.               |
+----------+----------+-------------------------------------------+
|  CR5-7   |    Y     |  Condition register fields.               |
+----------+----------+-------------------------------------------+
|  Others  |    N     |                                           |
+----------+----------+-------------------------------------------+

DRC & DRC Indexes
=================
::

     DR1                                  Guest
     +--+        +------------+         +---------+
     |  | <----> |            |         |  User   |
     +--+  DRC1  |            |   DRC   |  Space  |
                 |    PAPR    |  Index  +---------+
     DR2         | Hypervisor |         |         |
     +--+        |            | <-----> |  Kernel |
     |  | <----> |            |  Hcall  |         |
     +--+  DRC2  +------------+         +---------+

PAPR hypervisor terms shared hardware resources like PCI devices, NVDIMMs etc
available for use by LPARs as Dynamic Resource (DR). When a DR is allocated to
an LPAR, PHYP creates a data-structure called Dynamic Resource Connector (DRC)
to manage LPAR access. An LPAR refers to a DRC via an opaque 32-bit number
called DRC-Index. The DRC-index value is provided to the LPAR via device-tree
where its present as an attribute in the device tree node associated with the
DR.

HCALL Return-values
===================

After servicing the hcall, hypervisor sets the return-value in *r3* indicating
success or failure of the hcall. In case of a failure an error code indicates
the cause for error. These codes are defined and documented in arch specific
header [4]_.

In some cases a hcall can potentially take a long time and need to be issued
multiple times in order to be completely serviced. These hcalls will usually
accept an opaque value *continue-token* within there argument list and a
return value of *H_CONTINUE* indicates that hypervisor hasn't still finished
servicing the hcall yet.

To make such hcalls the guest need to set *continue-token == 0* for the
initial call and use the hypervisor returned value of *continue-token*
for each subsequent hcall until hypervisor returns a non *H_CONTINUE*
return value.

HCALL Op-codes
==============

Below is a partial list of HCALLs that are supported by PHYP. For the
corresponding opcode values please look into the arch specific header [4]_:

**H_SCM_READ_METADATA**

| Input: *drcIndex, offset, buffer-address, numBytesToRead*
| Out: *numBytesRead*
| Return Value: *H_Success, H_Parameter, H_P2, H_P3, H_Hardware*

Given a DRC Index of an NVDIMM, read N-bytes from the metadata area
associated with it, at a specified offset and copy it to provided buffer.
The metadata area stores configuration information such as label information,
bad-blocks etc. The metadata area is located out-of-band of NVDIMM storage
area hence a separate access semantics is provided.

**H_SCM_WRITE_METADATA**

| Input: *drcIndex, offset, data, numBytesToWrite*
| Out: *None*
| Return Value: *H_Success, H_Parameter, H_P2, H_P4, H_Hardware*

Given a DRC Index of an NVDIMM, write N-bytes to the metadata area
associated with it, at the specified offset and from the provided buffer.

**H_SCM_BIND_MEM**

| Input: *drcIndex, startingScmBlockIndex, numScmBlocksToBind,*
| *targetLogicalMemoryAddress, continue-token*
| Out: *continue-token, targetLogicalMemoryAddress, numScmBlocksToBound*
| Return Value: *H_Success, H_Parameter, H_P2, H_P3, H_P4, H_Overlap,*
| *H_Too_Big, H_P5, H_Busy*

Given a DRC-Index of an NVDIMM, map a continuous SCM blocks range
*(startingScmBlockIndex, startingScmBlockIndex+numScmBlocksToBind)* to the guest
at *targetLogicalMemoryAddress* within guest physical address space. In
case *targetLogicalMemoryAddress == 0xFFFFFFFF_FFFFFFFF* then hypervisor
assigns a target address to the guest. The HCALL can fail if the Guest has
an active PTE entry to the SCM block being bound.

**H_SCM_UNBIND_MEM**
| Input: drcIndex, startingScmLogicalMemoryAddress, numScmBlocksToUnbind
| Out: numScmBlocksUnbound
| Return Value: *H_Success, H_Parameter, H_P2, H_P3, H_In_Use, H_Overlap,*
| *H_Busy, H_LongBusyOrder1mSec, H_LongBusyOrder10mSec*

Given a DRC-Index of an NVDimm, unmap *numScmBlocksToUnbind* SCM blocks starting
at *startingScmLogicalMemoryAddress* from guest physical address space. The
HCALL can fail if the Guest has an active PTE entry to the SCM block being
unbound.

**H_SCM_QUERY_BLOCK_MEM_BINDING**

| Input: *drcIndex, scmBlockIndex*
| Out: *Guest-Physical-Address*
| Return Value: *H_Success, H_Parameter, H_P2, H_NotFound*

Given a DRC-Index and an SCM Block index return the guest physical address to
which the SCM block is mapped to.

**H_SCM_QUERY_LOGICAL_MEM_BINDING**

| Input: *Guest-Physical-Address*
| Out: *drcIndex, scmBlockIndex*
| Return Value: *H_Success, H_Parameter, H_P2, H_NotFound*

Given a guest physical address return which DRC Index and SCM block is mapped
to that address.

**H_SCM_UNBIND_ALL**

| Input: *scmTargetScope, drcIndex*
| Out: *None*
| Return Value: *H_Success, H_Parameter, H_P2, H_P3, H_In_Use, H_Busy,*
| *H_LongBusyOrder1mSec, H_LongBusyOrder10mSec*

Depending on the Target scope unmap all SCM blocks belonging to all NVDIMMs
or all SCM blocks belonging to a single NVDIMM identified by its drcIndex
from the LPAR memory.

**H_SCM_HEALTH**

| Input: drcIndex
| Out: *health-bitmap (r4), health-bit-valid-bitmap (r5)*
| Return Value: *H_Success, H_Parameter, H_Hardware*

Given a DRC Index return the info on predictive failure and overall health of
the PMEM device. The asserted bits in the health-bitmap indicate one or more states
(described in table below) of the PMEM device and health-bit-valid-bitmap indicate
which bits in health-bitmap are valid. The bits are reported in
reverse bit ordering for example a value of 0xC400000000000000
indicates bits 0, 1, and 5 are valid.

Health Bitmap Flags:

+------+-----------------------------------------------------------------------+
|  Bit |               Definition                                              |
+======+=======================================================================+
|  00  | PMEM device is unable to persist memory contents.                     |
|      | If the system is powered down, nothing will be saved.                 |
+------+-----------------------------------------------------------------------+
|  01  | PMEM device failed to persist memory contents. Either contents were   |
|      | not saved successfully on power down or were not restored properly on |
|      | power up.                                                             |
+------+-----------------------------------------------------------------------+
|  02  | PMEM device contents are persisted from previous IPL. The data from   |
|      | the last boot were successfully restored.                             |
+------+-----------------------------------------------------------------------+
|  03  | PMEM device contents are not persisted from previous IPL. There was no|
|      | data to restore from the last boot.                                   |
+------+-----------------------------------------------------------------------+
|  04  | PMEM device memory life remaining is critically low                   |
+------+-----------------------------------------------------------------------+
|  05  | PMEM device will be garded off next IPL due to failure                |
+------+-----------------------------------------------------------------------+
|  06  | PMEM device contents cannot persist due to current platform health    |
|      | status. A hardware failure may prevent data from being saved or       |
|      | restored.                                                             |
+------+-----------------------------------------------------------------------+
|  07  | PMEM device is unable to persist memory contents in certain conditions|
+------+-----------------------------------------------------------------------+
|  08  | PMEM device is encrypted                                              |
+------+-----------------------------------------------------------------------+
|  09  | PMEM device has successfully completed a requested erase or secure    |
|      | erase procedure.                                                      |
+------+-----------------------------------------------------------------------+
|10:63 | Reserved / Unused                                                     |
+------+-----------------------------------------------------------------------+

**H_SCM_PERFORMANCE_STATS**

| Input: drcIndex, resultBuffer Addr
| Out: None
| Return Value:  *H_Success, H_Parameter, H_Unsupported, H_Hardware, H_Authority, H_Privilege*

Given a DRC Index collect the performance statistics for NVDIMM and copy them
to the resultBuffer.

**H_SCM_FLUSH**

| Input: *drcIndex, continue-token*
| Out: *continue-token*
| Return Value: *H_SUCCESS, H_Parameter, H_P2, H_BUSY*

Given a DRC Index Flush the data to backend NVDIMM device.

The hcall returns H_BUSY when the flush takes longer time and the hcall needs
to be issued multiple times in order to be completely serviced. The
*continue-token* from the output to be passed in the argument list of
subsequent hcalls to the hypervisor until the hcall is completely serviced
at which point H_SUCCESS or other error is returned by the hypervisor.

References
==========
.. [1] "Power Architecture Platform Reference"
       https://en.wikipedia.org/wiki/Power_Architecture_Platform_Reference
.. [2] "Linux on Power Architecture Platform Reference"
       https://members.openpowerfoundation.org/document/dl/469
.. [3] "Definitions and Notation" Book III-Section 14.5.3
       https://openpowerfoundation.org/?resource_lib=power-isa-version-3-0
.. [4] arch/powerpc/include/asm/hvcall.h
.. [5] "64-Bit ELF V2 ABI Specification: Power Architecture"
       https://openpowerfoundation.org/?resource_lib=64-bit-elf-v2-abi-specification-power-architecture


code:
===================================================
PCI Express I/O Virtualization Resource on Powerenv
===================================================

Wei Yang <weiyang@linux.vnet.ibm.com>

Benjamin Herrenschmidt <benh@au1.ibm.com>

Bjorn Helgaas <bhelgaas@google.com>

26 Aug 2014

This document describes the requirement from hardware for PCI MMIO resource
sizing and assignment on PowerKVM and how generic PCI code handles this
requirement. The first two sections describe the concepts of Partitionable
Endpoints and the implementation on P8 (IODA2). The next two sections talks
about considerations on enabling SRIOV on IODA2.

1. Introduction to Partitionable Endpoints
==========================================

A Partitionable Endpoint (PE) is a way to group the various resources
associated with a device or a set of devices to provide isolation between
partitions (i.e., filtering of DMA, MSIs etc.) and to provide a mechanism
to freeze a device that is causing errors in order to limit the possibility
of propagation of bad data.

There is thus, in HW, a table of PE states that contains a pair of "frozen"
state bits (one for MMIO and one for DMA, they get set together but can be
cleared independently) for each PE.

When a PE is frozen, all stores in any direction are dropped and all loads
return all 1's value. MSIs are also blocked. There's a bit more state that
captures things like the details of the error that caused the freeze etc., but
that's not critical.

The interesting part is how the various PCIe transactions (MMIO, DMA, ...)
are matched to their corresponding PEs.

The following section provides a rough description of what we have on P8
(IODA2).  Keep in mind that this is all per PHB (PCI host bridge).  Each PHB
is a completely separate HW entity that replicates the entire logic, so has
its own set of PEs, etc.

2. Implementation of Partitionable Endpoints on P8 (IODA2)
==========================================================

P8 supports up to 256 Partitionable Endpoints per PHB.

  * Inbound

    For DMA, MSIs and inbound PCIe error messages, we have a table (in
    memory but accessed in HW by the chip) that provides a direct
    correspondence between a PCIe RID (bus/dev/fn) with a PE number.
    We call this the RTT.

    - For DMA we then provide an entire address space for each PE that can
      contain two "windows", depending on the value of PCI address bit 59.
      Each window can be configured to be remapped via a "TCE table" (IOMMU
      translation table), which has various configurable characteristics
      not described here.

    - For MSIs, we have two windows in the address space (one at the top of
      the 32-bit space and one much higher) which, via a combination of the
      address and MSI value, will result in one of the 2048 interrupts per
      bridge being triggered.  There's a PE# in the interrupt controller
      descriptor table as well which is compared with the PE# obtained from
      the RTT to "authorize" the device to emit that specific interrupt.

    - Error messages just use the RTT.

  * Outbound.  That's where the tricky part is.

    Like other PCI host bridges, the Power8 IODA2 PHB supports "windows"
    from the CPU address space to the PCI address space.  There is one M32
    window and sixteen M64 windows.  They have different characteristics.
    First what they have in common: they forward a configurable portion of
    the CPU address space to the PCIe bus and must be naturally aligned
    power of two in size.  The rest is different:

    - The M32 window:

      * Is limited to 4GB in size.

      * Drops the top bits of the address (above the size) and replaces
	them with a configurable value.  This is typically used to generate
	32-bit PCIe accesses.  We configure that window at boot from FW and
	don't touch it from Linux; it's usually set to forward a 2GB
	portion of address space from the CPU to PCIe
	0x8000_0000..0xffff_ffff.  (Note: The top 64KB are actually
	reserved for MSIs but this is not a problem at this point; we just
	need to ensure Linux doesn't assign anything there, the M32 logic
	ignores that however and will forward in that space if we try).

      * It is divided into 256 segments of equal size.  A table in the chip
	maps each segment to a PE#.  That allows portions of the MMIO space
	to be assigned to PEs on a segment granularity.  For a 2GB window,
	the segment granularity is 2GB/256 = 8MB.

    Now, this is the "main" window we use in Linux today (excluding
    SR-IOV).  We basically use the trick of forcing the bridge MMIO windows
    onto a segment alignment/granularity so that the space behind a bridge
    can be assigned to a PE.

    Ideally we would like to be able to have individual functions in PEs
    but that would mean using a completely different address allocation
    scheme where individual function BARs can be "grouped" to fit in one or
    more segments.

    - The M64 windows:

      * Must be at least 256MB in size.

      * Do not translate addresses (the address on PCIe is the same as the
	address on the PowerBus).  There is a way to also set the top 14
	bits which are not conveyed by PowerBus but we don't use this.

      * Can be configured to be segmented.  When not segmented, we can
	specify the PE# for the entire window.  When segmented, a window
	has 256 segments; however, there is no table for mapping a segment
	to a PE#.  The segment number *is* the PE#.

      * Support overlaps.  If an address is covered by multiple windows,
	there's a defined ordering for which window applies.

    We have code (fairly new compared to the M32 stuff) that exploits that
    for large BARs in 64-bit space:

    We configure an M64 window to cover the entire region of address space
    that has been assigned by FW for the PHB (about 64GB, ignore the space
    for the M32, it comes out of a different "reserve").  We configure it
    as segmented.

    Then we do the same thing as with M32, using the bridge alignment
    trick, to match to those giant segments.

    Since we cannot remap, we have two additional constraints:

    - We do the PE# allocation *after* the 64-bit space has been assigned
      because the addresses we use directly determine the PE#.  We then
      update the M32 PE# for the devices that use both 32-bit and 64-bit
      spaces or assign the remaining PE# to 32-bit only devices.

    - We cannot "group" segments in HW, so if a device ends up using more
      than one segment, we end up with more than one PE#.  There is a HW
      mechanism to make the freeze state cascade to "companion" PEs but
      that only works for PCIe error messages (typically used so that if
      you freeze a switch, it freezes all its children).  So we do it in
      SW.  We lose a bit of effectiveness of EEH in that case, but that's
      the best we found.  So when any of the PEs freezes, we freeze the
      other ones for that "domain".  We thus introduce the concept of
      "master PE" which is the one used for DMA, MSIs, etc., and "secondary
      PEs" that are used for the remaining M64 segments.

    We would like to investigate using additional M64 windows in "single
    PE" mode to overlay over specific BARs to work around some of that, for
    example for devices with very large BARs, e.g., GPUs.  It would make
    sense, but we haven't done it yet.

3. Considerations for SR-IOV on PowerKVM
========================================

  * SR-IOV Background

    The PCIe SR-IOV feature allows a single Physical Function (PF) to
    support several Virtual Functions (VFs).  Registers in the PF's SR-IOV
    Capability control the number of VFs and whether they are enabled.

    When VFs are enabled, they appear in Configuration Space like normal
    PCI devices, but the BARs in VF config space headers are unusual.  For
    a non-VF device, software uses BARs in the config space header to
    discover the BAR sizes and assign addresses for them.  For VF devices,
    software uses VF BAR registers in the *PF* SR-IOV Capability to
    discover sizes and assign addresses.  The BARs in the VF's config space
    header are read-only zeros.

    When a VF BAR in the PF SR-IOV Capability is programmed, it sets the
    base address for all the corresponding VF(n) BARs.  For example, if the
    PF SR-IOV Capability is programmed to enable eight VFs, and it has a
    1MB VF BAR0, the address in that VF BAR sets the base of an 8MB region.
    This region is divided into eight contiguous 1MB regions, each of which
    is a BAR0 for one of the VFs.  Note that even though the VF BAR
    describes an 8MB region, the alignment requirement is for a single VF,
    i.e., 1MB in this example.

  There are several strategies for isolating VFs in PEs:

  - M32 window: There's one M32 window, and it is split into 256
    equally-sized segments.  The finest granularity possible is a 256MB
    window with 1MB segments.  VF BARs that are 1MB or larger could be
    mapped to separate PEs in this window.  Each segment can be
    individually mapped to a PE via the lookup table, so this is quite
    flexible, but it works best when all the VF BARs are the same size.  If
    they are different sizes, the entire window has to be small enough that
    the segment size matches the smallest VF BAR, which means larger VF
    BARs span several segments.

  - Non-segmented M64 window: A non-segmented M64 window is mapped entirely
    to a single PE, so it could only isolate one VF.

  - Single segmented M64 windows: A segmented M64 window could be used just
    like the M32 window, but the segments can't be individually mapped to
    PEs (the segment number is the PE#), so there isn't as much
    flexibility.  A VF with multiple BARs would have to be in a "domain" of
    multiple PEs, which is not as well isolated as a single PE.

  - Multiple segmented M64 windows: As usual, each window is split into 256
    equally-sized segments, and the segment number is the PE#.  But if we
    use several M64 windows, they can be set to different base addresses
    and different segment sizes.  If we have VFs that each have a 1MB BAR
    and a 32MB BAR, we could use one M64 window to assign 1MB segments and
    another M64 window to assign 32MB segments.

  Finally, the plan to use M64 windows for SR-IOV, which will be described
  more in the next two sections.  For a given VF BAR, we need to
  effectively reserve the entire 256 segments (256 * VF BAR size) and
  position the VF BAR to start at the beginning of a free range of
  segments/PEs inside that M64 window.

  The goal is of course to be able to give a separate PE for each VF.

  The IODA2 platform has 16 M64 windows, which are used to map MMIO
  range to PE#.  Each M64 window defines one MMIO range and this range is
  divided into 256 segments, with each segment corresponding to one PE.

  We decide to leverage this M64 window to map VFs to individual PEs, since
  SR-IOV VF BARs are all the same size.

  But doing so introduces another problem: total_VFs is usually smaller
  than the number of M64 window segments, so if we map one VF BAR directly
  to one M64 window, some part of the M64 window will map to another
  device's MMIO range.

  IODA supports 256 PEs, so segmented windows contain 256 segments, so if
  total_VFs is less than 256, we have the situation in Figure 1.0, where
  segments [total_VFs, 255] of the M64 window may map to some MMIO range on
  other devices::

     0      1                     total_VFs - 1
     +------+------+-     -+------+------+
     |      |      |  ...  |      |      |
     +------+------+-     -+------+------+

                           VF(n) BAR space

     0      1                     total_VFs - 1                255
     +------+------+-     -+------+------+-      -+------+------+
     |      |      |  ...  |      |      |   ...  |      |      |
     +------+------+-     -+------+------+-      -+------+------+

                           M64 window

		Figure 1.0 Direct map VF(n) BAR space

  Our current solution is to allocate 256 segments even if the VF(n) BAR
  space doesn't need that much, as shown in Figure 1.1::

     0      1                     total_VFs - 1                255
     +------+------+-     -+------+------+-      -+------+------+
     |      |      |  ...  |      |      |   ...  |      |      |
     +------+------+-     -+------+------+-      -+------+------+

                           VF(n) BAR space + extra

     0      1                     total_VFs - 1                255
     +------+------+-     -+------+------+-      -+------+------+
     |      |      |  ...  |      |      |   ...  |      |      |
     +------+------+-     -+------+------+-      -+------+------+

			   M64 window

		Figure 1.1 Map VF(n) BAR space + extra

  Allocating the extra space ensures that the entire M64 window will be
  assigned to this one SR-IOV device and none of the space will be
  available for other devices.  Note that this only expands the space
  reserved in software; there are still only total_VFs VFs, and they only
  respond to segments [0, total_VFs - 1].  There's nothing in hardware that
  responds to segments [total_VFs, 255].

4. Implications for the Generic PCI Code
========================================

The PCIe SR-IOV spec requires that the base of the VF(n) BAR space be
aligned to the size of an individual VF BAR.

In IODA2, the MMIO address determines the PE#.  If the address is in an M32
window, we can set the PE# by updating the table that translates segments
to PE#s.  Similarly, if the address is in an unsegmented M64 window, we can
set the PE# for the window.  But if it's in a segmented M64 window, the
segment number is the PE#.

Therefore, the only way to control the PE# for a VF is to change the base
of the VF(n) BAR space in the VF BAR.  If the PCI core allocates the exact
amount of space required for the VF(n) BAR space, the VF BAR value is fixed
and cannot be changed.

On the other hand, if the PCI core allocates additional space, the VF BAR
value can be changed as long as the entire VF(n) BAR space remains inside
the space allocated by the core.

Ideally the segment size will be the same as an individual VF BAR size.
Then each VF will be in its own PE.  The VF BARs (and therefore the PE#s)
are contiguous.  If VF0 is in PE(x), then VF(n) is in PE(x+n).  If we
allocate 256 segments, there are (256 - numVFs) choices for the PE# of VF0.

If the segment size is smaller than the VF BAR size, it will take several
segments to cover a VF BAR, and a VF will be in several PEs.  This is
possible, but the isolation isn't as good, and it reduces the number of PE#
choices because instead of consuming only numVFs segments, the VF(n) BAR
space will consume (numVFs * n) segments.  That means there aren't as many
available segments for adjusting base of the VF(n) BAR space.


code:
========================
PMU Event Based Branches
========================

Event Based Branches (EBBs) are a feature which allows the hardware to
branch directly to a specified user space address when certain events occur.

The full specification is available in Power ISA v2.07:

  https://www.power.org/documentation/power-isa-version-2-07/

One type of event for which EBBs can be configured is PMU exceptions. This
document describes the API for configuring the Power PMU to generate EBBs,
using the Linux perf_events API.

Terminology
-----------

Throughout this document we will refer to an "EBB event" or "EBB events". This
just refers to a struct perf_event which has set the "EBB" flag in its
attr.config. All events which can be configured on the hardware PMU are
possible "EBB events".

Background
----------

When a PMU EBB occurs it is delivered to the currently running process. As such
EBBs can only sensibly be used by programs for self-monitoring.

It is a feature of the perf_events API that events can be created on other
processes, subject to standard permission checks. This is also true of EBB
events, however unless the target process enables EBBs (via mtspr(BESCR)) no
EBBs will ever be delivered.

This makes it possible for a process to enable EBBs for itself, but not
actually configure any events. At a later time another process can come along
and attach an EBB event to the process, which will then cause EBBs to be
delivered to the first process. It's not clear if this is actually useful.

When the PMU is configured for EBBs, all PMU interrupts are delivered to the
user process. This means once an EBB event is scheduled on the PMU, no non-EBB
events can be configured. This means that EBB events can not be run
concurrently with regular 'perf' commands, or any other perf events.

It is however safe to run 'perf' commands on a process which is using EBBs. The
kernel will in general schedule the EBB event, and perf will be notified that
its events could not run.

The exclusion between EBB events and regular events is implemented using the
existing "pinned" and "exclusive" attributes of perf_events. This means EBB
events will be given priority over other events, unless they are also pinned.
If an EBB event and a regular event are both pinned, then whichever is enabled
first will be scheduled and the other will be put in error state. See the
section below titled "Enabling an EBB event" for more information.

Creating an EBB event
---------------------

To request that an event is counted using EBB, the event code should have bit
63 set.

EBB events must be created with a particular, and restrictive, set of
attributes - this is so that they interoperate correctly with the rest of the
perf_events subsystem.

An EBB event must be created with the "pinned" and "exclusive" attributes set.
Note that if you are creating a group of EBB events, only the leader can have
these attributes set.

An EBB event must NOT set any of the "inherit", "sample_period", "freq" or
"enable_on_exec" attributes.

An EBB event must be attached to a task. This is specified to perf_event_open()
by passing a pid value, typically 0 indicating the current task.

All events in a group must agree on whether they want EBB. That is all events
must request EBB, or none may request EBB.

EBB events must specify the PMC they are to be counted on. This ensures
userspace is able to reliably determine which PMC the event is scheduled on.

Enabling an EBB event
---------------------

Once an EBB event has been successfully opened, it must be enabled with the
perf_events API. This can be achieved either via the ioctl() interface, or the
prctl() interface.

However, due to the design of the perf_events API, enabling an event does not
guarantee that it has been scheduled on the PMU. To ensure that the EBB event
has been scheduled on the PMU, you must perform a read() on the event. If the
read() returns EOF, then the event has not been scheduled and EBBs are not
enabled.

This behaviour occurs because the EBB event is pinned and exclusive. When the
EBB event is enabled it will force all other non-pinned events off the PMU. In
this case the enable will be successful. However if there is already an event
pinned on the PMU then the enable will not be successful.

Reading an EBB event
--------------------

It is possible to read() from an EBB event. However the results are
meaningless. Because interrupts are being delivered to the user process the
kernel is not able to count the event, and so will return a junk value.

Closing an EBB event
--------------------

When an EBB event is finished with, you can close it using close() as for any
regular event. If this is the last EBB event the PMU will be deconfigured and
no further PMU EBBs will be delivered.

EBB Handler
-----------

The EBB handler is just regular userspace code, however it must be written in
the style of an interrupt handler. When the handler is entered all registers
are live (possibly) and so must be saved somehow before the handler can invoke
other code.

It's up to the program how to handle this. For C programs a relatively simple
option is to create an interrupt frame on the stack and save registers there.

Fork
----

EBB events are not inherited across fork. If the child process wishes to use
EBBs it should open a new event for itself. Similarly the EBB state in
BESCR/EBBHR/EBBRR is cleared across fork().


code:
======
Ptrace
======

GDB intends to support the following hardware debug features of BookE
processors:

4 hardware breakpoints (IAC)
2 hardware watchpoints (read, write and read-write) (DAC)
2 value conditions for the hardware watchpoints (DVC)

For that, we need to extend ptrace so that GDB can query and set these
resources. Since we're extending, we're trying to create an interface
that's extendable and that covers both BookE and server processors, so
that GDB doesn't need to special-case each of them. We added the
following 3 new ptrace requests.

1. PTRACE_PPC_GETHWDEBUGINFO
============================

Query for GDB to discover the hardware debug features. The main info to
be returned here is the minimum alignment for the hardware watchpoints.
BookE processors don't have restrictions here, but server processors have
an 8-byte alignment restriction for hardware watchpoints. We'd like to avoid
adding special cases to GDB based on what it sees in AUXV.

Since we're at it, we added other useful info that the kernel can return to
GDB: this query will return the number of hardware breakpoints, hardware
watchpoints and whether it supports a range of addresses and a condition.
The query will fill the following structure provided by the requesting process::

  struct ppc_debug_info {
       unit32_t version;
       unit32_t num_instruction_bps;
       unit32_t num_data_bps;
       unit32_t num_condition_regs;
       unit32_t data_bp_alignment;
       unit32_t sizeof_condition; /* size of the DVC register */
       uint64_t features; /* bitmask of the individual flags */
  };

features will have bits indicating whether there is support for::

  #define PPC_DEBUG_FEATURE_INSN_BP_RANGE		0x1
  #define PPC_DEBUG_FEATURE_INSN_BP_MASK		0x2
  #define PPC_DEBUG_FEATURE_DATA_BP_RANGE		0x4
  #define PPC_DEBUG_FEATURE_DATA_BP_MASK		0x8
  #define PPC_DEBUG_FEATURE_DATA_BP_DAWR		0x10
  #define PPC_DEBUG_FEATURE_DATA_BP_ARCH_31		0x20

2. PTRACE_SETHWDEBUG

Sets a hardware breakpoint or watchpoint, according to the provided structure::

  struct ppc_hw_breakpoint {
        uint32_t version;
  #define PPC_BREAKPOINT_TRIGGER_EXECUTE  0x1
  #define PPC_BREAKPOINT_TRIGGER_READ     0x2
 #define PPC_BREAKPOINT_TRIGGER_WRITE    0x4
        uint32_t trigger_type;       /* only some combinations allowed */
  #define PPC_BREAKPOINT_MODE_EXACT               0x0
  #define PPC_BREAKPOINT_MODE_RANGE_INCLUSIVE     0x1
  #define PPC_BREAKPOINT_MODE_RANGE_EXCLUSIVE     0x2
  #define PPC_BREAKPOINT_MODE_MASK                0x3
        uint32_t addr_mode;          /* address match mode */

  #define PPC_BREAKPOINT_CONDITION_MODE   0x3
  #define PPC_BREAKPOINT_CONDITION_NONE   0x0
  #define PPC_BREAKPOINT_CONDITION_AND    0x1
  #define PPC_BREAKPOINT_CONDITION_EXACT  0x1	/* different name for the same thing as above */
  #define PPC_BREAKPOINT_CONDITION_OR     0x2
  #define PPC_BREAKPOINT_CONDITION_AND_OR 0x3
  #define PPC_BREAKPOINT_CONDITION_BE_ALL 0x00ff0000	/* byte enable bits */
  #define PPC_BREAKPOINT_CONDITION_BE(n)  (1<<((n)+16))
        uint32_t condition_mode;     /* break/watchpoint condition flags */

        uint64_t addr;
        uint64_t addr2;
        uint64_t condition_value;
  };

A request specifies one event, not necessarily just one register to be set.
For instance, if the request is for a watchpoint with a condition, both the
DAC and DVC registers will be set in the same request.

With this GDB can ask for all kinds of hardware breakpoints and watchpoints
that the BookE supports. COMEFROM breakpoints available in server processors
are not contemplated, but that is out of the scope of this work.

ptrace will return an integer (handle) uniquely identifying the breakpoint or
watchpoint just created. This integer will be used in the PTRACE_DELHWDEBUG
request to ask for its removal. Return -ENOSPC if the requested breakpoint
can't be allocated on the registers.

Some examples of using the structure to:

- set a breakpoint in the first breakpoint register::

    p.version         = PPC_DEBUG_CURRENT_VERSION;
    p.trigger_type    = PPC_BREAKPOINT_TRIGGER_EXECUTE;
    p.addr_mode       = PPC_BREAKPOINT_MODE_EXACT;
    p.condition_mode  = PPC_BREAKPOINT_CONDITION_NONE;
    p.addr            = (uint64_t) address;
    p.addr2           = 0;
    p.condition_value = 0;

- set a watchpoint which triggers on reads in the second watchpoint register::

    p.version         = PPC_DEBUG_CURRENT_VERSION;
    p.trigger_type    = PPC_BREAKPOINT_TRIGGER_READ;
    p.addr_mode       = PPC_BREAKPOINT_MODE_EXACT;
    p.condition_mode  = PPC_BREAKPOINT_CONDITION_NONE;
    p.addr            = (uint64_t) address;
    p.addr2           = 0;
    p.condition_value = 0;

- set a watchpoint which triggers only with a specific value::

    p.version         = PPC_DEBUG_CURRENT_VERSION;
    p.trigger_type    = PPC_BREAKPOINT_TRIGGER_READ;
    p.addr_mode       = PPC_BREAKPOINT_MODE_EXACT;
    p.condition_mode  = PPC_BREAKPOINT_CONDITION_AND | PPC_BREAKPOINT_CONDITION_BE_ALL;
    p.addr            = (uint64_t) address;
    p.addr2           = 0;
    p.condition_value = (uint64_t) condition;

- set a ranged hardware breakpoint::

    p.version         = PPC_DEBUG_CURRENT_VERSION;
    p.trigger_type    = PPC_BREAKPOINT_TRIGGER_EXECUTE;
    p.addr_mode       = PPC_BREAKPOINT_MODE_RANGE_INCLUSIVE;
    p.condition_mode  = PPC_BREAKPOINT_CONDITION_NONE;
    p.addr            = (uint64_t) begin_range;
    p.addr2           = (uint64_t) end_range;
    p.condition_value = 0;

- set a watchpoint in server processors (BookS)::

    p.version         = 1;
    p.trigger_type    = PPC_BREAKPOINT_TRIGGER_RW;
    p.addr_mode       = PPC_BREAKPOINT_MODE_RANGE_INCLUSIVE;
    or
    p.addr_mode       = PPC_BREAKPOINT_MODE_EXACT;

    p.condition_mode  = PPC_BREAKPOINT_CONDITION_NONE;
    p.addr            = (uint64_t) begin_range;
    /* For PPC_BREAKPOINT_MODE_RANGE_INCLUSIVE addr2 needs to be specified, where
     * addr2 - addr <= 8 Bytes.
     */
    p.addr2           = (uint64_t) end_range;
    p.condition_value = 0;

3. PTRACE_DELHWDEBUG

Takes an integer which identifies an existing breakpoint or watchpoint
(i.e., the value returned from PTRACE_SETHWDEBUG), and deletes the
corresponding breakpoint or watchpoint..


code:
=========================================
Freescale QUICC Engine Firmware Uploading
=========================================

(c) 2007 Timur Tabi <timur at freescale.com>,
    Freescale Semiconductor

.. Table of Contents

   I - Software License for Firmware

   II - Microcode Availability

   III - Description and Terminology

   IV - Microcode Programming Details

   V - Firmware Structure Layout

   VI - Sample Code for Creating Firmware Files

Revision Information
====================

November 30, 2007: Rev 1.0 - Initial version

I - Software License for Firmware
=================================

Each firmware file comes with its own software license.  For information on
the particular license, please see the license text that is distributed with
the firmware.

II - Microcode Availability
===========================

Firmware files are distributed through various channels.  Some are available on
http://opensource.freescale.com.  For other firmware files, please contact
your Freescale representative or your operating system vendor.

III - Description and Terminology
=================================

In this document, the term 'microcode' refers to the sequence of 32-bit
integers that compose the actual QE microcode.

The term 'firmware' refers to a binary blob that contains the microcode as
well as other data that

	1) describes the microcode's purpose
	2) describes how and where to upload the microcode
	3) specifies the values of various registers
	4) includes additional data for use by specific device drivers

Firmware files are binary files that contain only a firmware.

IV - Microcode Programming Details
===================================

The QE architecture allows for only one microcode present in I-RAM for each
RISC processor.  To replace any current microcode, a full QE reset (which
disables the microcode) must be performed first.

QE microcode is uploaded using the following procedure:

1) The microcode is placed into I-RAM at a specific location, using the
   IRAM.IADD and IRAM.IDATA registers.

2) The CERCR.CIR bit is set to 0 or 1, depending on whether the firmware
   needs split I-RAM.  Split I-RAM is only meaningful for SOCs that have
   QEs with multiple RISC processors, such as the 8360.  Splitting the I-RAM
   allows each processor to run a different microcode, effectively creating an
   asymmetric multiprocessing (AMP) system.

3) The TIBCR trap registers are loaded with the addresses of the trap handlers
   in the microcode.

4) The RSP.ECCR register is programmed with the value provided.

5) If necessary, device drivers that need the virtual traps and extended mode
   data will use them.

Virtual Microcode Traps

These virtual traps are conditional branches in the microcode.  These are
"soft" provisional introduced in the ROMcode in order to enable higher
flexibility and save h/w traps If new features are activated or an issue is
being fixed in the RAM package utilizing they should be activated.  This data
structure signals the microcode which of these virtual traps is active.

This structure contains 6 words that the application should copy to some
specific been defined.  This table describes the structure::

	---------------------------------------------------------------
	| Offset in |                  | Destination Offset | Size of |
	|   array   |     Protocol     |   within PRAM      | Operand |
	--------------------------------------------------------------|
	|     0     | Ethernet         |      0xF8          | 4 bytes |
	|           | interworking     |                    |         |
	---------------------------------------------------------------
	|     4     | ATM              |      0xF8          | 4 bytes |
	|           | interworking     |                    |         |
	---------------------------------------------------------------
	|     8     | PPP              |      0xF8          | 4 bytes |
	|           | interworking     |                    |         |
	---------------------------------------------------------------
	|     12    | Ethernet RX      |      0x22          | 1 byte  |
	|           | Distributor Page |                    |         |
	---------------------------------------------------------------
	|     16    | ATM Globtal      |      0x28          | 1 byte  |
	|           | Params Table     |                    |         |
	---------------------------------------------------------------
	|     20    | Insert Frame     |      0xF8          | 4 bytes |
	---------------------------------------------------------------

Extended Modes

This is a double word bit array (64 bits) that defines special functionality
which has an impact on the software drivers.  Each bit has its own impact
and has special instructions for the s/w associated with it.  This structure is
described in this table::

	-----------------------------------------------------------------------
	| Bit #  |     Name     |   Description                               |
	-----------------------------------------------------------------------
	|   0    | General      | Indicates that prior to each host command   |
	|        | push command | given by the application, the software must |
	|        |              | assert a special host command (push command)|
	|        |              | CECDR = 0x00800000.                         |
	|        |              | CECR = 0x01c1000f.                          |
	-----------------------------------------------------------------------
	|   1    | UCC ATM      | Indicates that after issuing ATM RX INIT    |
	|        | RX INIT      | command, the host must issue another special|
	|        | push command | command (push command) and immediately      |
	|        |              | following that re-issue the ATM RX INIT     |
	|        |              | command. (This makes the sequence of        |
	|        |              | initializing the ATM receiver a sequence of |
	|        |              | three host commands)                        |
	|        |              | CECDR = 0x00800000.                         |
	|        |              | CECR = 0x01c1000f.                          |
	-----------------------------------------------------------------------
	|   2    | Add/remove   | Indicates that following the specific host  |
	|        | command      | command: "Add/Remove entry in Hash Lookup   |
	|        | validation   | Table" used in Interworking setup, the user |
	|        |              | must issue another command.                 |
	|        |              | CECDR = 0xce000003.                         |
	|        |              | CECR = 0x01c10f58.                          |
	-----------------------------------------------------------------------
	|   3    | General push | Indicates that the s/w has to initialize    |
	|        | command      | some pointers in the Ethernet thread pages  |
	|        |              | which are used when Header Compression is   |
	|        |              | activated.  The full details of these       |
	|        |              | pointers is located in the software drivers.|
	-----------------------------------------------------------------------
	|   4    | General push | Indicates that after issuing Ethernet TX    |
	|        | command      | INIT command, user must issue this command  |
	|        |              | for each SNUM of Ethernet TX thread.        |
	|        |              | CECDR = 0x00800003.                         |
	|        |              | CECR = 0x7'b{0}, 8'b{Enet TX thread SNUM},  |
	|        |              |        1'b{1}, 12'b{0}, 4'b{1}              |
	-----------------------------------------------------------------------
	| 5 - 31 |     N/A      | Reserved, set to zero.                      |
	-----------------------------------------------------------------------

V - Firmware Structure Layout
==============================

QE microcode from Freescale is typically provided as a header file.  This
header file contains macros that define the microcode binary itself as well as
some other data used in uploading that microcode.  The format of these files
do not lend themselves to simple inclusion into other code.  Hence,
the need for a more portable format.  This section defines that format.

Instead of distributing a header file, the microcode and related data are
embedded into a binary blob.  This blob is passed to the qe_upload_firmware()
function, which parses the blob and performs everything necessary to upload
the microcode.

All integers are big-endian.  See the comments for function
qe_upload_firmware() for up-to-date implementation information.

This structure supports versioning, where the version of the structure is
embedded into the structure itself.  To ensure forward and backwards
compatibility, all versions of the structure must use the same 'qe_header'
structure at the beginning.

'header' (type: struct qe_header):
	The 'length' field is the size, in bytes, of the entire structure,
	including all the microcode embedded in it, as well as the CRC (if
	present).

	The 'magic' field is an array of three bytes that contains the letters
	'Q', 'E', and 'F'.  This is an identifier that indicates that this
	structure is a QE Firmware structure.

	The 'version' field is a single byte that indicates the version of this
	structure.  If the layout of the structure should ever need to be
	changed to add support for additional types of microcode, then the
	version number should also be changed.

The 'id' field is a null-terminated string(suitable for printing) that
identifies the firmware.

The 'count' field indicates the number of 'microcode' structures.  There
must be one and only one 'microcode' structure for each RISC processor.
Therefore, this field also represents the number of RISC processors for this
SOC.

The 'soc' structure contains the SOC numbers and revisions used to match
the microcode to the SOC itself.  Normally, the microcode loader should
check the data in this structure with the SOC number and revisions, and
only upload the microcode if there's a match.  However, this check is not
made on all platforms.

Although it is not recommended, you can specify '0' in the soc.model
field to skip matching SOCs altogether.

The 'model' field is a 16-bit number that matches the actual SOC. The
'major' and 'minor' fields are the major and minor revision numbers,
respectively, of the SOC.

For example, to match the 8323, revision 1.0::

     soc.model = 8323
     soc.major = 1
     soc.minor = 0

'padding' is necessary for structure alignment.  This field ensures that the
'extended_modes' field is aligned on a 64-bit boundary.

'extended_modes' is a bitfield that defines special functionality which has an
impact on the device drivers.  Each bit has its own impact and has special
instructions for the driver associated with it.  This field is stored in
the QE library and available to any driver that calles qe_get_firmware_info().

'vtraps' is an array of 8 words that contain virtual trap values for each
virtual traps.  As with 'extended_modes', this field is stored in the QE
library and available to any driver that calles qe_get_firmware_info().

'microcode' (type: struct qe_microcode):
	For each RISC processor there is one 'microcode' structure.  The first
	'microcode' structure is for the first RISC, and so on.

	The 'id' field is a null-terminated string suitable for printing that
	identifies this particular microcode.

	'traps' is an array of 16 words that contain hardware trap values
	for each of the 16 traps.  If trap[i] is 0, then this particular
	trap is to be ignored (i.e. not written to TIBCR[i]).  The entire value
	is written as-is to the TIBCR[i] register, so be sure to set the EN
	and T_IBP bits if necessary.

	'eccr' is the value to program into the ECCR register.

	'iram_offset' is the offset into IRAM to start writing the
	microcode.

	'count' is the number of 32-bit words in the microcode.

	'code_offset' is the offset, in bytes, from the beginning of this
	structure where the microcode itself can be found.  The first
	microcode binary should be located immediately after the 'microcode'
	array.

	'major', 'minor', and 'revision' are the major, minor, and revision
	version numbers, respectively, of the microcode.  If all values are 0,
	then these fields are ignored.

	'reserved' is necessary for structure alignment.  Since 'microcode'
	is an array, the 64-bit 'extended_modes' field needs to be aligned
	on a 64-bit boundary, and this can only happen if the size of
	'microcode' is a multiple of 8 bytes.  To ensure that, we add
	'reserved'.

After the last microcode is a 32-bit CRC.  It can be calculated using
this algorithm::

  u32 crc32(const u8 *p, unsigned int len)
  {
	unsigned int i;
	u32 crc = 0;

	while (len--) {
	   crc ^= *p++;
	   for (i = 0; i < 8; i++)
		   crc = (crc >> 1) ^ ((crc & 1) ? 0xedb88320 : 0);
	}
	return crc;
  }

VI - Sample Code for Creating Firmware Files
============================================

A Python program that creates firmware binaries from the header files normally
distributed by Freescale can be found on http://opensource.freescale.com.


code:
===============================================
Power Architecture 64-bit Linux system call ABI
===============================================

syscall
=======

Invocation
----------
The syscall is made with the sc instruction, and returns with execution
continuing at the instruction following the sc instruction.

If PPC_FEATURE2_SCV appears in the AT_HWCAP2 ELF auxiliary vector, the
scv 0 instruction is an alternative that may provide better performance,
with some differences to calling sequence.

syscall calling sequence\ [1]_ matches the Power Architecture 64-bit ELF ABI
specification C function calling sequence, including register preservation
rules, with the following differences.

.. [1] Some syscalls (typically low-level management functions) may have
       different calling sequences (e.g., rt_sigreturn).

Parameters
----------
The system call number is specified in r0.

There is a maximum of 6 integer parameters to a syscall, passed in r3-r8.

Return value
------------
- For the sc instruction, both a value and an error condition are returned.
  cr0.SO is the error condition, and r3 is the return value. When cr0.SO is
  clear, the syscall succeeded and r3 is the return value. When cr0.SO is set,
  the syscall failed and r3 is the error value (that normally corresponds to
  errno).

- For the scv 0 instruction, the return value indicates failure if it is
  -4095..-1 (i.e., it is >= -MAX_ERRNO (-4095) as an unsigned comparison),
  in which case the error value is the negated return value.

Stack
-----
System calls do not modify the caller's stack frame. For example, the caller's
stack frame LR and CR save fields are not used.

Register preservation rules
---------------------------
Register preservation rules match the ELF ABI calling sequence with some
differences.

For the sc instruction, the differences from the ELF ABI are as follows:

+--------------+--------------------+-----------------------------------------+
| Register     | Preservation Rules | Purpose                                 |
+==============+====================+=========================================+
| r0           | Volatile           | (System call number.)                   |
+--------------+--------------------+-----------------------------------------+
| r3           | Volatile           | (Parameter 1, and return value.)        |
+--------------+--------------------+-----------------------------------------+
| r4-r8        | Volatile           | (Parameters 2-6.)                       |
+--------------+--------------------+-----------------------------------------+
| cr0          | Volatile           | (cr0.SO is the return error condition.) |
+--------------+--------------------+-----------------------------------------+
| cr1, cr5-7   | Nonvolatile        |                                         |
+--------------+--------------------+-----------------------------------------+
| lr           | Nonvolatile        |                                         |
+--------------+--------------------+-----------------------------------------+

For the scv 0 instruction, the differences from the ELF ABI are as follows:

+--------------+--------------------+-----------------------------------------+
| Register     | Preservation Rules | Purpose                                 |
+==============+====================+=========================================+
| r0           | Volatile           | (System call number.)                   |
+--------------+--------------------+-----------------------------------------+
| r3           | Volatile           | (Parameter 1, and return value.)        |
+--------------+--------------------+-----------------------------------------+
| r4-r8        | Volatile           | (Parameters 2-6.)                       |
+--------------+--------------------+-----------------------------------------+

All floating point and vector data registers as well as control and status
registers are nonvolatile.

Transactional Memory
--------------------
Syscall behavior can change if the processor is in transactional or suspended
transaction state, and the syscall can affect the behavior of the transaction.

If the processor is in suspended state when a syscall is made, the syscall
will be performed as normal, and will return as normal. The syscall will be
performed in suspended state, so its side effects will be persistent according
to the usual transactional memory semantics. A syscall may or may not result
in the transaction being doomed by hardware.

If the processor is in transactional state when a syscall is made, then the
behavior depends on the presence of PPC_FEATURE2_HTM_NOSC in the AT_HWCAP2 ELF
auxiliary vector.

- If present, which is the case for newer kernels, then the syscall will not
  be performed and the transaction will be doomed by the kernel with the
  failure code TM_CAUSE_SYSCALL | TM_CAUSE_PERSISTENT in the TEXASR SPR.

- If not present (older kernels), then the kernel will suspend the
  transactional state and the syscall will proceed as in the case of a
  suspended state syscall, and will resume the transactional state before
  returning to the caller. This case is not well defined or supported, so this
  behavior should not be relied upon.

scv 0 syscalls will always behave as PPC_FEATURE2_HTM_NOSC.

ptrace
------
When ptracing system calls (PTRACE_SYSCALL), the pt_regs.trap value contains
the system call type that can be used to distinguish between sc and scv 0
system calls, and the different register conventions can be accounted for.

If the value of (pt_regs.trap & 0xfff0) is 0xc00 then the system call was
performed with the sc instruction, if it is 0x3000 then the system call was
performed with the scv 0 instruction.

vsyscall
========

vsyscall calling sequence matches the syscall calling sequence, with the
following differences. Some vsyscalls may have different calling sequences.

Parameters and return value
---------------------------
r0 is not used as an input. The vsyscall is selected by its address.

Stack
-----
The vsyscall may or may not use the caller's stack frame save areas.

Register preservation rules
---------------------------

=========== ========
r0          Volatile
cr1, cr5-7  Volatile
lr          Volatile
=========== ========

Invocation
----------
The vsyscall is performed with a branch-with-link instruction to the vsyscall
function address.

Transactional Memory
--------------------
vsyscalls will run in the same transactional state as the caller. A vsyscall
may or may not result in the transaction being doomed by hardware.


code:
============================
Transactional Memory support
============================

POWER kernel support for this feature is currently limited to supporting
its use by user programs.  It is not currently used by the kernel itself.

This file aims to sum up how it is supported by Linux and what behaviour you
can expect from your user programs.

Basic overview
==============

Hardware Transactional Memory is supported on POWER8 processors, and is a
feature that enables a different form of atomic memory access.  Several new
instructions are presented to delimit transactions; transactions are
guaranteed to either complete atomically or roll back and undo any partial
changes.

A simple transaction looks like this::

  begin_move_money:
    tbegin
    beq   abort_handler

    ld    r4, SAVINGS_ACCT(r3)
    ld    r5, CURRENT_ACCT(r3)
    subi  r5, r5, 1
    addi  r4, r4, 1
    std   r4, SAVINGS_ACCT(r3)
    std   r5, CURRENT_ACCT(r3)

    tend

    b     continue

  abort_handler:
    ... test for odd failures ...

    /* Retry the transaction if it failed because it conflicted with
     * someone else: */
    b     begin_move_money

The 'tbegin' instruction denotes the start point, and 'tend' the end point.
Between these points the processor is in 'Transactional' state; any memory
references will complete in one go if there are no conflicts with other
transactional or non-transactional accesses within the system.  In this
example, the transaction completes as though it were normal straight-line code
IF no other processor has touched SAVINGS_ACCT(r3) or CURRENT_ACCT(r3); an
atomic move of money from the current account to the savings account has been
performed.  Even though the normal ld/std instructions are used (note no
lwarx/stwcx), either *both* SAVINGS_ACCT(r3) and CURRENT_ACCT(r3) will be
updated, or neither will be updated.

If, in the meantime, there is a conflict with the locations accessed by the
transaction, the transaction will be aborted by the CPU.  Register and memory
state will roll back to that at the 'tbegin', and control will continue from
'tbegin+4'.  The branch to abort_handler will be taken this second time; the
abort handler can check the cause of the failure, and retry.

Checkpointed registers include all GPRs, FPRs, VRs/VSRs, LR, CCR/CR, CTR, FPCSR
and a few other status/flag regs; see the ISA for details.

Causes of transaction aborts
============================

- Conflicts with cache lines used by other processors
- Signals
- Context switches
- See the ISA for full documentation of everything that will abort transactions.

Syscalls
========

Syscalls made from within an active transaction will not be performed and the
transaction will be doomed by the kernel with the failure code TM_CAUSE_SYSCALL
| TM_CAUSE_PERSISTENT.

Syscalls made from within a suspended transaction are performed as normal and
the transaction is not explicitly doomed by the kernel.  However, what the
kernel does to perform the syscall may result in the transaction being doomed
by the hardware.  The syscall is performed in suspended mode so any side
effects will be persistent, independent of transaction success or failure.  No
guarantees are provided by the kernel about which syscalls will affect
transaction success.

Care must be taken when relying on syscalls to abort during active transactions
if the calls are made via a library.  Libraries may cache values (which may
give the appearance of success) or perform operations that cause transaction
failure before entering the kernel (which may produce different failure codes).
Examples are glibc's getpid() and lazy symbol resolution.

Signals
=======

Delivery of signals (both sync and async) during transactions provides a second
thread state (ucontext/mcontext) to represent the second transactional register
state.  Signal delivery 'treclaim's to capture both register states, so signals
abort transactions.  The usual ucontext_t passed to the signal handler
represents the checkpointed/original register state; the signal appears to have
arisen at 'tbegin+4'.

If the sighandler ucontext has uc_link set, a second ucontext has been
delivered.  For future compatibility the MSR.TS field should be checked to
determine the transactional state -- if so, the second ucontext in uc->uc_link
represents the active transactional registers at the point of the signal.

For 64-bit processes, uc->uc_mcontext.regs->msr is a full 64-bit MSR and its TS
field shows the transactional mode.

For 32-bit processes, the mcontext's MSR register is only 32 bits; the top 32
bits are stored in the MSR of the second ucontext, i.e. in
uc->uc_link->uc_mcontext.regs->msr.  The top word contains the transactional
state TS.

However, basic signal handlers don't need to be aware of transactions
and simply returning from the handler will deal with things correctly:

Transaction-aware signal handlers can read the transactional register state
from the second ucontext.  This will be necessary for crash handlers to
determine, for example, the address of the instruction causing the SIGSEGV.

Example signal handler::

    void crash_handler(int sig, siginfo_t *si, void *uc)
    {
      ucontext_t *ucp = uc;
      ucontext_t *transactional_ucp = ucp->uc_link;

      if (ucp_link) {
        u64 msr = ucp->uc_mcontext.regs->msr;
        /* May have transactional ucontext! */
  #ifndef __powerpc64__
        msr |= ((u64)transactional_ucp->uc_mcontext.regs->msr) << 32;
  #endif
        if (MSR_TM_ACTIVE(msr)) {
           /* Yes, we crashed during a transaction.  Oops. */
   fprintf(stderr, "Transaction to be restarted at 0x%llx, but "
                           "crashy instruction was at 0x%llx\n",
                           ucp->uc_mcontext.regs->nip,
                           transactional_ucp->uc_mcontext.regs->nip);
        }
      }

      fix_the_problem(ucp->dar);
    }

When in an active transaction that takes a signal, we need to be careful with
the stack.  It's possible that the stack has moved back up after the tbegin.
The obvious case here is when the tbegin is called inside a function that
returns before a tend.  In this case, the stack is part of the checkpointed
transactional memory state.  If we write over this non transactionally or in
suspend, we are in trouble because if we get a tm abort, the program counter and
stack pointer will be back at the tbegin but our in memory stack won't be valid
anymore.

To avoid this, when taking a signal in an active transaction, we need to use
the stack pointer from the checkpointed state, rather than the speculated
state.  This ensures that the signal context (written tm suspended) will be
written below the stack required for the rollback.  The transaction is aborted
because of the treclaim, so any memory written between the tbegin and the
signal will be rolled back anyway.

For signals taken in non-TM or suspended mode, we use the
normal/non-checkpointed stack pointer.

Any transaction initiated inside a sighandler and suspended on return
from the sighandler to the kernel will get reclaimed and discarded.

Failure cause codes used by kernel
==================================

These are defined in <asm/reg.h>, and distinguish different reasons why the
kernel aborted a transaction:

 ====================== ================================
 TM_CAUSE_RESCHED       Thread was rescheduled.
 TM_CAUSE_TLBI          Software TLB invalid.
 TM_CAUSE_FAC_UNAV      FP/VEC/VSX unavailable trap.
 TM_CAUSE_SYSCALL       Syscall from active transaction.
 TM_CAUSE_SIGNAL        Signal delivered.
 TM_CAUSE_MISC          Currently unused.
 TM_CAUSE_ALIGNMENT     Alignment fault.
 TM_CAUSE_EMULATE       Emulation that touched memory.
 ====================== ================================

These can be checked by the user program's abort handler as TEXASR[0:7].  If
bit 7 is set, it indicates that the error is considered persistent.  For example
a TM_CAUSE_ALIGNMENT will be persistent while a TM_CAUSE_RESCHED will not.

GDB
===

GDB and ptrace are not currently TM-aware.  If one stops during a transaction,
it looks like the transaction has just started (the checkpointed state is
presented).  The transaction cannot then be continued and will take the failure
handler route.  Furthermore, the transactional 2nd register state will be
inaccessible.  GDB can currently be used on programs using TM, but not sensibly
in parts within transactions.

POWER9
======

TM on POWER9 has issues with storing the complete register state. This
is described in this commit::

    commit 4bb3c7a0208fc13ca70598efd109901a7cd45ae7
    Author: Paul Mackerras <paulus@ozlabs.org>
    Date:   Wed Mar 21 21:32:01 2018 +1100
    KVM: PPC: Book3S HV: Work around transactional memory bugs in POWER9

To account for this different POWER9 chips have TM enabled in
different ways.

On POWER9N DD2.01 and below, TM is disabled. ie
HWCAP2[PPC_FEATURE2_HTM] is not set.

On POWER9N DD2.1 TM is configured by firmware to always abort a
transaction when tm suspend occurs. So tsuspend will cause a
transaction to be aborted and rolled back. Kernel exceptions will also
cause the transaction to be aborted and rolled back and the exception
will not occur. If userspace constructs a sigcontext that enables TM
suspend, the sigcontext will be rejected by the kernel. This mode is
advertised to users with HWCAP2[PPC_FEATURE2_HTM_NO_SUSPEND] set.
HWCAP2[PPC_FEATURE2_HTM] is not set in this mode.

On POWER9N DD2.2 and above, KVM and POWERVM emulate TM for guests (as
described in commit 4bb3c7a0208f), hence TM is enabled for guests
ie. HWCAP2[PPC_FEATURE2_HTM] is set for guest userspace. Guests that
makes heavy use of TM suspend (tsuspend or kernel suspend) will result
in traps into the hypervisor and hence will suffer a performance
degradation. Host userspace has TM disabled
ie. HWCAP2[PPC_FEATURE2_HTM] is not set. (although we make enable it
at some point in the future if we bring the emulation into host
userspace context switching).

POWER9C DD1.2 and above are only available with POWERVM and hence
Linux only runs as a guest. On these systems TM is emulated like on
POWER9N DD2.2.

Guest migration from POWER8 to POWER9 will work with POWER9N DD2.2 and
POWER9C DD1.2. Since earlier POWER9 processors don't support TM
emulation, migration from POWER8 to POWER9 is not supported there.

Kernel implementation
=====================

h/rfid mtmsrd quirk
-------------------

As defined in the ISA, rfid has a quirk which is useful in early
exception handling. When in a userspace transaction and we enter the
kernel via some exception, MSR will end up as TM=0 and TS=01 (ie. TM
off but TM suspended). Regularly the kernel will want change bits in
the MSR and will perform an rfid to do this. In this case rfid can
have SRR0 TM = 0 and TS = 00 (ie. TM off and non transaction) and the
resulting MSR will retain TM = 0 and TS=01 from before (ie. stay in
suspend). This is a quirk in the architecture as this would normally
be a transition from TS=01 to TS=00 (ie. suspend -> non transactional)
which is an illegal transition.

This quirk is described the architecture in the definition of rfid
with these lines:

  if (MSR 29:31 ¬ = 0b010 | SRR1 29:31 ¬ = 0b000) then
     MSR 29:31 <- SRR1 29:31

hrfid and mtmsrd have the same quirk.

The Linux kernel uses this quirk in its early exception handling.