code:
================================
Coherent Accelerator (CXL) Flash
================================

Introduction
============

    The IBM Power architecture provides support for CAPI (Coherent
    Accelerator Power Interface), which is available to certain PCIe slots
    on Power 8 systems. CAPI can be thought of as a special tunneling
    protocol through PCIe that allow PCIe adapters to look like special
    purpose co-processors which can read or write an application's
    memory and generate page faults. As a result, the host interface to
    an adapter running in CAPI mode does not require the data buffers to
    be mapped to the device's memory (IOMMU bypass) nor does it require
    memory to be pinned.

    On Linux, Coherent Accelerator (CXL) kernel services present CAPI
    devices as a PCI device by implementing a virtual PCI host bridge.
    This abstraction simplifies the infrastructure and programming
    model, allowing for drivers to look similar to other native PCI
    device drivers.

    CXL provides a mechanism by which user space applications can
    directly talk to a device (network or storage) bypassing the typical
    kernel/device driver stack. The CXL Flash Adapter Driver enables a
    user space application direct access to Flash storage.

    The CXL Flash Adapter Driver is a kernel module that sits in the
    SCSI stack as a low level device driver (below the SCSI disk and
    protocol drivers) for the IBM CXL Flash Adapter. This driver is
    responsible for the initialization of the adapter, setting up the
    special path for user space access, and performing error recovery. It
    communicates directly the Flash Accelerator Functional Unit (AFU)
    as described in Documentation/powerpc/cxl.rst.

    The cxlflash driver supports two, mutually exclusive, modes of
    operation at the device (LUN) level:

        - Any flash device (LUN) can be configured to be accessed as a
          regular disk device (i.e.: /dev/sdc). This is the default mode.

        - Any flash device (LUN) can be configured to be accessed from
          user space with a special block library. This mode further
          specifies the means of accessing the device and provides for
          either raw access to the entire LUN (referred to as direct
          or physical LUN access) or access to a kernel/AFU-mediated
          partition of the LUN (referred to as virtual LUN access). The
          segmentation of a disk device into virtual LUNs is assisted
          by special translation services provided by the Flash AFU.

Overview
========

    The Coherent Accelerator Interface Architecture (CAIA) introduces a
    concept of a master context. A master typically has special privileges
    granted to it by the kernel or hypervisor allowing it to perform AFU
    wide management and control. The master may or may not be involved
    directly in each user I/O, but at the minimum is involved in the
    initial setup before the user application is allowed to send requests
    directly to the AFU.

    The CXL Flash Adapter Driver establishes a master context with the
    AFU. It uses memory mapped I/O (MMIO) for this control and setup. The
    Adapter Problem Space Memory Map looks like this::

                     +-------------------------------+
                     |    512 * 64 KB User MMIO      |
                     |        (per context)          |
                     |       User Accessible         |
                     +-------------------------------+
                     |    512 * 128 B per context    |
                     |    Provisioning and Control   |
                     |   Trusted Process accessible  |
                     +-------------------------------+
                     |         64 KB Global          |
                     |   Trusted Process accessible  |
                     +-------------------------------+

    This driver configures itself into the SCSI software stack as an
    adapter driver. The driver is the only entity that is considered a
    Trusted Process to program the Provisioning and Control and Global
    areas in the MMIO Space shown above.  The master context driver
    discovers all LUNs attached to the CXL Flash adapter and instantiates
    scsi block devices (/dev/sdb, /dev/sdc etc.) for each unique LUN
    seen from each path.

    Once these scsi block devices are instantiated, an application
    written to a specification provided by the block library may get
    access to the Flash from user space (without requiring a system call).

    This master context driver also provides a series of ioctls for this
    block library to enable this user space access.  The driver supports
    two modes for accessing the block device.

    The first mode is called a virtual mode. In this mode a single scsi
    block device (/dev/sdb) may be carved up into any number of distinct
    virtual LUNs. The virtual LUNs may be resized as long as the sum of
    the sizes of all the virtual LUNs, along with the meta-data associated
    with it does not exceed the physical capacity.

    The second mode is called the physical mode. In this mode a single
    block device (/dev/sdb) may be opened directly by the block library
    and the entire space for the LUN is available to the application.

    Only the physical mode provides persistence of the data.  i.e. The
    data written to the block device will survive application exit and
    restart and also reboot. The virtual LUNs do not persist (i.e. do
    not survive after the application terminates or the system reboots).

Block library API
=================

    Applications intending to get access to the CXL Flash from user
    space should use the block library, as it abstracts the details of
    interfacing directly with the cxlflash driver that are necessary for
    performing administrative actions (i.e.: setup, tear down, resize).
    The block library can be thought of as a 'user' of services,
    implemented as IOCTLs, that are provided by the cxlflash driver
    specifically for devices (LUNs) operating in user space access
    mode. While it is not a requirement that applications understand
    the interface between the block library and the cxlflash driver,
    a high-level overview of each supported service (IOCTL) is provided
    below.

    The block library can be found on GitHub:
    http://github.com/open-power/capiflash

CXL Flash Driver LUN IOCTLs
===========================

    Users, such as the block library, that wish to interface with a flash
    device (LUN) via user space access need to use the services provided
    by the cxlflash driver. As these services are implemented as ioctls,
    a file descriptor handle must first be obtained in order to establish
    the communication channel between a user and the kernel.  This file
    descriptor is obtained by opening the device special file associated
    with the scsi disk device (/dev/sdb) that was created during LUN
    discovery. As per the location of the cxlflash driver within the
    SCSI protocol stack, this open is actually not seen by the cxlflash
    driver. Upon successful open, the user receives a file descriptor
    (herein referred to as fd1) that should be used for issuing the
    subsequent ioctls listed below.

    The structure definitions for these IOCTLs are available in:
    uapi/scsi/cxlflash_ioctl.h

DK_CXLFLASH_ATTACH
------------------

    This ioctl obtains, initializes, and starts a context using the CXL
    kernel services. These services specify a context id (u16) by which
    to uniquely identify the context and its allocated resources. The
    services additionally provide a second file descriptor (herein
    referred to as fd2) that is used by the block library to initiate
    memory mapped I/O (via mmap()) to the CXL flash device and poll for
    completion events. This file descriptor is intentionally installed by
    this driver and not the CXL kernel services to allow for intermediary
    notification and access in the event of a non-user-initiated close(),
    such as a killed process. This design point is described in further
    detail in the description for the DK_CXLFLASH_DETACH ioctl.

    There are a few important aspects regarding the "tokens" (context id
    and fd2) that are provided back to the user:

        - These tokens are only valid for the process under which they
          were created. The child of a forked process cannot continue
          to use the context id or file descriptor created by its parent
          (see DK_CXLFLASH_VLUN_CLONE for further details).

        - These tokens are only valid for the lifetime of the context and
          the process under which they were created. Once either is
          destroyed, the tokens are to be considered stale and subsequent
          usage will result in errors.

	- A valid adapter file descriptor (fd2 >= 0) is only returned on
	  the initial attach for a context. Subsequent attaches to an
	  existing context (DK_CXLFLASH_ATTACH_REUSE_CONTEXT flag present)
	  do not provide the adapter file descriptor as it was previously
	  made known to the application.

        - When a context is no longer needed, the user shall detach from
          the context via the DK_CXLFLASH_DETACH ioctl. When this ioctl
	  returns with a valid adapter file descriptor and the return flag
	  DK_CXLFLASH_APP_CLOSE_ADAP_FD is present, the application _must_
	  close the adapter file descriptor following a successful detach.

	- When this ioctl returns with a valid fd2 and the return flag
	  DK_CXLFLASH_APP_CLOSE_ADAP_FD is present, the application _must_
	  close fd2 in the following circumstances:

		+ Following a successful detach of the last user of the context
		+ Following a successful recovery on the context's original fd2
		+ In the child process of a fork(), following a clone ioctl,
		  on the fd2 associated with the source context

        - At any time, a close on fd2 will invalidate the tokens. Applications
	  should exercise caution to only close fd2 when appropriate (outlined
	  in the previous bullet) to avoid premature loss of I/O.

DK_CXLFLASH_USER_DIRECT
-----------------------
    This ioctl is responsible for transitioning the LUN to direct
    (physical) mode access and configuring the AFU for direct access from
    user space on a per-context basis. Additionally, the block size and
    last logical block address (LBA) are returned to the user.

    As mentioned previously, when operating in user space access mode,
    LUNs may be accessed in whole or in part. Only one mode is allowed
    at a time and if one mode is active (outstanding references exist),
    requests to use the LUN in a different mode are denied.

    The AFU is configured for direct access from user space by adding an
    entry to the AFU's resource handle table. The index of the entry is
    treated as a resource handle that is returned to the user. The user
    is then able to use the handle to reference the LUN during I/O.

DK_CXLFLASH_USER_VIRTUAL
------------------------
    This ioctl is responsible for transitioning the LUN to virtual mode
    of access and configuring the AFU for virtual access from user space
    on a per-context basis. Additionally, the block size and last logical
    block address (LBA) are returned to the user.

    As mentioned previously, when operating in user space access mode,
    LUNs may be accessed in whole or in part. Only one mode is allowed
    at a time and if one mode is active (outstanding references exist),
    requests to use the LUN in a different mode are denied.

    The AFU is configured for virtual access from user space by adding
    an entry to the AFU's resource handle table. The index of the entry
    is treated as a resource handle that is returned to the user. The
    user is then able to use the handle to reference the LUN during I/O.

    By default, the virtual LUN is created with a size of 0. The user
    would need to use the DK_CXLFLASH_VLUN_RESIZE ioctl to adjust the grow
    the virtual LUN to a desired size. To avoid having to perform this
    resize for the initial creation of the virtual LUN, the user has the
    option of specifying a size as part of the DK_CXLFLASH_USER_VIRTUAL
    ioctl, such that when success is returned to the user, the
    resource handle that is provided is already referencing provisioned
    storage. This is reflected by the last LBA being a non-zero value.

    When a LUN is accessible from more than one port, this ioctl will
    return with the DK_CXLFLASH_ALL_PORTS_ACTIVE return flag set. This
    provides the user with a hint that I/O can be retried in the event
    of an I/O error as the LUN can be reached over multiple paths.

DK_CXLFLASH_VLUN_RESIZE
-----------------------
    This ioctl is responsible for resizing a previously created virtual
    LUN and will fail if invoked upon a LUN that is not in virtual
    mode. Upon success, an updated last LBA is returned to the user
    indicating the new size of the virtual LUN associated with the
    resource handle.

    The partitioning of virtual LUNs is jointly mediated by the cxlflash
    driver and the AFU. An allocation table is kept for each LUN that is
    operating in the virtual mode and used to program a LUN translation
    table that the AFU references when provided with a resource handle.

    This ioctl can return -EAGAIN if an AFU sync operation takes too long.
    In addition to returning a failure to user, cxlflash will also schedule
    an asynchronous AFU reset. Should the user choose to retry the operation,
    it is expected to succeed. If this ioctl fails with -EAGAIN, the user
    can either retry the operation or treat it as a failure.

DK_CXLFLASH_RELEASE
-------------------
    This ioctl is responsible for releasing a previously obtained
    reference to either a physical or virtual LUN. This can be
    thought of as the inverse of the DK_CXLFLASH_USER_DIRECT or
    DK_CXLFLASH_USER_VIRTUAL ioctls. Upon success, the resource handle
    is no longer valid and the entry in the resource handle table is
    made available to be used again.

    As part of the release process for virtual LUNs, the virtual LUN
    is first resized to 0 to clear out and free the translation tables
    associated with the virtual LUN reference.

DK_CXLFLASH_DETACH
------------------
    This ioctl is responsible for unregistering a context with the
    cxlflash driver and release outstanding resources that were
    not explicitly released via the DK_CXLFLASH_RELEASE ioctl. Upon
    success, all "tokens" which had been provided to the user from the
    DK_CXLFLASH_ATTACH onward are no longer valid.

    When the DK_CXLFLASH_APP_CLOSE_ADAP_FD flag was returned on a successful
    attach, the application _must_ close the fd2 associated with the context
    following the detach of the final user of the context.

DK_CXLFLASH_VLUN_CLONE
----------------------
    This ioctl is responsible for cloning a previously created
    context to a more recently created context. It exists solely to
    support maintaining user space access to storage after a process
    forks. Upon success, the child process (which invoked the ioctl)
    will have access to the same LUNs via the same resource handle(s)
    as the parent, but under a different context.

    Context sharing across processes is not supported with CXL and
    therefore each fork must be met with establishing a new context
    for the child process. This ioctl simplifies the state management
    and playback required by a user in such a scenario. When a process
    forks, child process can clone the parents context by first creating
    a context (via DK_CXLFLASH_ATTACH) and then using this ioctl to
    perform the clone from the parent to the child.

    The clone itself is fairly simple. The resource handle and lun
    translation tables are copied from the parent context to the child's
    and then synced with the AFU.

    When the DK_CXLFLASH_APP_CLOSE_ADAP_FD flag was returned on a successful
    attach, the application _must_ close the fd2 associated with the source
    context (still resident/accessible in the parent process) following the
    clone. This is to avoid a stale entry in the file descriptor table of the
    child process.

    This ioctl can return -EAGAIN if an AFU sync operation takes too long.
    In addition to returning a failure to user, cxlflash will also schedule
    an asynchronous AFU reset. Should the user choose to retry the operation,
    it is expected to succeed. If this ioctl fails with -EAGAIN, the user
    can either retry the operation or treat it as a failure.

DK_CXLFLASH_VERIFY
------------------
    This ioctl is used to detect various changes such as the capacity of
    the disk changing, the number of LUNs visible changing, etc. In cases
    where the changes affect the application (such as a LUN resize), the
    cxlflash driver will report the changed state to the application.

    The user calls in when they want to validate that a LUN hasn't been
    changed in response to a check condition. As the user is operating out
    of band from the kernel, they will see these types of events without
    the kernel's knowledge. When encountered, the user's architected
    behavior is to call in to this ioctl, indicating what they want to
    verify and passing along any appropriate information. For now, only
    verifying a LUN change (ie: size different) with sense data is
    supported.

DK_CXLFLASH_RECOVER_AFU
-----------------------
    This ioctl is used to drive recovery (if such an action is warranted)
    of a specified user context. Any state associated with the user context
    is re-established upon successful recovery.

    User contexts are put into an error condition when the device needs to
    be reset or is terminating. Users are notified of this error condition
    by seeing all 0xF's on an MMIO read. Upon encountering this, the
    architected behavior for a user is to call into this ioctl to recover
    their context. A user may also call into this ioctl at any time to
    check if the device is operating normally. If a failure is returned
    from this ioctl, the user is expected to gracefully clean up their
    context via release/detach ioctls. Until they do, the context they
    hold is not relinquished. The user may also optionally exit the process
    at which time the context/resources they held will be freed as part of
    the release fop.

    When the DK_CXLFLASH_APP_CLOSE_ADAP_FD flag was returned on a successful
    attach, the application _must_ unmap and close the fd2 associated with the
    original context following this ioctl returning success and indicating that
    the context was recovered (DK_CXLFLASH_RECOVER_AFU_CONTEXT_RESET).

DK_CXLFLASH_MANAGE_LUN
----------------------
    This ioctl is used to switch a LUN from a mode where it is available
    for file-system access (legacy), to a mode where it is set aside for
    exclusive user space access (superpipe). In case a LUN is visible
    across multiple ports and adapters, this ioctl is used to uniquely
    identify each LUN by its World Wide Node Name (WWNN).

CXL Flash Driver Host IOCTLs
============================

    Each host adapter instance that is supported by the cxlflash driver
    has a special character device associated with it to enable a set of
    host management function. These character devices are hosted in a
    class dedicated for cxlflash and can be accessed via `/dev/cxlflash/*`.

    Applications can be written to perform various functions using the
    host ioctl APIs below.

    The structure definitions for these IOCTLs are available in:
    uapi/scsi/cxlflash_ioctl.h

HT_CXLFLASH_LUN_PROVISION
-------------------------
    This ioctl is used to create and delete persistent LUNs on cxlflash
    devices that lack an external LUN management interface. It is only
    valid when used with AFUs that support the LUN provision capability.

    When sufficient space is available, LUNs can be created by specifying
    the target port to host the LUN and a desired size in 4K blocks. Upon
    success, the LUN ID and WWID of the created LUN will be returned and
    the SCSI bus can be scanned to detect the change in LUN topology. Note
    that partial allocations are not supported. Should a creation fail due
    to a space issue, the target port can be queried for its current LUN
    geometry.

    To remove a LUN, the device must first be disassociated from the Linux
    SCSI subsystem. The LUN deletion can then be initiated by specifying a
    target port and LUN ID. Upon success, the LUN geometry associated with
    the port will be updated to reflect new number of provisioned LUNs and
    available capacity.

    To query the LUN geometry of a port, the target port is specified and
    upon success, the following information is presented:

        - Maximum number of provisioned LUNs allowed for the port
        - Current number of provisioned LUNs for the port
        - Maximum total capacity of provisioned LUNs for the port (4K blocks)
        - Current total capacity of provisioned LUNs for the port (4K blocks)

    With this information, the number of available LUNs and capacity can be
    can be calculated.

HT_CXLFLASH_AFU_DEBUG
---------------------
    This ioctl is used to debug AFUs by supporting a command pass-through
    interface. It is only valid when used with AFUs that support the AFU
    debug capability.

    With exception of buffer management, AFU debug commands are opaque to
    cxlflash and treated as pass-through. For debug commands that do require
    data transfer, the user supplies an adequately sized data buffer and must
    specify the data transfer direction with respect to the host. There is a
    maximum transfer size of 256K imposed. Note that partial read completions
    are not supported - when errors are experienced with a host read data
    transfer, the data buffer is not copied back to the user.


code:
=====================
DAWR issues on POWER9
=====================

On older POWER9 processors, the Data Address Watchpoint Register (DAWR) can
cause a checkstop if it points to cache inhibited (CI) memory. Currently Linux
has no way to distinguish CI memory when configuring the DAWR, so on affected
systems, the DAWR is disabled.

Affected processor revisions
============================

This issue is only present on processors prior to v2.3. The revision can be
found in /proc/cpuinfo::

    processor       : 0
    cpu             : POWER9, altivec supported
    clock           : 3800.000000MHz
    revision        : 2.3 (pvr 004e 1203)

On a system with the issue, the DAWR is disabled as detailed below.

Technical Details:
==================

DAWR has 6 different ways of being set.
1) ptrace
2) h_set_mode(DAWR)
3) h_set_dabr()
4) kvmppc_set_one_reg()
5) xmon

For ptrace, we now advertise zero breakpoints on POWER9 via the
PPC_PTRACE_GETHWDBGINFO call. This results in GDB falling back to
software emulation of the watchpoint (which is slow).

h_set_mode(DAWR) and h_set_dabr() will now return an error to the
guest on a POWER9 host. Current Linux guests ignore this error, so
they will silently not get the DAWR.

kvmppc_set_one_reg() will store the value in the vcpu but won't
actually set it on POWER9 hardware. This is done so we don't break
migration from POWER8 to POWER9, at the cost of silently losing the
DAWR on the migration.

For xmon, the 'bd' command will return an error on P9.

Consequences for users
======================

For GDB watchpoints (ie 'watch' command) on POWER9 bare metal , GDB
will accept the command. Unfortunately since there is no hardware
support for the watchpoint, GDB will software emulate the watchpoint
making it run very slowly.

The same will also be true for any guests started on a POWER9
host. The watchpoint will fail and GDB will fall back to software
emulation.

If a guest is started on a POWER8 host, GDB will accept the watchpoint
and configure the hardware to use the DAWR. This will run at full
speed since it can use the hardware emulation. Unfortunately if this
guest is migrated to a POWER9 host, the watchpoint will be lost on the
POWER9. Loads and stores to the watchpoint locations will not be
trapped in GDB. The watchpoint is remembered, so if the guest is
migrated back to the POWER8 host, it will start working again.

Force enabling the DAWR
=======================
Kernels (since ~v5.2) have an option to force enable the DAWR via::

  echo Y > /sys/kernel/debug/powerpc/dawr_enable_dangerous

This enables the DAWR even on POWER9.

This is a dangerous setting, USE AT YOUR OWN RISK.

Some users may not care about a bad user crashing their box
(ie. single user/desktop systems) and really want the DAWR.  This
allows them to force enable DAWR.

This flag can also be used to disable DAWR access. Once this is
cleared, all DAWR access should be cleared immediately and your
machine once again safe from crashing.

Userspace may get confused by toggling this. If DAWR is force
enabled/disabled between getting the number of breakpoints (via
PTRACE_GETHWDBGINFO) and setting the breakpoint, userspace will get an
inconsistent view of what's available. Similarly for guests.

For the DAWR to be enabled in a KVM guest, the DAWR needs to be force
enabled in the host AND the guest. For this reason, this won't work on
POWERVM as it doesn't allow the HCALL to work. Writes of 'Y' to the
dawr_enable_dangerous file will fail if the hypervisor doesn't support
writing the DAWR.

To double check the DAWR is working, run this kernel selftest:

  tools/testing/selftests/powerpc/ptrace/ptrace-hwbreak.c

Any errors/failures/skips mean something is wrong.

repo_name: angery/linux


code:
===================================
DSCR (Data Stream Control Register)
===================================

DSCR register in powerpc allows user to have some control of prefetch of data
stream in the processor. Please refer to the ISA documents or related manual
for more detailed information regarding how to use this DSCR to attain this
control of the prefetches . This document here provides an overview of kernel
support for DSCR, related kernel objects, it's functionalities and exported
user interface.

(A) Data Structures:

	(1) thread_struct::

		dscr		/* Thread DSCR value */
		dscr_inherit	/* Thread has changed default DSCR */

	(2) PACA::

		dscr_default	/* per-CPU DSCR default value */

	(3) sysfs.c::

		dscr_default	/* System DSCR default value */

(B) Scheduler Changes:

	Scheduler will write the per-CPU DSCR default which is stored in the
	CPU's PACA value into the register if the thread has dscr_inherit value
	cleared which means that it has not changed the default DSCR till now.
	If the dscr_inherit value is set which means that it has changed the
	default DSCR value, scheduler will write the changed value which will
	now be contained in thread struct's dscr into the register instead of
	the per-CPU default PACA based DSCR value.

	NOTE: Please note here that the system wide global DSCR value never
	gets used directly in the scheduler process context switch at all.

(C) SYSFS Interface:

	- Global DSCR default:		/sys/devices/system/cpu/dscr_default
	- CPU specific DSCR default:	/sys/devices/system/cpu/cpuN/dscr

	Changing the global DSCR default in the sysfs will change all the CPU
	specific DSCR defaults immediately in their PACA structures. Again if
	the current process has the dscr_inherit clear, it also writes the new
	value into every CPU's DSCR register right away and updates the current
	thread's DSCR value as well.

	Changing the CPU specific DSCR default value in the sysfs does exactly
	the same thing as above but unlike the global one above, it just changes
	stuff for that particular CPU instead for all the CPUs on the system.

(D) User Space Instructions:

	The DSCR register can be accessed in the user space using any of these
	two SPR numbers available for that purpose.

	(1) Problem state SPR:		0x03	(Un-privileged, POWER8 only)
	(2) Privileged state SPR:	0x11	(Privileged)

	Accessing DSCR through privileged SPR number (0x11) from user space
	works, as it is emulated following an illegal instruction exception
	inside the kernel. Both mfspr and mtspr instructions are emulated.

	Accessing DSCR through user level SPR (0x03) from user space will first
	create a facility unavailable exception. Inside this exception handler
	all mfspr instruction based read attempts will get emulated and returned
	where as the first mtspr instruction based write attempts will enable
	the DSCR facility for the next time around (both for read and write) by
	setting DSCR facility in the FSCR register.

(E) Specifics about 'dscr_inherit':

	The thread struct element 'dscr_inherit' represents whether the thread
	in question has attempted and changed the DSCR itself using any of the
	following methods. This element signifies whether the thread wants to
	use the CPU default DSCR value or its own changed DSCR value in the
	kernel.

		(1) mtspr instruction	(SPR number 0x03)
		(2) mtspr instruction	(SPR number 0x11)
		(3) ptrace interface	(Explicitly set user DSCR value)

	Any child of the process created after this event in the process inherits
	this same behaviour as well.


code:
==========================
PCI Bus EEH Error Recovery
==========================

Linas Vepstas <linas@austin.ibm.com>

12 January 2005

Overview:
---------
The IBM POWER-based pSeries and iSeries computers include PCI bus
controller chips that have extended capabilities for detecting and
reporting a large variety of PCI bus error conditions.  These features
go under the name of "EEH", for "Enhanced Error Handling".  The EEH
hardware features allow PCI bus errors to be cleared and a PCI
card to be "rebooted", without also having to reboot the operating
system.

This is in contrast to traditional PCI error handling, where the
PCI chip is wired directly to the CPU, and an error would cause
a CPU machine-check/check-stop condition, halting the CPU entirely.
Another "traditional" technique is to ignore such errors, which
can lead to data corruption, both of user data or of kernel data,
hung/unresponsive adapters, or system crashes/lockups.  Thus,
the idea behind EEH is that the operating system can become more
reliable and robust by protecting it from PCI errors, and giving
the OS the ability to "reboot"/recover individual PCI devices.

Future systems from other vendors, based on the PCI-E specification,
may contain similar features.

Causes of EEH Errors
--------------------
EEH was originally designed to guard against hardware failure, such
as PCI cards dying from heat, humidity, dust, vibration and bad
electrical connections. The vast majority of EEH errors seen in
"real life" are due to either poorly seated PCI cards, or,
unfortunately quite commonly, due to device driver bugs, device firmware
bugs, and sometimes PCI card hardware bugs.

The most common software bug, is one that causes the device to
attempt to DMA to a location in system memory that has not been
reserved for DMA access for that card.  This is a powerful feature,
as it prevents what; otherwise, would have been silent memory
corruption caused by the bad DMA.  A number of device driver
bugs have been found and fixed in this way over the past few
years.  Other possible causes of EEH errors include data or
address line parity errors (for example, due to poor electrical
connectivity due to a poorly seated card), and PCI-X split-completion
errors (due to software, device firmware, or device PCI hardware bugs).
The vast majority of "true hardware failures" can be cured by
physically removing and re-seating the PCI card.

Detection and Recovery
----------------------
In the following discussion, a generic overview of how to detect
and recover from EEH errors will be presented. This is followed
by an overview of how the current implementation in the Linux
kernel does it.  The actual implementation is subject to change,
and some of the finer points are still being debated.  These
may in turn be swayed if or when other architectures implement
similar functionality.

When a PCI Host Bridge (PHB, the bus controller connecting the
PCI bus to the system CPU electronics complex) detects a PCI error
condition, it will "isolate" the affected PCI card.  Isolation
will block all writes (either to the card from the system, or
from the card to the system), and it will cause all reads to
return all-ff's (0xff, 0xffff, 0xffffffff for 8/16/32-bit reads).
This value was chosen because it is the same value you would
get if the device was physically unplugged from the slot.
This includes access to PCI memory, I/O space, and PCI config
space.  Interrupts; however, will continue to be delivered.

Detection and recovery are performed with the aid of ppc64
firmware.  The programming interfaces in the Linux kernel
into the firmware are referred to as RTAS (Run-Time Abstraction
Services).  The Linux kernel does not (should not) access
the EEH function in the PCI chipsets directly, primarily because
there are a number of different chipsets out there, each with
different interfaces and quirks. The firmware provides a
uniform abstraction layer that will work with all pSeries
and iSeries hardware (and be forwards-compatible).

If the OS or device driver suspects that a PCI slot has been
EEH-isolated, there is a firmware call it can make to determine if
this is the case. If so, then the device driver should put itself
into a consistent state (given that it won't be able to complete any
pending work) and start recovery of the card.  Recovery normally
would consist of resetting the PCI device (holding the PCI #RST
line high for two seconds), followed by setting up the device
config space (the base address registers (BAR's), latency timer,
cache line size, interrupt line, and so on).  This is followed by a
reinitialization of the device driver.  In a worst-case scenario,
the power to the card can be toggled, at least on hot-plug-capable
slots.  In principle, layers far above the device driver probably
do not need to know that the PCI card has been "rebooted" in this
way; ideally, there should be at most a pause in Ethernet/disk/USB
I/O while the card is being reset.

If the card cannot be recovered after three or four resets, the
kernel/device driver should assume the worst-case scenario, that the
card has died completely, and report this error to the sysadmin.
In addition, error messages are reported through RTAS and also through
syslogd (/var/log/messages) to alert the sysadmin of PCI resets.
The correct way to deal with failed adapters is to use the standard
PCI hotplug tools to remove and replace the dead card.

Current PPC64 Linux EEH Implementation
--------------------------------------
At this time, a generic EEH recovery mechanism has been implemented,
so that individual device drivers do not need to be modified to support
EEH recovery.  This generic mechanism piggy-backs on the PCI hotplug
infrastructure,  and percolates events up through the userspace/udev
infrastructure.  Following is a detailed description of how this is
accomplished.

EEH must be enabled in the PHB's very early during the boot process,
and if a PCI slot is hot-plugged. The former is performed by
eeh_init() in arch/powerpc/platforms/pseries/eeh.c, and the later by
drivers/pci/hotplug/pSeries_pci.c calling in to the eeh.c code.
EEH must be enabled before a PCI scan of the device can proceed.
Current Power5 hardware will not work unless EEH is enabled;
although older Power4 can run with it disabled.  Effectively,
EEH can no longer be turned off.  PCI devices *must* be
registered with the EEH code; the EEH code needs to know about
the I/O address ranges of the PCI device in order to detect an
error.  Given an arbitrary address, the routine
pci_get_device_by_addr() will find the pci device associated
with that address (if any).

The default arch/powerpc/include/asm/io.h macros readb(), inb(), insb(),
etc. include a check to see if the i/o read returned all-0xff's.
If so, these make a call to eeh_dn_check_failure(), which in turn
asks the firmware if the all-ff's value is the sign of a true EEH
error.  If it is not, processing continues as normal.  The grand
total number of these false alarms or "false positives" can be
seen in /proc/ppc64/eeh (subject to change).  Normally, almost
all of these occur during boot, when the PCI bus is scanned, where
a large number of 0xff reads are part of the bus scan procedure.

If a frozen slot is detected, code in
arch/powerpc/platforms/pseries/eeh.c will print a stack trace to
syslog (/var/log/messages).  This stack trace has proven to be very
useful to device-driver authors for finding out at what point the EEH
error was detected, as the error itself usually occurs slightly
beforehand.

Next, it uses the Linux kernel notifier chain/work queue mechanism to
allow any interested parties to find out about the failure.  Device
drivers, or other parts of the kernel, can use
`eeh_register_notifier(struct notifier_block *)` to find out about EEH
events.  The event will include a pointer to the pci device, the
device node and some state info.  Receivers of the event can "do as
they wish"; the default handler will be described further in this
section.

To assist in the recovery of the device, eeh.c exports the
following functions:

rtas_set_slot_reset()
   assert the  PCI #RST line for 1/8th of a second
rtas_configure_bridge()
   ask firmware to configure any PCI bridges
   located topologically under the pci slot.
eeh_save_bars() and eeh_restore_bars():
   save and restore the PCI
   config-space info for a device and any devices under it.

A handler for the EEH notifier_block events is implemented in
drivers/pci/hotplug/pSeries_pci.c, called handle_eeh_events().
It saves the device BAR's and then calls rpaphp_unconfig_pci_adapter().
This last call causes the device driver for the card to be stopped,
which causes uevents to go out to user space. This triggers
user-space scripts that might issue commands such as "ifdown eth0"
for ethernet cards, and so on.  This handler then sleeps for 5 seconds,
hoping to give the user-space scripts enough time to complete.
It then resets the PCI card, reconfigures the device BAR's, and
any bridges underneath. It then calls rpaphp_enable_pci_slot(),
which restarts the device driver and triggers more user-space
events (for example, calling "ifup eth0" for ethernet cards).

Device Shutdown and User-Space Events
-------------------------------------
This section documents what happens when a pci slot is unconfigured,
focusing on how the device driver gets shut down, and on how the
events get delivered to user-space scripts.

Following is an example sequence of events that cause a device driver
close function to be called during the first phase of an EEH reset.
The following sequence is an example of the pcnet32 device driver::

    rpa_php_unconfig_pci_adapter (struct slot *)  // in rpaphp_pci.c
    {
      calls
      pci_remove_bus_device (struct pci_dev *) // in /drivers/pci/remove.c
      {
        calls
        pci_destroy_dev (struct pci_dev *)
        {
          calls
          device_unregister (&dev->dev) // in /drivers/base/core.c
          {
            calls
            device_del (struct device *)
            {
              calls
              bus_remove_device() // in /drivers/base/bus.c
              {
                calls
                device_release_driver()
                {
                  calls
                  struct device_driver->remove() which is just
                  pci_device_remove()  // in /drivers/pci/pci_driver.c
                  {
                    calls
                    struct pci_driver->remove() which is just
                    pcnet32_remove_one() // in /drivers/net/pcnet32.c
                    {
                      calls
                      unregister_netdev() // in /net/core/dev.c
                      {
                        calls
                        dev_close()  // in /net/core/dev.c
                        {
                           calls dev->stop();
                           which is just pcnet32_close() // in pcnet32.c
                           {
                             which does what you wanted
                             to stop the device
                           }
                        }
                     }
                   which
                   frees pcnet32 device driver memory
                }
     }}}}}}

in drivers/pci/pci_driver.c,
struct device_driver->remove() is just pci_device_remove()
which calls struct pci_driver->remove() which is pcnet32_remove_one()
which calls unregister_netdev()  (in net/core/dev.c)
which calls dev_close()  (in net/core/dev.c)
which calls dev->stop() which is pcnet32_close()
which then does the appropriate shutdown.

---

Following is the analogous stack trace for events sent to user-space
when the pci device is unconfigured::

  rpa_php_unconfig_pci_adapter() {             // in rpaphp_pci.c
    calls
    pci_remove_bus_device (struct pci_dev *) { // in /drivers/pci/remove.c
      calls
      pci_destroy_dev (struct pci_dev *) {
        calls
        device_unregister (&dev->dev) {        // in /drivers/base/core.c
          calls
          device_del(struct device * dev) {    // in /drivers/base/core.c
            calls
            kobject_del() {                    //in /libs/kobject.c
              calls
              kobject_uevent() {               // in /libs/kobject.c
                calls
                kset_uevent() {                // in /lib/kobject.c
                  calls
                  kset->uevent_ops->uevent()   // which is really just
                  a call to
                  dev_uevent() {               // in /drivers/base/core.c
                    calls
                    dev->bus->uevent() which is really just a call to
                    pci_uevent () {            // in drivers/pci/hotplug.c
                      which prints device name, etc....
                   }
                 }
                 then kobject_uevent() sends a netlink uevent to userspace
                 --> userspace uevent
                 (during early boot, nobody listens to netlink events and
                 kobject_uevent() executes uevent_helper[], which runs the
                 event process /sbin/hotplug)
             }
           }
           kobject_del() then calls sysfs_remove_dir(), which would
           trigger any user-space daemon that was watching /sysfs,
           and notice the delete event.

Pro's and Con's of the Current Design
-------------------------------------
There are several issues with the current EEH software recovery design,
which may be addressed in future revisions.  But first, note that the
big plus of the current design is that no changes need to be made to
individual device drivers, so that the current design throws a wide net.
The biggest negative of the design is that it potentially disturbs
network daemons and file systems that didn't need to be disturbed.

-  A minor complaint is that resetting the network card causes
   user-space back-to-back ifdown/ifup burps that potentially disturb
   network daemons, that didn't need to even know that the pci
   card was being rebooted.

-  A more serious concern is that the same reset, for SCSI devices,
   causes havoc to mounted file systems.  Scripts cannot post-facto
   unmount a file system without flushing pending buffers, but this
   is impossible, because I/O has already been stopped.  Thus,
   ideally, the reset should happen at or below the block layer,
   so that the file systems are not disturbed.

   Reiserfs does not tolerate errors returned from the block device.
   Ext3fs seems to be tolerant, retrying reads/writes until it does
   succeed. Both have been only lightly tested in this scenario.

   The SCSI-generic subsystem already has built-in code for performing
   SCSI device resets, SCSI bus resets, and SCSI host-bus-adapter
   (HBA) resets.  These are cascaded into a chain of attempted
   resets if a SCSI command fails. These are completely hidden
   from the block layer.  It would be very natural to add an EEH
   reset into this chain of events.

-  If a SCSI error occurs for the root device, all is lost unless
   the sysadmin had the foresight to run /bin, /sbin, /etc, /var
   and so on, out of ramdisk/tmpfs.

Conclusions
-----------
There's forward progress ...


code:
.. _elf_hwcaps_powerpc:

==================
POWERPC ELF HWCAPs
==================

This document describes the usage and semantics of the powerpc ELF HWCAPs.

1. Introduction
---------------

Some hardware or software features are only available on some CPU
implementations, and/or with certain kernel configurations, but have no other
discovery mechanism available to userspace code. The kernel exposes the
presence of these features to userspace through a set of flags called HWCAPs,
exposed in the auxiliary vector.

Userspace software can test for features by acquiring the AT_HWCAP or
AT_HWCAP2 entry of the auxiliary vector, and testing whether the relevant
flags are set, e.g.::

	bool floating_point_is_present(void)
	{
		unsigned long HWCAPs = getauxval(AT_HWCAP);
		if (HWCAPs & PPC_FEATURE_HAS_FPU)
			return true;

		return false;
	}

Where software relies on a feature described by a HWCAP, it should check the
relevant HWCAP flag to verify that the feature is present before attempting to
make use of the feature.

HWCAP is the preferred method to test for the presence of a feature rather
than probing through other means, which may not be reliable or may cause
unpredictable behaviour.

Software that targets a particular platform does not necessarily have to
test for required or implied features. For example if the program requires
FPU, VMX, VSX, it is not necessary to test those HWCAPs, and it may be
impossible to do so if the compiler generates code requiring those features.

2. Facilities
-------------

The Power ISA uses the term "facility" to describe a class of instructions,
registers, interrupts, etc. The presence or absence of a facility indicates
whether this class is available to be used, but the specifics depend on the
ISA version. For example, if the VSX facility is available, the VSX
instructions that can be used differ between the v3.0B and v3.1B ISA
versions.

3. Categories
-------------

The Power ISA before v3.0 uses the term "category" to describe certain
classes of instructions and operating modes which may be optional or
mutually exclusive, the exact meaning of the HWCAP flag may depend on
context, e.g., the presence of the BOOKE feature implies that the server
category is not implemented.

4. HWCAP allocation
-------------------

HWCAPs are allocated as described in Power Architecture 64-Bit ELF V2 ABI
Specification (which will be reflected in the kernel's uapi headers).

5. The HWCAPs exposed in AT_HWCAP
---------------------------------

PPC_FEATURE_32
    32-bit CPU

PPC_FEATURE_64
    64-bit CPU (userspace may be running in 32-bit mode).

PPC_FEATURE_601_INSTR
    The processor is PowerPC 601.
    Unused in the kernel since f0ed73f3fa2c ("powerpc: Remove PowerPC 601")

PPC_FEATURE_HAS_ALTIVEC
    Vector (aka Altivec, VMX) facility is available.

PPC_FEATURE_HAS_FPU
    Floating point facility is available.

PPC_FEATURE_HAS_MMU
    Memory management unit is present and enabled.

PPC_FEATURE_HAS_4xxMAC
    The processor is 40x or 44x family.

PPC_FEATURE_UNIFIED_CACHE
    The processor has a unified L1 cache for instructions and data, as
    found in NXP e200.
    Unused in the kernel since 39c8bf2b3cc1 ("powerpc: Retire e200 core (mpc555x processor)")

PPC_FEATURE_HAS_SPE
    Signal Processing Engine facility is available.

PPC_FEATURE_HAS_EFP_SINGLE
    Embedded Floating Point single precision operations are available.

PPC_FEATURE_HAS_EFP_DOUBLE
    Embedded Floating Point double precision operations are available.

PPC_FEATURE_NO_TB
    The timebase facility (mftb instruction) is not available.
    This is a 601 specific HWCAP, so if it is known that the processor
    running is not a 601, via other HWCAPs or other means, it is not
    required to test this bit before using the timebase.
    Unused in the kernel since f0ed73f3fa2c ("powerpc: Remove PowerPC 601")

PPC_FEATURE_POWER4
    The processor is POWER4 or PPC970/FX/MP.
    POWER4 support dropped from the kernel since 471d7ff8b51b ("powerpc/64s: Remove POWER4 support")

PPC_FEATURE_POWER5
    The processor is POWER5.

PPC_FEATURE_POWER5_PLUS
    The processor is POWER5+.

PPC_FEATURE_CELL
    The processor is Cell.

PPC_FEATURE_BOOKE
    The processor implements the embedded category ("BookE") architecture.

PPC_FEATURE_SMT
    The processor implements SMT.

PPC_FEATURE_ICACHE_SNOOP
    The processor icache is coherent with the dcache, and instruction storage
    can be made consistent with data storage for the purpose of executing
    instructions with the sequence (as described in, e.g., POWER9 Processor
    User's Manual, 4.6.2.2 Instruction Cache Block Invalidate (icbi))::

        sync
        icbi (to any address)
        isync

PPC_FEATURE_ARCH_2_05
    The processor supports the v2.05 userlevel architecture. Processors
    supporting later architectures DO NOT set this feature.

PPC_FEATURE_PA6T
    The processor is PA6T.

PPC_FEATURE_HAS_DFP
    DFP facility is available.

PPC_FEATURE_POWER6_EXT
    The processor is POWER6.

PPC_FEATURE_ARCH_2_06
    The processor supports the v2.06 userlevel architecture. Processors
    supporting later architectures also set this feature.

PPC_FEATURE_HAS_VSX
    VSX facility is available.

PPC_FEATURE_PSERIES_PERFMON_COMPAT
    The processor supports architected PMU events in the range 0xE0-0xFF.

PPC_FEATURE_TRUE_LE
    The processor supports true little-endian mode.

PPC_FEATURE_PPC_LE
    The processor supports "PowerPC Little-Endian", that uses address
    munging to make storage access appear to be little-endian, but the
    data is stored in a different format that is unsuitable to be
    accessed by other agents not running in this mode.

6. The HWCAPs exposed in AT_HWCAP2
----------------------------------

PPC_FEATURE2_ARCH_2_07
    The processor supports the v2.07 userlevel architecture. Processors
    supporting later architectures also set this feature.

PPC_FEATURE2_HTM
    Transactional Memory feature is available.

PPC_FEATURE2_DSCR
    DSCR facility is available.

PPC_FEATURE2_EBB
    EBB facility is available.

PPC_FEATURE2_ISEL
    isel instruction is available. This is superseded by ARCH_2_07 and
    later.

PPC_FEATURE2_TAR
    TAR facility is available.

PPC_FEATURE2_VEC_CRYPTO
    v2.07 crypto instructions are available.

PPC_FEATURE2_HTM_NOSC
    System calls fail if called in a transactional state, see
    Documentation/powerpc/syscall64-abi.rst

PPC_FEATURE2_ARCH_3_00
    The processor supports the v3.0B / v3.0C userlevel architecture. Processors
    supporting later architectures also set this feature.

PPC_FEATURE2_HAS_IEEE128
    IEEE 128-bit binary floating point is supported with VSX
    quad-precision instructions and data types.

PPC_FEATURE2_DARN
    darn instruction is available.

PPC_FEATURE2_SCV
    The scv 0 instruction may be used for system calls, see
    Documentation/powerpc/syscall64-abi.rst.

PPC_FEATURE2_HTM_NO_SUSPEND
    A limited Transactional Memory facility that does not support suspend is
    available, see Documentation/powerpc/transactional_memory.rst.

PPC_FEATURE2_ARCH_3_1
    The processor supports the v3.1 userlevel architecture. Processors
    supporting later architectures also set this feature.

PPC_FEATURE2_MMA
    MMA facility is available.


code:
==========================
ELF Note PowerPC Namespace
==========================

The PowerPC namespace in an ELF Note of the kernel binary is used to store
capabilities and information which can be used by a bootloader or userland.

Types and Descriptors
---------------------

The types to be used with the "PowerPC" namespace are defined in [#f1]_.

	1) PPC_ELFNOTE_CAPABILITIES

Define the capabilities supported/required by the kernel. This type uses a
bitmap as "descriptor" field. Each bit is described below:

- Ultravisor-capable bit (PowerNV only).

.. code-block:: c

	#define PPCCAP_ULTRAVISOR_BIT (1 << 0)

Indicate that the powerpc kernel binary knows how to run in an
ultravisor-enabled system.

In an ultravisor-enabled system, some machine resources are now controlled
by the ultravisor. If the kernel is not ultravisor-capable, but it ends up
being run on a machine with ultravisor, the kernel will probably crash
trying to access ultravisor resources. For instance, it may crash in early
boot trying to set the partition table entry 0.

In an ultravisor-enabled system, a bootloader could warn the user or prevent
the kernel from being run if the PowerPC ultravisor capability doesn't exist
or the Ultravisor-capable bit is not set.

References
----------

.. [#f1] arch/powerpc/include/asm/elfnote.h


code:
.. SPDX-License-Identifier: GPL-2.0

.. kernel-feat:: $srctree/Documentation/features powerpc


code:
======================
Firmware-Assisted Dump
======================

July 2011

The goal of firmware-assisted dump is to enable the dump of
a crashed system, and to do so from a fully-reset system, and
to minimize the total elapsed time until the system is back
in production use.

- Firmware-Assisted Dump (FADump) infrastructure is intended to replace
  the existing phyp assisted dump.
- Fadump uses the same firmware interfaces and memory reservation model
  as phyp assisted dump.
- Unlike phyp dump, FADump exports the memory dump through /proc/vmcore
  in the ELF format in the same way as kdump. This helps us reuse the
  kdump infrastructure for dump capture and filtering.
- Unlike phyp dump, userspace tool does not need to refer any sysfs
  interface while reading /proc/vmcore.
- Unlike phyp dump, FADump allows user to release all the memory reserved
  for dump, with a single operation of echo 1 > /sys/kernel/fadump_release_mem.
- Once enabled through kernel boot parameter, FADump can be
  started/stopped through /sys/kernel/fadump_registered interface (see
  sysfs files section below) and can be easily integrated with kdump
  service start/stop init scripts.

Comparing with kdump or other strategies, firmware-assisted
dump offers several strong, practical advantages:

-  Unlike kdump, the system has been reset, and loaded
   with a fresh copy of the kernel.  In particular,
   PCI and I/O devices have been reinitialized and are
   in a clean, consistent state.
-  Once the dump is copied out, the memory that held the dump
   is immediately available to the running kernel. And therefore,
   unlike kdump, FADump doesn't need a 2nd reboot to get back
   the system to the production configuration.

The above can only be accomplished by coordination with,
and assistance from the Power firmware. The procedure is
as follows:

-  The first kernel registers the sections of memory with the
   Power firmware for dump preservation during OS initialization.
   These registered sections of memory are reserved by the first
   kernel during early boot.

-  When system crashes, the Power firmware will copy the registered
   low memory regions (boot memory) from source to destination area.
   It will also save hardware PTE's.

   NOTE:
         The term 'boot memory' means size of the low memory chunk
         that is required for a kernel to boot successfully when
         booted with restricted memory. By default, the boot memory
         size will be the larger of 5% of system RAM or 256MB.
         Alternatively, user can also specify boot memory size
         through boot parameter 'crashkernel=' which will override
         the default calculated size. Use this option if default
         boot memory size is not sufficient for second kernel to
         boot successfully. For syntax of crashkernel= parameter,
         refer to Documentation/admin-guide/kdump/kdump.rst. If any
         offset is provided in crashkernel= parameter, it will be
         ignored as FADump uses a predefined offset to reserve memory
         for boot memory dump preservation in case of a crash.

-  After the low memory (boot memory) area has been saved, the
   firmware will reset PCI and other hardware state.  It will
   *not* clear the RAM. It will then launch the bootloader, as
   normal.

-  The freshly booted kernel will notice that there is a new node
   (rtas/ibm,kernel-dump on pSeries or ibm,opal/dump/mpipl-boot
   on OPAL platform) in the device tree, indicating that
   there is crash data available from a previous boot. During
   the early boot OS will reserve rest of the memory above
   boot memory size effectively booting with restricted memory
   size. This will make sure that this kernel (also, referred
   to as second kernel or capture kernel) will not touch any
   of the dump memory area.

-  User-space tools will read /proc/vmcore to obtain the contents
   of memory, which holds the previous crashed kernel dump in ELF
   format. The userspace tools may copy this info to disk, or
   network, nas, san, iscsi, etc. as desired.

-  Once the userspace tool is done saving dump, it will echo
   '1' to /sys/kernel/fadump_release_mem to release the reserved
   memory back to general use, except the memory required for
   next firmware-assisted dump registration.

   e.g.::

     # echo 1 > /sys/kernel/fadump_release_mem

Please note that the firmware-assisted dump feature
is only available on POWER6 and above systems on pSeries
(PowerVM) platform and POWER9 and above systems with OP940
or later firmware versions on PowerNV (OPAL) platform.
Note that, OPAL firmware exports ibm,opal/dump node when
FADump is supported on PowerNV platform.

On OPAL based machines, system first boots into an intermittent
kernel (referred to as petitboot kernel) before booting into the
capture kernel. This kernel would have minimal kernel and/or
userspace support to process crash data. Such kernel needs to
preserve previously crash'ed kernel's memory for the subsequent
capture kernel boot to process this crash data. Kernel config
option CONFIG_PRESERVE_FA_DUMP has to be enabled on such kernel
to ensure that crash data is preserved to process later.

-- On OPAL based machines (PowerNV), if the kernel is build with
   CONFIG_OPAL_CORE=y, OPAL memory at the time of crash is also
   exported as /sys/firmware/opal/mpipl/core file. This procfs file is
   helpful in debugging OPAL crashes with GDB. The kernel memory
   used for exporting this procfs file can be released by echo'ing
   '1' to /sys/firmware/opal/mpipl/release_core node.

   e.g.
     # echo 1 > /sys/firmware/opal/mpipl/release_core

Implementation details:
-----------------------

During boot, a check is made to see if firmware supports
this feature on that particular machine. If it does, then
we check to see if an active dump is waiting for us. If yes
then everything but boot memory size of RAM is reserved during
early boot (See Fig. 2). This area is released once we finish
collecting the dump from user land scripts (e.g. kdump scripts)
that are run. If there is dump data, then the
/sys/kernel/fadump_release_mem file is created, and the reserved
memory is held.

If there is no waiting dump data, then only the memory required to
hold CPU state, HPTE region, boot memory dump, FADump header and
elfcore header, is usually reserved at an offset greater than boot
memory size (see Fig. 1). This area is *not* released: this region
will be kept permanently reserved, so that it can act as a receptacle
for a copy of the boot memory content in addition to CPU state and
HPTE region, in the case a crash does occur.

Since this reserved memory area is used only after the system crash,
there is no point in blocking this significant chunk of memory from
production kernel. Hence, the implementation uses the Linux kernel's
Contiguous Memory Allocator (CMA) for memory reservation if CMA is
configured for kernel. With CMA reservation this memory will be
available for applications to use it, while kernel is prevented from
using it. With this FADump will still be able to capture all of the
kernel memory and most of the user space memory except the user pages
that were present in CMA region::

  o Memory Reservation during first kernel

  Low memory                                                 Top of memory
  0    boot memory size   |<--- Reserved dump area --->|       |
  |           |           |    Permanent Reservation   |       |
  V           V           |                            |       V
  +-----------+-----/ /---+---+----+-------+-----+-----+----+--+
  |           |           |///|////|  DUMP | HDR | ELF |////|  |
  +-----------+-----/ /---+---+----+-------+-----+-----+----+--+
        |                   ^    ^     ^      ^           ^
        |                   |    |     |      |           |
        \                  CPU  HPTE   /      |           |
         ------------------------------       |           |
      Boot memory content gets transferred    |           |
      to reserved area by firmware at the     |           |
      time of crash.                          |           |
                                          FADump Header   |
                                           (meta area)    |
                                                          |
                                                          |
                      Metadata: This area holds a metadata structure whose
                      address is registered with f/w and retrieved in the
                      second kernel after crash, on platforms that support
                      tags (OPAL). Having such structure with info needed
                      to process the crashdump eases dump capture process.

                   Fig. 1

  o Memory Reservation during second kernel after crash

  Low memory                                              Top of memory
  0      boot memory size                                      |
  |           |<------------ Crash preserved area ------------>|
  V           V           |<--- Reserved dump area --->|       |
  +-----------+-----/ /---+---+----+-------+-----+-----+----+--+
  |           |           |///|////|  DUMP | HDR | ELF |////|  |
  +-----------+-----/ /---+---+----+-------+-----+-----+----+--+
        |                                           |
        V                                           V
   Used by second                             /proc/vmcore
   kernel to boot

        +---+
        |///| -> Regions (CPU, HPTE & Metadata) marked like this in the above
        +---+    figures are not always present. For example, OPAL platform
                 does not have CPU & HPTE regions while Metadata region is
                 not supported on pSeries currently.

                   Fig. 2

Currently the dump will be copied from /proc/vmcore to a new file upon
user intervention. The dump data available through /proc/vmcore will be
in ELF format. Hence the existing kdump infrastructure (kdump scripts)
to save the dump works fine with minor modifications. KDump scripts on
major Distro releases have already been modified to work seamlessly (no
user intervention in saving the dump) when FADump is used, instead of
KDump, as dump mechanism.

The tools to examine the dump will be same as the ones
used for kdump.

How to enable firmware-assisted dump (FADump):
----------------------------------------------

1. Set config option CONFIG_FA_DUMP=y and build kernel.
2. Boot into linux kernel with 'fadump=on' kernel cmdline option.
   By default, FADump reserved memory will be initialized as CMA area.
   Alternatively, user can boot linux kernel with 'fadump=nocma' to
   prevent FADump to use CMA.
3. Optionally, user can also set 'crashkernel=' kernel cmdline
   to specify size of the memory to reserve for boot memory dump
   preservation.

NOTE:
     1. 'fadump_reserve_mem=' parameter has been deprecated. Instead
        use 'crashkernel=' to specify size of the memory to reserve
        for boot memory dump preservation.
     2. If firmware-assisted dump fails to reserve memory then it
        will fallback to existing kdump mechanism if 'crashkernel='
        option is set at kernel cmdline.
     3. if user wants to capture all of user space memory and ok with
        reserved memory not available to production system, then
        'fadump=nocma' kernel parameter can be used to fallback to
        old behaviour.

Sysfs/debugfs files:
--------------------

Firmware-assisted dump feature uses sysfs file system to hold
the control files and debugfs file to display memory reserved region.

Here is the list of files under kernel sysfs:

 /sys/kernel/fadump_enabled
    This is used to display the FADump status.

    - 0 = FADump is disabled
    - 1 = FADump is enabled

    This interface can be used by kdump init scripts to identify if
    FADump is enabled in the kernel and act accordingly.

 /sys/kernel/fadump_registered
    This is used to display the FADump registration status as well
    as to control (start/stop) the FADump registration.

    - 0 = FADump is not registered.
    - 1 = FADump is registered and ready to handle system crash.

    To register FADump echo 1 > /sys/kernel/fadump_registered and
    echo 0 > /sys/kernel/fadump_registered for un-register and stop the
    FADump. Once the FADump is un-registered, the system crash will not
    be handled and vmcore will not be captured. This interface can be
    easily integrated with kdump service start/stop.

 /sys/kernel/fadump/mem_reserved

   This is used to display the memory reserved by FADump for saving the
   crash dump.

 /sys/kernel/fadump_release_mem
    This file is available only when FADump is active during
    second kernel. This is used to release the reserved memory
    region that are held for saving crash dump. To release the
    reserved memory echo 1 to it::

	echo 1  > /sys/kernel/fadump_release_mem

    After echo 1, the content of the /sys/kernel/debug/powerpc/fadump_region
    file will change to reflect the new memory reservations.

    The existing userspace tools (kdump infrastructure) can be easily
    enhanced to use this interface to release the memory reserved for
    dump and continue without 2nd reboot.

Note: /sys/kernel/fadump_release_opalcore sysfs has moved to
      /sys/firmware/opal/mpipl/release_core

 /sys/firmware/opal/mpipl/release_core

    This file is available only on OPAL based machines when FADump is
    active during capture kernel. This is used to release the memory
    used by the kernel to export /sys/firmware/opal/mpipl/core file. To
    release this memory, echo '1' to it:

    echo 1  > /sys/firmware/opal/mpipl/release_core

Note: The following FADump sysfs files are deprecated.

+----------------------------------+--------------------------------+
| Deprecated                       | Alternative                    |
+----------------------------------+--------------------------------+
| /sys/kernel/fadump_enabled       | /sys/kernel/fadump/enabled     |
+----------------------------------+--------------------------------+
| /sys/kernel/fadump_registered    | /sys/kernel/fadump/registered  |
+----------------------------------+--------------------------------+
| /sys/kernel/fadump_release_mem   | /sys/kernel/fadump/release_mem |
+----------------------------------+--------------------------------+

Here is the list of files under powerpc debugfs:
(Assuming debugfs is mounted on /sys/kernel/debug directory.)

 /sys/kernel/debug/powerpc/fadump_region
    This file shows the reserved memory regions if FADump is
    enabled otherwise this file is empty. The output format
    is::

      <region>: [<start>-<end>] <reserved-size> bytes, Dumped: <dump-size>

    and for kernel DUMP region is:

    DUMP: Src: <src-addr>, Dest: <dest-addr>, Size: <size>, Dumped: # bytes

    e.g.
    Contents when FADump is registered during first kernel::

      # cat /sys/kernel/debug/powerpc/fadump_region
      CPU : [0x0000006ffb0000-0x0000006fff001f] 0x40020 bytes, Dumped: 0x0
      HPTE: [0x0000006fff0020-0x0000006fff101f] 0x1000 bytes, Dumped: 0x0
      DUMP: [0x0000006fff1020-0x0000007fff101f] 0x10000000 bytes, Dumped: 0x0

    Contents when FADump is active during second kernel::

      # cat /sys/kernel/debug/powerpc/fadump_region
      CPU : [0x0000006ffb0000-0x0000006fff001f] 0x40020 bytes, Dumped: 0x40020
      HPTE: [0x0000006fff0020-0x0000006fff101f] 0x1000 bytes, Dumped: 0x1000
      DUMP: [0x0000006fff1020-0x0000007fff101f] 0x10000000 bytes, Dumped: 0x10000000
          : [0x00000010000000-0x0000006ffaffff] 0x5ffb0000 bytes, Dumped: 0x5ffb0000

NOTE:
      Please refer to Documentation/filesystems/debugfs.rst on
      how to mount the debugfs filesystem.

TODO:
-----
 - Need to come up with the better approach to find out more
   accurate boot memory size that is required for a kernel to
   boot successfully when booted with restricted memory.
 - The FADump implementation introduces a FADump crash info structure
   in the scratch area before the ELF core header. The idea of introducing
   this structure is to pass some important crash info data to the second
   kernel which will help second kernel to populate ELF core header with
   correct data before it gets exported through /proc/vmcore. The current
   design implementation does not address a possibility of introducing
   additional fields (in future) to this structure without affecting
   compatibility. Need to come up with the better approach to address this.

   The possible approaches are:

	1. Introduce version field for version tracking, bump up the version
	whenever a new field is added to the structure in future. The version
	field can be used to find out what fields are valid for the current
	version of the structure.
	2. Reserve the area of predefined size (say PAGE_SIZE) for this
	structure and have unused area as reserved (initialized to zero)
	for future field additions.

   The advantage of approach 1 over 2 is we don't need to reserve extra space.

Author: Mahesh Salgaonkar <mahesh@linux.vnet.ibm.com>

This document is based on the original documentation written for phyp

assisted dump by Linas Vepstas and Manish Ahuja.


code:
===============================================================
HVCS IBM "Hypervisor Virtual Console Server" Installation Guide
===============================================================

for Linux Kernel 2.6.4+

Copyright (C) 2004 IBM Corporation

.. ===========================================================================
.. NOTE:Eight space tabs are the optimum editor setting for reading this file.
.. ===========================================================================

Author(s): Ryan S. Arnold <rsa@us.ibm.com>

Date Created: March, 02, 2004
Last Changed: August, 24, 2004

.. Table of contents:

	1.  Driver Introduction:
	2.  System Requirements
	3.  Build Options:
		3.1  Built-in:
		3.2  Module:
	4.  Installation:
	5.  Connection:
	6.  Disconnection:
	7.  Configuration:
	8.  Questions & Answers:
	9.  Reporting Bugs:

1. Driver Introduction:
=======================

This is the device driver for the IBM Hypervisor Virtual Console Server,
"hvcs".  The IBM hvcs provides a tty driver interface to allow Linux user
space applications access to the system consoles of logically partitioned
operating systems (Linux and AIX) running on the same partitioned Power5
ppc64 system.  Physical hardware consoles per partition are not practical
on this hardware so system consoles are accessed by this driver using
firmware interfaces to virtual terminal devices.

2. System Requirements:
=======================

This device driver was written using 2.6.4 Linux kernel APIs and will only
build and run on kernels of this version or later.

This driver was written to operate solely on IBM Power5 ppc64 hardware
though some care was taken to abstract the architecture dependent firmware
calls from the driver code.

Sysfs must be mounted on the system so that the user can determine which
major and minor numbers are associated with each vty-server.  Directions
for sysfs mounting are outside the scope of this document.

3. Build Options:
=================

The hvcs driver registers itself as a tty driver.  The tty layer
dynamically allocates a block of major and minor numbers in a quantity
requested by the registering driver.  The hvcs driver asks the tty layer
for 64 of these major/minor numbers by default to use for hvcs device node
entries.

If the default number of device entries is adequate then this driver can be
built into the kernel.  If not, the default can be over-ridden by inserting
the driver as a module with insmod parameters.

3.1 Built-in:
-------------

The following menuconfig example demonstrates selecting to build this
driver into the kernel::

	Device Drivers  --->
		Character devices  --->
			<*> IBM Hypervisor Virtual Console Server Support

Begin the kernel make process.

3.2 Module:
-----------

The following menuconfig example demonstrates selecting to build this
driver as a kernel module::

	Device Drivers  --->
		Character devices  --->
			<M> IBM Hypervisor Virtual Console Server Support

The make process will build the following kernel modules:

	- hvcs.ko
	- hvcserver.ko

To insert the module with the default allocation execute the following
commands in the order they appear::

	insmod hvcserver.ko
	insmod hvcs.ko

The hvcserver module contains architecture specific firmware calls and must
be inserted first, otherwise the hvcs module will not find some of the
symbols it expects.

To override the default use an insmod parameter as follows (requesting 4
tty devices as an example)::

	insmod hvcs.ko hvcs_parm_num_devs=4

There is a maximum number of dev entries that can be specified on insmod.
We think that 1024 is currently a decent maximum number of server adapters
to allow.  This can always be changed by modifying the constant in the
source file before building.

NOTE: The length of time it takes to insmod the driver seems to be related
to the number of tty interfaces the registering driver requests.

In order to remove the driver module execute the following command::

	rmmod hvcs.ko

The recommended method for installing hvcs as a module is to use depmod to
build a current modules.dep file in /lib/modules/`uname -r` and then
execute::

	modprobe hvcs hvcs_parm_num_devs=4

The modules.dep file indicates that hvcserver.ko needs to be inserted
before hvcs.ko and modprobe uses this file to smartly insert the modules in
the proper order.

The following modprobe command is used to remove hvcs and hvcserver in the
proper order::

	modprobe -r hvcs

4. Installation:
================

The tty layer creates sysfs entries which contain the major and minor
numbers allocated for the hvcs driver.  The following snippet of "tree"
output of the sysfs directory shows where these numbers are presented::

	sys/
	|-- *other sysfs base dirs*
	|
	|-- class
	|   |-- *other classes of devices*
	|   |
	|   `-- tty
	|       |-- *other tty devices*
	|       |
	|       |-- hvcs0
	|       |   `-- dev
	|       |-- hvcs1
	|       |   `-- dev
	|       |-- hvcs2
	|       |   `-- dev
	|       |-- hvcs3
	|       |   `-- dev
	|       |
	|       |-- *other tty devices*
	|
	|-- *other sysfs base dirs*

For the above examples the following output is a result of cat'ing the
"dev" entry in the hvcs directory::

	Pow5:/sys/class/tty/hvcs0/ # cat dev
	254:0

	Pow5:/sys/class/tty/hvcs1/ # cat dev
	254:1

	Pow5:/sys/class/tty/hvcs2/ # cat dev
	254:2

	Pow5:/sys/class/tty/hvcs3/ # cat dev
	254:3

The output from reading the "dev" attribute is the char device major and
minor numbers that the tty layer has allocated for this driver's use.  Most
systems running hvcs will already have the device entries created or udev
will do it automatically.

Given the example output above, to manually create a /dev/hvcs* node entry
mknod can be used as follows::

	mknod /dev/hvcs0 c 254 0
	mknod /dev/hvcs1 c 254 1
	mknod /dev/hvcs2 c 254 2
	mknod /dev/hvcs3 c 254 3

Using mknod to manually create the device entries makes these device nodes
persistent.  Once created they will exist prior to the driver insmod.

Attempting to connect an application to /dev/hvcs* prior to insertion of
the hvcs module will result in an error message similar to the following::

	"/dev/hvcs*: No such device".

NOTE: Just because there is a device node present doesn't mean that there
is a vty-server device configured for that node.

5. Connection
=============

Since this driver controls devices that provide a tty interface a user can
interact with the device node entries using any standard tty-interactive
method (e.g. "cat", "dd", "echo").  The intent of this driver however, is
to provide real time console interaction with a Linux partition's console,
which requires the use of applications that provide bi-directional,
interactive I/O with a tty device.

Applications (e.g. "minicom" and "screen") that act as terminal emulators
or perform terminal type control sequence conversion on the data being
passed through them are NOT acceptable for providing interactive console
I/O.  These programs often emulate antiquated terminal types (vt100 and
ANSI) and expect inbound data to take the form of one of these supported
terminal types but they either do not convert, or do not _adequately_
convert, outbound data into the terminal type of the terminal which invoked
them (though screen makes an attempt and can apparently be configured with
much termcap wrestling.)

For this reason kermit and cu are two of the recommended applications for
interacting with a Linux console via an hvcs device.  These programs simply
act as a conduit for data transfer to and from the tty device.  They do not
require inbound data to take the form of a particular terminal type, nor do
they cook outbound data to a particular terminal type.

In order to ensure proper functioning of console applications one must make
sure that once connected to a /dev/hvcs console that the console's $TERM
env variable is set to the exact terminal type of the terminal emulator
used to launch the interactive I/O application.  If one is using xterm and
kermit to connect to /dev/hvcs0 when the console prompt becomes available
one should "export TERM=xterm" on the console.  This tells ncurses
applications that are invoked from the console that they should output
control sequences that xterm can understand.

As a precautionary measure an hvcs user should always "exit" from their
session before disconnecting an application such as kermit from the device
node.  If this is not done, the next user to connect to the console will
continue using the previous user's logged in session which includes
using the $TERM variable that the previous user supplied.

Hotplug add and remove of vty-server adapters affects which /dev/hvcs* node
is used to connect to each vty-server adapter.  In order to determine which
vty-server adapter is associated with which /dev/hvcs* node a special sysfs
attribute has been added to each vty-server sysfs entry.  This entry is
called "index" and showing it reveals an integer that refers to the
/dev/hvcs* entry to use to connect to that device.  For instance cating the
index attribute of vty-server adapter 30000004 shows the following::

	Pow5:/sys/bus/vio/drivers/hvcs/30000004 # cat index
	2

This index of '2' means that in order to connect to vty-server adapter
30000004 the user should interact with /dev/hvcs2.

It should be noted that due to the system hotplug I/O capabilities of a
system the /dev/hvcs* entry that interacts with a particular vty-server
adapter is not guaranteed to remain the same across system reboots.  Look
in the Q & A section for more on this issue.

6. Disconnection
================

As a security feature to prevent the delivery of stale data to an
unintended target the Power5 system firmware disables the fetching of data
and discards that data when a connection between a vty-server and a vty has
been severed.  As an example, when a vty-server is immediately disconnected
from a vty following output of data to the vty the vty adapter may not have
enough time between when it received the data interrupt and when the
connection was severed to fetch the data from firmware before the fetch is
disabled by firmware.

When hvcs is being used to serve consoles this behavior is not a huge issue
because the adapter stays connected for large amounts of time following
almost all data writes.  When hvcs is being used as a tty conduit to tunnel
data between two partitions [see Q & A below] this is a huge problem
because the standard Linux behavior when cat'ing or dd'ing data to a device
is to open the tty, send the data, and then close the tty.  If this driver
manually terminated vty-server connections on tty close this would close
the vty-server and vty connection before the target vty has had a chance to
fetch the data.

Additionally, disconnecting a vty-server and vty only on module removal or
adapter removal is impractical because other vty-servers in other
partitions may require the usage of the target vty at any time.

Due to this behavioral restriction disconnection of vty-servers from the
connected vty is a manual procedure using a write to a sysfs attribute
outlined below, on the other hand the initial vty-server connection to a
vty is established automatically by this driver.  Manual vty-server
connection is never required.

In order to terminate the connection between a vty-server and vty the
"vterm_state" sysfs attribute within each vty-server's sysfs entry is used.
Reading this attribute reveals the current connection state of the
vty-server adapter.  A zero means that the vty-server is not connected to a
vty.  A one indicates that a connection is active.

Writing a '0' (zero) to the vterm_state attribute will disconnect the VTERM
connection between the vty-server and target vty ONLY if the vterm_state
previously read '1'.  The write directive is ignored if the vterm_state
read '0' or if any value other than '0' was written to the vterm_state
attribute.  The following example will show the method used for verifying
the vty-server connection status and disconnecting a vty-server connection::

	Pow5:/sys/bus/vio/drivers/hvcs/30000004 # cat vterm_state
	1

	Pow5:/sys/bus/vio/drivers/hvcs/30000004 # echo 0 > vterm_state

	Pow5:/sys/bus/vio/drivers/hvcs/30000004 # cat vterm_state
	0

All vty-server connections are automatically terminated when the device is
hotplug removed and when the module is removed.

7. Configuration
================

Each vty-server has a sysfs entry in the /sys/devices/vio directory, which
is symlinked in several other sysfs tree directories, notably under the
hvcs driver entry, which looks like the following example::

	Pow5:/sys/bus/vio/drivers/hvcs # ls
	.  ..  30000003  30000004  rescan

By design, firmware notifies the hvcs driver of vty-server lifetimes and
partner vty removals but not the addition of partner vtys.  Since an HMC
Super Admin can add partner info dynamically we have provided the hvcs
driver sysfs directory with the "rescan" update attribute which will query
firmware and update the partner info for all the vty-servers that this
driver manages.  Writing a '1' to the attribute triggers the update.  An
explicit example follows:

	Pow5:/sys/bus/vio/drivers/hvcs # echo 1 > rescan

Reading the attribute will indicate a state of '1' or '0'.  A one indicates
that an update is in process.  A zero indicates that an update has
completed or was never executed.

Vty-server entries in this directory are a 32 bit partition unique unit
address that is created by firmware.  An example vty-server sysfs entry
looks like the following::

	Pow5:/sys/bus/vio/drivers/hvcs/30000004 # ls
	.   current_vty   devspec       name          partner_vtys
	..  index         partner_clcs  vterm_state

Each entry is provided, by default with a "name" attribute.  Reading the
"name" attribute will reveal the device type as shown in the following
example::

	Pow5:/sys/bus/vio/drivers/hvcs/30000003 # cat name
	vty-server

Each entry is also provided, by default, with a "devspec" attribute which
reveals the full device specification when read, as shown in the following
example::

	Pow5:/sys/bus/vio/drivers/hvcs/30000004 # cat devspec
	/vdevice/vty-server@30000004

Each vty-server sysfs dir is provided with two read-only attributes that
provide lists of easily parsed partner vty data: "partner_vtys" and
"partner_clcs"::

	Pow5:/sys/bus/vio/drivers/hvcs/30000004 # cat partner_vtys
	30000000
	30000001
	30000002
	30000000
	30000000

	Pow5:/sys/bus/vio/drivers/hvcs/30000004 # cat partner_clcs
	U5112.428.103048A-V3-C0
	U5112.428.103048A-V3-C2
	U5112.428.103048A-V3-C3
	U5112.428.103048A-V4-C0
	U5112.428.103048A-V5-C0

Reading partner_vtys returns a list of partner vtys.  Vty unit address
numbering is only per-partition-unique so entries will frequently repeat.

Reading partner_clcs returns a list of "converged location codes" which are
composed of a system serial number followed by "-V*", where the '*' is the
target partition number, and "-C*", where the '*' is the slot of the
adapter.  The first vty partner corresponds to the first clc item, the
second vty partner to the second clc item, etc.

A vty-server can only be connected to a single vty at a time.  The entry,
"current_vty" prints the clc of the currently selected partner vty when
read.

The current_vty can be changed by writing a valid partner clc to the entry
as in the following example::

	Pow5:/sys/bus/vio/drivers/hvcs/30000004 # echo U5112.428.10304
	8A-V4-C0 > current_vty

Changing the current_vty when a vty-server is already connected to a vty
does not affect the current connection.  The change takes effect when the
currently open connection is freed.

Information on the "vterm_state" attribute was covered earlier on the
chapter entitled "disconnection".

8. Questions & Answers:
=======================

Q: What are the security concerns involving hvcs?

A: There are three main security concerns:

	1. The creator of the /dev/hvcs* nodes has the ability to restrict
	the access of the device entries to certain users or groups.  It
	may be best to create a special hvcs group privilege for providing
	access to system consoles.

	2. To provide network security when grabbing the console it is
	suggested that the user connect to the console hosting partition
	using a secure method, such as SSH or sit at a hardware console.

	3. Make sure to exit the user session when done with a console or
	the next vty-server connection (which may be from another
	partition) will experience the previously logged in session.

---------------------------------------------------------------------------

Q: How do I multiplex a console that I grab through hvcs so that other
people can see it:

A: You can use "screen" to directly connect to the /dev/hvcs* device and
setup a session on your machine with the console group privileges.  As
pointed out earlier by default screen doesn't provide the termcap settings
for most terminal emulators to provide adequate character conversion from
term type "screen" to others.  This means that curses based programs may
not display properly in screen sessions.

---------------------------------------------------------------------------

Q: Why are the colors all messed up?
Q: Why are the control characters acting strange or not working?
Q: Why is the console output all strange and unintelligible?

A: Please see the preceding section on "Connection" for a discussion of how
applications can affect the display of character control sequences.
Additionally, just because you logged into the console using and xterm
doesn't mean someone else didn't log into the console with the HMC console
(vt320) before you and leave the session logged in.  The best thing to do
is to export TERM to the terminal type of your terminal emulator when you
get the console.  Additionally make sure to "exit" the console before you
disconnect from the console.  This will ensure that the next user gets
their own TERM type set when they login.

---------------------------------------------------------------------------

Q: When I try to CONNECT kermit to an hvcs device I get:
"Sorry, can't open connection: /dev/hvcs*"What is happening?

A: Some other Power5 console mechanism has a connection to the vty and
isn't giving it up.  You can try to force disconnect the consoles from the
HMC by right clicking on the partition and then selecting "close terminal".
Otherwise you have to hunt down the people who have console authority.  It
is possible that you already have the console open using another kermit
session and just forgot about it.  Please review the console options for
Power5 systems to determine the many ways a system console can be held.

OR

A: Another user may not have a connectivity method currently attached to a
/dev/hvcs device but the vterm_state may reveal that they still have the
vty-server connection established.  They need to free this using the method
outlined in the section on "Disconnection" in order for others to connect
to the target vty.

OR

A: The user profile you are using to execute kermit probably doesn't have
permissions to use the /dev/hvcs* device.

OR

A: You probably haven't inserted the hvcs.ko module yet but the /dev/hvcs*
entry still exists (on systems without udev).

OR

A: There is not a corresponding vty-server device that maps to an existing
/dev/hvcs* entry.

---------------------------------------------------------------------------

Q: When I try to CONNECT kermit to an hvcs device I get:
"Sorry, write access to UUCP lockfile directory denied."

A: The /dev/hvcs* entry you have specified doesn't exist where you said it
does?  Maybe you haven't inserted the module (on systems with udev).

---------------------------------------------------------------------------

Q: If I already have one Linux partition installed can I use hvcs on said
partition to provide the console for the install of a second Linux
partition?

A: Yes granted that your are connected to the /dev/hvcs* device using
kermit or cu or some other program that doesn't provide terminal emulation.

---------------------------------------------------------------------------

Q: Can I connect to more than one partition's console at a time using this
driver?

A: Yes.  Of course this means that there must be more than one vty-server
configured for this partition and each must point to a disconnected vty.

---------------------------------------------------------------------------

Q: Does the hvcs driver support dynamic (hotplug) addition of devices?

A: Yes, if you have dlpar and hotplug enabled for your system and it has
been built into the kernel the hvcs drivers is configured to dynamically
handle additions of new devices and removals of unused devices.

---------------------------------------------------------------------------

Q: For some reason /dev/hvcs* doesn't map to the same vty-server adapter
after a reboot.  What happened?

A: Assignment of vty-server adapters to /dev/hvcs* entries is always done
in the order that the adapters are exposed.  Due to hotplug capabilities of
this driver assignment of hotplug added vty-servers may be in a different
order than how they would be exposed on module load.  Rebooting or
reloading the module after dynamic addition may result in the /dev/hvcs*
and vty-server coupling changing if a vty-server adapter was added in a
slot between two other vty-server adapters.  Refer to the section above
on how to determine which vty-server goes with which /dev/hvcs* node.
Hint; look at the sysfs "index" attribute for the vty-server.

---------------------------------------------------------------------------

Q: Can I use /dev/hvcs* as a conduit to another partition and use a tty
device on that partition as the other end of the pipe?

A: Yes, on Power5 platforms the hvc_console driver provides a tty interface
for extra /dev/hvc* devices (where /dev/hvc0 is most likely the console).
In order to get a tty conduit working between the two partitions the HMC
Super Admin must create an additional "serial server" for the target
partition with the HMC gui which will show up as /dev/hvc* when the target
partition is rebooted.

The HMC Super Admin then creates an additional "serial client" for the
current partition and points this at the target partition's newly created
"serial server" adapter (remember the slot).  This shows up as an
additional /dev/hvcs* device.

Now a program on the target system can be configured to read or write to
/dev/hvc* and another program on the current partition can be configured to
read or write to /dev/hvcs*.  Now you have a tty conduit between two
partitions.

---------------------------------------------------------------------------

9. Reporting Bugs:
==================

The proper channel for reporting bugs is either through the Linux OS
distribution company that provided your OS or by posting issues to the
PowerPC development mailing list at:

linuxppc-dev@lists.ozlabs.org

This request is to provide a documented and searchable public exchange
of the problems and solutions surrounding this driver for the benefit of
all users.