code:
===========================
ARM64 CPU Feature Registers
===========================

Author: Suzuki K Poulose <suzuki.poulose@arm.com>

This file describes the ABI for exporting the AArch64 CPU ID/feature
registers to userspace. The availability of this ABI is advertised
via the HWCAP_CPUID in HWCAPs.

1. Motivation
-------------

The ARM architecture defines a set of feature registers, which describe
the capabilities of the CPU/system. Access to these system registers is
restricted from EL0 and there is no reliable way for an application to
extract this information to make better decisions at runtime. There is
limited information available to the application via HWCAPs, however
there are some issues with their usage.

 a) Any change to the HWCAPs requires an update to userspace (e.g libc)
    to detect the new changes, which can take a long time to appear in
    distributions. Exposing the registers allows applications to get the
    information without requiring updates to the toolchains.

 b) Access to HWCAPs is sometimes limited (e.g prior to libc, or
    when ld is initialised at startup time).

 c) HWCAPs cannot represent non-boolean information effectively. The
    architecture defines a canonical format for representing features
    in the ID registers; this is well defined and is capable of
    representing all valid architecture variations.

2. Requirements
---------------

 a) Safety:

    Applications should be able to use the information provided by the
    infrastructure to run safely across the system. This has greater
    implications on a system with heterogeneous CPUs.
    The infrastructure exports a value that is safe across all the
    available CPU on the system.

    e.g, If at least one CPU doesn't implement CRC32 instructions, while
    others do, we should report that the CRC32 is not implemented.
    Otherwise an application could crash when scheduled on the CPU
    which doesn't support CRC32.

 b) Security:

    Applications should only be able to receive information that is
    relevant to the normal operation in userspace. Hence, some of the
    fields are masked out(i.e, made invisible) and their values are set to
    indicate the feature is 'not supported'. See Section 4 for the list
    of visible features. Also, the kernel may manipulate the fields
    based on what it supports. e.g, If FP is not supported by the
    kernel, the values could indicate that the FP is not available
    (even when the CPU provides it).

 c) Implementation Defined Features

    The infrastructure doesn't expose any register which is
    IMPLEMENTATION DEFINED as per ARMv8-A Architecture.

 d) CPU Identification:

    MIDR_EL1 is exposed to help identify the processor. On a
    heterogeneous system, this could be racy (just like getcpu()). The
    process could be migrated to another CPU by the time it uses the
    register value, unless the CPU affinity is set. Hence, there is no
    guarantee that the value reflects the processor that it is
    currently executing on. The REVIDR is not exposed due to this
    constraint, as REVIDR makes sense only in conjunction with the
    MIDR. Alternately, MIDR_EL1 and REVIDR_EL1 are exposed via sysfs
    at::

	/sys/devices/system/cpu/cpu$ID/regs/identification/
	                                              \- midr
	                                              \- revidr

3. Implementation
--------------------

The infrastructure is built on the emulation of the 'MRS' instruction.
Accessing a restricted system register from an application generates an
exception and ends up in SIGILL being delivered to the process.
The infrastructure hooks into the exception handler and emulates the
operation if the source belongs to the supported system register space.

The infrastructure emulates only the following system register space::

	Op0=3, Op1=0, CRn=0, CRm=0,2,3,4,5,6,7

(See Table C5-6 'System instruction encodings for non-Debug System
register accesses' in ARMv8 ARM DDI 0487A.h, for the list of
registers).

The following rules are applied to the value returned by the
infrastructure:

 a) The value of an 'IMPLEMENTATION DEFINED' field is set to 0.
 b) The value of a reserved field is populated with the reserved
    value as defined by the architecture.
 c) The value of a 'visible' field holds the system wide safe value
    for the particular feature (except for MIDR_EL1, see section 4).
 d) All other fields (i.e, invisible fields) are set to indicate
    the feature is missing (as defined by the architecture).

4. List of registers with visible features
-------------------------------------------

  1) ID_AA64ISAR0_EL1 - Instruction Set Attribute Register 0

     +------------------------------+---------+---------+
     | Name                         |  bits   | visible |
     +------------------------------+---------+---------+
     | RNDR                         | [63-60] |    y    |
     +------------------------------+---------+---------+
     | TS                           | [55-52] |    y    |
     +------------------------------+---------+---------+
     | FHM                          | [51-48] |    y    |
     +------------------------------+---------+---------+
     | DP                           | [47-44] |    y    |
     +------------------------------+---------+---------+
     | SM4                          | [43-40] |    y    |
     +------------------------------+---------+---------+
     | SM3                          | [39-36] |    y    |
     +------------------------------+---------+---------+
     | SHA3                         | [35-32] |    y    |
     +------------------------------+---------+---------+
     | RDM                          | [31-28] |    y    |
     +------------------------------+---------+---------+
     | ATOMICS                      | [23-20] |    y    |
     +------------------------------+---------+---------+
     | CRC32                        | [19-16] |    y    |
     +------------------------------+---------+---------+
     | SHA2                         | [15-12] |    y    |
     +------------------------------+---------+---------+
     | SHA1                         | [11-8]  |    y    |
     +------------------------------+---------+---------+
     | AES                          | [7-4]   |    y    |
     +------------------------------+---------+---------+

  2) ID_AA64PFR0_EL1 - Processor Feature Register 0

     +------------------------------+---------+---------+
     | Name                         |  bits   | visible |
     +------------------------------+---------+---------+
     | DIT                          | [51-48] |    y    |
     +------------------------------+---------+---------+
     | SVE                          | [35-32] |    y    |
     +------------------------------+---------+---------+
     | GIC                          | [27-24] |    n    |
     +------------------------------+---------+---------+
     | AdvSIMD                      | [23-20] |    y    |
     +------------------------------+---------+---------+
     | FP                           | [19-16] |    y    |
     +------------------------------+---------+---------+
     | EL3                          | [15-12] |    n    |
     +------------------------------+---------+---------+
     | EL2                          | [11-8]  |    n    |
     +------------------------------+---------+---------+
     | EL1                          | [7-4]   |    n    |
     +------------------------------+---------+---------+
     | EL0                          | [3-0]   |    n    |
     +------------------------------+---------+---------+

  3) ID_AA64PFR1_EL1 - Processor Feature Register 1

     +------------------------------+---------+---------+
     | Name                         |  bits   | visible |
     +------------------------------+---------+---------+
     | MTE                          | [11-8]  |    y    |
     +------------------------------+---------+---------+
     | SSBS                         | [7-4]   |    y    |
     +------------------------------+---------+---------+
     | BT                           | [3-0]   |    y    |
     +------------------------------+---------+---------+

  4) MIDR_EL1 - Main ID Register

     +------------------------------+---------+---------+
     | Name                         |  bits   | visible |
     +------------------------------+---------+---------+
     | Implementer                  | [31-24] |    y    |
     +------------------------------+---------+---------+
     | Variant                      | [23-20] |    y    |
     +------------------------------+---------+---------+
     | Architecture                 | [19-16] |    y    |
     +------------------------------+---------+---------+
     | PartNum                      | [15-4]  |    y    |
     +------------------------------+---------+---------+
     | Revision                     | [3-0]   |    y    |
     +------------------------------+---------+---------+

   NOTE: The 'visible' fields of MIDR_EL1 will contain the value
   as available on the CPU where it is fetched and is not a system
   wide safe value.

  5) ID_AA64ISAR1_EL1 - Instruction set attribute register 1

     +------------------------------+---------+---------+
     | Name                         |  bits   | visible |
     +------------------------------+---------+---------+
     | I8MM                         | [55-52] |    y    |
     +------------------------------+---------+---------+
     | DGH                          | [51-48] |    y    |
     +------------------------------+---------+---------+
     | BF16                         | [47-44] |    y    |
     +------------------------------+---------+---------+
     | SB                           | [39-36] |    y    |
     +------------------------------+---------+---------+
     | FRINTTS                      | [35-32] |    y    |
     +------------------------------+---------+---------+
     | GPI                          | [31-28] |    y    |
     +------------------------------+---------+---------+
     | GPA                          | [27-24] |    y    |
     +------------------------------+---------+---------+
     | LRCPC                        | [23-20] |    y    |
     +------------------------------+---------+---------+
     | FCMA                         | [19-16] |    y    |
     +------------------------------+---------+---------+
     | JSCVT                        | [15-12] |    y    |
     +------------------------------+---------+---------+
     | API                          | [11-8]  |    y    |
     +------------------------------+---------+---------+
     | APA                          | [7-4]   |    y    |
     +------------------------------+---------+---------+
     | DPB                          | [3-0]   |    y    |
     +------------------------------+---------+---------+

  6) ID_AA64MMFR0_EL1 - Memory model feature register 0

     +------------------------------+---------+---------+
     | Name                         |  bits   | visible |
     +------------------------------+---------+---------+
     | ECV                          | [63-60] |    y    |
     +------------------------------+---------+---------+

  7) ID_AA64MMFR2_EL1 - Memory model feature register 2

     +------------------------------+---------+---------+
     | Name                         |  bits   | visible |
     +------------------------------+---------+---------+
     | AT                           | [35-32] |    y    |
     +------------------------------+---------+---------+

  8) ID_AA64ZFR0_EL1 - SVE feature ID register 0

     +------------------------------+---------+---------+
     | Name                         |  bits   | visible |
     +------------------------------+---------+---------+
     | F64MM                        | [59-56] |    y    |
     +------------------------------+---------+---------+
     | F32MM                        | [55-52] |    y    |
     +------------------------------+---------+---------+
     | I8MM                         | [47-44] |    y    |
     +------------------------------+---------+---------+
     | SM4                          | [43-40] |    y    |
     +------------------------------+---------+---------+
     | SHA3                         | [35-32] |    y    |
     +------------------------------+---------+---------+
     | BF16                         | [23-20] |    y    |
     +------------------------------+---------+---------+
     | BitPerm                      | [19-16] |    y    |
     +------------------------------+---------+---------+
     | AES                          | [7-4]   |    y    |
     +------------------------------+---------+---------+
     | SVEVer                       | [3-0]   |    y    |
     +------------------------------+---------+---------+

  8) ID_AA64MMFR1_EL1 - Memory model feature register 1

     +------------------------------+---------+---------+
     | Name                         |  bits   | visible |
     +------------------------------+---------+---------+
     | AFP                          | [47-44] |    y    |
     +------------------------------+---------+---------+

  9) ID_AA64ISAR2_EL1 - Instruction set attribute register 2

     +------------------------------+---------+---------+
     | Name                         |  bits   | visible |
     +------------------------------+---------+---------+
     | RPRES                        | [7-4]   |    y    |
     +------------------------------+---------+---------+
     | WFXT                         | [3-0]   |    y    |
     +------------------------------+---------+---------+

  10) MVFR0_EL1 - AArch32 Media and VFP Feature Register 0

     +------------------------------+---------+---------+
     | Name                         |  bits   | visible |
     +------------------------------+---------+---------+
     | FPDP                         | [11-8]  |    y    |
     +------------------------------+---------+---------+

  11) MVFR1_EL1 - AArch32 Media and VFP Feature Register 1

     +------------------------------+---------+---------+
     | Name                         |  bits   | visible |
     +------------------------------+---------+---------+
     | SIMDFMAC                     | [31-28] |    y    |
     +------------------------------+---------+---------+
     | SIMDSP                       | [19-16] |    y    |
     +------------------------------+---------+---------+
     | SIMDInt                      | [15-12] |    y    |
     +------------------------------+---------+---------+
     | SIMDLS                       | [11-8]  |    y    |
     +------------------------------+---------+---------+

  12) ID_ISAR5_EL1 - AArch32 Instruction Set Attribute Register 5

     +------------------------------+---------+---------+
     | Name                         |  bits   | visible |
     +------------------------------+---------+---------+
     | CRC32                        | [19-16] |    y    |
     +------------------------------+---------+---------+
     | SHA2                         | [15-12] |    y    |
     +------------------------------+---------+---------+
     | SHA1                         | [11-8]  |    y    |
     +------------------------------+---------+---------+
     | AES                          | [7-4]   |    y    |
     +------------------------------+---------+---------+

Appendix I: Example
-------------------

::

  /*
   * Sample program to demonstrate the MRS emulation ABI.
   *
   * Copyright (C) 2015-2016, ARM Ltd
   *
   * Author: Suzuki K Poulose <suzuki.poulose@arm.com>
   *
   * This program is free software; you can redistribute it and/or modify
   * it under the terms of the GNU General Public License version 2 as
   * published by the Free Software Foundation.
   *
   * This program is distributed in the hope that it will be useful,
   * but WITHOUT ANY WARRANTY; without even the implied warranty of
   * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   * GNU General Public License for more details.
   * This program is free software; you can redistribute it and/or modify
   * it under the terms of the GNU General Public License version 2 as
   * published by the Free Software Foundation.
   *
   * This program is distributed in the hope that it will be useful,
   * but WITHOUT ANY WARRANTY; without even the implied warranty of
   * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   * GNU General Public License for more details.
   */

  #include <asm/hwcap.h>
  #include <stdio.h>
  #include <sys/auxv.h>

  #define get_cpu_ftr(id) ({					\
		unsigned long __val;				\
		asm("mrs %0, "#id : "=r" (__val));		\
		printf("%-20s: 0x%016lx\n", #id, __val);	\
	})

  int main(void)
  {

	if (!(getauxval(AT_HWCAP) & HWCAP_CPUID)) {
		fputs("CPUID registers unavailable\n", stderr);
		return 1;
	}

	get_cpu_ftr(ID_AA64ISAR0_EL1);
	get_cpu_ftr(ID_AA64ISAR1_EL1);
	get_cpu_ftr(ID_AA64MMFR0_EL1);
	get_cpu_ftr(ID_AA64MMFR1_EL1);
	get_cpu_ftr(ID_AA64PFR0_EL1);
	get_cpu_ftr(ID_AA64PFR1_EL1);
	get_cpu_ftr(ID_AA64DFR0_EL1);
	get_cpu_ftr(ID_AA64DFR1_EL1);

	get_cpu_ftr(MIDR_EL1);
	get_cpu_ftr(MPIDR_EL1);
	get_cpu_ftr(REVIDR_EL1);

  #if 0
	/* Unexposed register access causes SIGILL */
	get_cpu_ftr(ID_MMFR0_EL1);
  #endif

	return 0;
  }


code:
.. _elf_hwcaps_index:

================
ARM64 ELF hwcaps
================

This document describes the usage and semantics of the arm64 ELF hwcaps.

1. Introduction
---------------

Some hardware or software features are only available on some CPU
implementations, and/or with certain kernel configurations, but have no
architected discovery mechanism available to userspace code at EL0. The
kernel exposes the presence of these features to userspace through a set
of flags called hwcaps, exposed in the auxilliary vector.

Userspace software can test for features by acquiring the AT_HWCAP or
AT_HWCAP2 entry of the auxiliary vector, and testing whether the relevant
flags are set, e.g.::

	bool floating_point_is_present(void)
	{
		unsigned long hwcaps = getauxval(AT_HWCAP);
		if (hwcaps & HWCAP_FP)
			return true;

		return false;
	}

Where software relies on a feature described by a hwcap, it should check
the relevant hwcap flag to verify that the feature is present before
attempting to make use of the feature.

Features cannot be probed reliably through other means. When a feature
is not available, attempting to use it may result in unpredictable
behaviour, and is not guaranteed to result in any reliable indication
that the feature is unavailable, such as a SIGILL.

2. Interpretation of hwcaps
---------------------------

The majority of hwcaps are intended to indicate the presence of features
which are described by architected ID registers inaccessible to
userspace code at EL0. These hwcaps are defined in terms of ID register
fields, and should be interpreted with reference to the definition of
these fields in the ARM Architecture Reference Manual (ARM ARM).

Such hwcaps are described below in the form::

    Functionality implied by idreg.field == val.

Such hwcaps indicate the availability of functionality that the ARM ARM
defines as being present when idreg.field has value val, but do not
indicate that idreg.field is precisely equal to val, nor do they
indicate the absence of functionality implied by other values of
idreg.field.

Other hwcaps may indicate the presence of features which cannot be
described by ID registers alone. These may be described without
reference to ID registers, and may refer to other documentation.

3. The hwcaps exposed in AT_HWCAP
---------------------------------

HWCAP_FP
    Functionality implied by ID_AA64PFR0_EL1.FP == 0b0000.

HWCAP_ASIMD
    Functionality implied by ID_AA64PFR0_EL1.AdvSIMD == 0b0000.

HWCAP_EVTSTRM
    The generic timer is configured to generate events at a frequency of
    approximately 10KHz.

HWCAP_AES
    Functionality implied by ID_AA64ISAR0_EL1.AES == 0b0001.

HWCAP_PMULL
    Functionality implied by ID_AA64ISAR0_EL1.AES == 0b0010.

HWCAP_SHA1
    Functionality implied by ID_AA64ISAR0_EL1.SHA1 == 0b0001.

HWCAP_SHA2
    Functionality implied by ID_AA64ISAR0_EL1.SHA2 == 0b0001.

HWCAP_CRC32
    Functionality implied by ID_AA64ISAR0_EL1.CRC32 == 0b0001.

HWCAP_ATOMICS
    Functionality implied by ID_AA64ISAR0_EL1.Atomic == 0b0010.

HWCAP_FPHP
    Functionality implied by ID_AA64PFR0_EL1.FP == 0b0001.

HWCAP_ASIMDHP
    Functionality implied by ID_AA64PFR0_EL1.AdvSIMD == 0b0001.

HWCAP_CPUID
    EL0 access to certain ID registers is available, to the extent
    described by Documentation/arm64/cpu-feature-registers.rst.

    These ID registers may imply the availability of features.

HWCAP_ASIMDRDM
    Functionality implied by ID_AA64ISAR0_EL1.RDM == 0b0001.

HWCAP_JSCVT
    Functionality implied by ID_AA64ISAR1_EL1.JSCVT == 0b0001.

HWCAP_FCMA
    Functionality implied by ID_AA64ISAR1_EL1.FCMA == 0b0001.

HWCAP_LRCPC
    Functionality implied by ID_AA64ISAR1_EL1.LRCPC == 0b0001.

HWCAP_DCPOP
    Functionality implied by ID_AA64ISAR1_EL1.DPB == 0b0001.

HWCAP_SHA3
    Functionality implied by ID_AA64ISAR0_EL1.SHA3 == 0b0001.

HWCAP_SM3
    Functionality implied by ID_AA64ISAR0_EL1.SM3 == 0b0001.

HWCAP_SM4
    Functionality implied by ID_AA64ISAR0_EL1.SM4 == 0b0001.

HWCAP_ASIMDDP
    Functionality implied by ID_AA64ISAR0_EL1.DP == 0b0001.

HWCAP_SHA512
    Functionality implied by ID_AA64ISAR0_EL1.SHA2 == 0b0010.

HWCAP_SVE
    Functionality implied by ID_AA64PFR0_EL1.SVE == 0b0001.

HWCAP_ASIMDFHM
   Functionality implied by ID_AA64ISAR0_EL1.FHM == 0b0001.

HWCAP_DIT
    Functionality implied by ID_AA64PFR0_EL1.DIT == 0b0001.

HWCAP_USCAT
    Functionality implied by ID_AA64MMFR2_EL1.AT == 0b0001.

HWCAP_ILRCPC
    Functionality implied by ID_AA64ISAR1_EL1.LRCPC == 0b0010.

HWCAP_FLAGM
    Functionality implied by ID_AA64ISAR0_EL1.TS == 0b0001.

HWCAP_SSBS
    Functionality implied by ID_AA64PFR1_EL1.SSBS == 0b0010.

HWCAP_SB
    Functionality implied by ID_AA64ISAR1_EL1.SB == 0b0001.

HWCAP_PACA
    Functionality implied by ID_AA64ISAR1_EL1.APA == 0b0001 or
    ID_AA64ISAR1_EL1.API == 0b0001, as described by
    Documentation/arm64/pointer-authentication.rst.

HWCAP_PACG
    Functionality implied by ID_AA64ISAR1_EL1.GPA == 0b0001 or
    ID_AA64ISAR1_EL1.GPI == 0b0001, as described by
    Documentation/arm64/pointer-authentication.rst.

HWCAP2_DCPODP
    Functionality implied by ID_AA64ISAR1_EL1.DPB == 0b0010.

HWCAP2_SVE2
    Functionality implied by ID_AA64ZFR0_EL1.SVEVer == 0b0001.

HWCAP2_SVEAES
    Functionality implied by ID_AA64ZFR0_EL1.AES == 0b0001.

HWCAP2_SVEPMULL
    Functionality implied by ID_AA64ZFR0_EL1.AES == 0b0010.

HWCAP2_SVEBITPERM
    Functionality implied by ID_AA64ZFR0_EL1.BitPerm == 0b0001.

HWCAP2_SVESHA3
    Functionality implied by ID_AA64ZFR0_EL1.SHA3 == 0b0001.

HWCAP2_SVESM4
    Functionality implied by ID_AA64ZFR0_EL1.SM4 == 0b0001.

HWCAP2_FLAGM2
    Functionality implied by ID_AA64ISAR0_EL1.TS == 0b0010.

HWCAP2_FRINT
    Functionality implied by ID_AA64ISAR1_EL1.FRINTTS == 0b0001.

HWCAP2_SVEI8MM
    Functionality implied by ID_AA64ZFR0_EL1.I8MM == 0b0001.

HWCAP2_SVEF32MM
    Functionality implied by ID_AA64ZFR0_EL1.F32MM == 0b0001.

HWCAP2_SVEF64MM
    Functionality implied by ID_AA64ZFR0_EL1.F64MM == 0b0001.

HWCAP2_SVEBF16
    Functionality implied by ID_AA64ZFR0_EL1.BF16 == 0b0001.

HWCAP2_I8MM
    Functionality implied by ID_AA64ISAR1_EL1.I8MM == 0b0001.

HWCAP2_BF16
    Functionality implied by ID_AA64ISAR1_EL1.BF16 == 0b0001.

HWCAP2_DGH
    Functionality implied by ID_AA64ISAR1_EL1.DGH == 0b0001.

HWCAP2_RNG
    Functionality implied by ID_AA64ISAR0_EL1.RNDR == 0b0001.

HWCAP2_BTI
    Functionality implied by ID_AA64PFR0_EL1.BT == 0b0001.

HWCAP2_MTE
    Functionality implied by ID_AA64PFR1_EL1.MTE == 0b0010, as described
    by Documentation/arm64/memory-tagging-extension.rst.

HWCAP2_ECV
    Functionality implied by ID_AA64MMFR0_EL1.ECV == 0b0001.

HWCAP2_AFP
    Functionality implied by ID_AA64MFR1_EL1.AFP == 0b0001.

HWCAP2_RPRES
    Functionality implied by ID_AA64ISAR2_EL1.RPRES == 0b0001.

HWCAP2_MTE3
    Functionality implied by ID_AA64PFR1_EL1.MTE == 0b0011, as described
    by Documentation/arm64/memory-tagging-extension.rst.

HWCAP2_SME
    Functionality implied by ID_AA64PFR1_EL1.SME == 0b0001, as described
    by Documentation/arm64/sme.rst.

HWCAP2_SME_I16I64
    Functionality implied by ID_AA64SMFR0_EL1.I16I64 == 0b1111.

HWCAP2_SME_F64F64
    Functionality implied by ID_AA64SMFR0_EL1.F64F64 == 0b1.

HWCAP2_SME_I8I32
    Functionality implied by ID_AA64SMFR0_EL1.I8I32 == 0b1111.

HWCAP2_SME_F16F32
    Functionality implied by ID_AA64SMFR0_EL1.F16F32 == 0b1.

HWCAP2_SME_B16F32
    Functionality implied by ID_AA64SMFR0_EL1.B16F32 == 0b1.

HWCAP2_SME_F32F32
    Functionality implied by ID_AA64SMFR0_EL1.F32F32 == 0b1.

HWCAP2_SME_FA64
    Functionality implied by ID_AA64SMFR0_EL1.FA64 == 0b1.

HWCAP2_WFXT
    Functionality implied by ID_AA64ISAR2_EL1.WFXT == 0b0010.

HWCAP2_EBF16
    Functionality implied by ID_AA64ISAR1_EL1.BF16 == 0b0010.

HWCAP2_SVE_EBF16
    Functionality implied by ID_AA64ZFR0_EL1.BF16 == 0b0010.

HWCAP2_CSSC
    Functionality implied by ID_AA64ISAR2_EL1.CSSC == 0b0001.

HWCAP2_RPRFM
    Functionality implied by ID_AA64ISAR2_EL1.RPRFM == 0b0001.

HWCAP2_SVE2P1
    Functionality implied by ID_AA64ZFR0_EL1.SVEver == 0b0010.

4. Unused AT_HWCAP bits
-----------------------

For interoperation with userspace, the kernel guarantees that bits 62
and 63 of AT_HWCAP will always be returned as 0.

repo_name: angery/linux


code:
.. SPDX-License-Identifier: GPL-2.0

.. kernel-feat:: $srctree/Documentation/features arm64


code:
.. _hugetlbpage_index:

====================
HugeTLBpage on ARM64
====================

Hugepage relies on making efficient use of TLBs to improve performance of
address translations. The benefit depends on both -

  - the size of hugepages
  - size of entries supported by the TLBs

The ARM64 port supports two flavours of hugepages.

1) Block mappings at the pud/pmd level
--------------------------------------

These are regular hugepages where a pmd or a pud page table entry points to a
block of memory. Regardless of the supported size of entries in TLB, block
mappings reduce the depth of page table walk needed to translate hugepage
addresses.

2) Using the Contiguous bit
---------------------------

The architecture provides a contiguous bit in the translation table entries
(D4.5.3, ARM DDI 0487C.a) that hints to the MMU to indicate that it is one of a
contiguous set of entries that can be cached in a single TLB entry.

The contiguous bit is used in Linux to increase the mapping size at the pmd and
pte (last) level. The number of supported contiguous entries varies by page size
and level of the page table.

The following hugepage sizes are supported -

  ====== ========   ====    ========    ===
  -      CONT PTE    PMD    CONT PMD    PUD
  ====== ========   ====    ========    ===
  4K:         64K     2M         32M     1G
  16K:         2M    32M          1G
  64K:         2M   512M         16G
  ====== ========   ====    ========    ===


code:
.. _arm64_index:

==================
ARM64 Architecture
==================

.. toctree::
    :maxdepth: 1

    acpi_object_usage
    amu
    arm-acpi
    asymmetric-32bit
    booting
    cpu-feature-registers
    elf_hwcaps
    hugetlbpage
    legacy_instructions
    memory
    memory-tagging-extension
    perf
    pointer-authentication
    silicon-errata
    sme
    sve
    tagged-address-abi
    tagged-pointers

    features

.. only::  subproject and html

   Indices
   =======

   * :ref:`genindex`


code:
#!/bin/sh

# Print out the KASAN_SHADOW_OFFSETS required to place the KASAN SHADOW
# start address at the top of the linear region

print_kasan_offset () {
	printf "%02d\t" $1
	printf "0x%08x00000000\n" $(( (0xffffffff & (-1 << ($1 - 1 - 32))) \
			- (1 << (64 - 32 - $2)) ))
}

echo KASAN_SHADOW_SCALE_SHIFT = 3
printf "VABITS\tKASAN_SHADOW_OFFSET\n"
print_kasan_offset 48 3
print_kasan_offset 47 3
print_kasan_offset 42 3
print_kasan_offset 39 3
print_kasan_offset 36 3
echo
echo KASAN_SHADOW_SCALE_SHIFT = 4
printf "VABITS\tKASAN_SHADOW_OFFSET\n"
print_kasan_offset 48 4
print_kasan_offset 47 4
print_kasan_offset 42 4
print_kasan_offset 39 4
print_kasan_offset 36 4


code:
===================
Legacy instructions
===================

The arm64 port of the Linux kernel provides infrastructure to support
emulation of instructions which have been deprecated, or obsoleted in
the architecture. The infrastructure code uses undefined instruction
hooks to support emulation. Where available it also allows turning on
the instruction execution in hardware.

The emulation mode can be controlled by writing to sysctl nodes
(/proc/sys/abi). The following explains the different execution
behaviours and the corresponding values of the sysctl nodes -

* Undef
    Value: 0

  Generates undefined instruction abort. Default for instructions that
  have been obsoleted in the architecture, e.g., SWP

* Emulate
    Value: 1

  Uses software emulation. To aid migration of software, in this mode
  usage of emulated instruction is traced as well as rate limited
  warnings are issued. This is the default for deprecated
  instructions, .e.g., CP15 barriers

* Hardware Execution
    Value: 2

  Although marked as deprecated, some implementations may support the
  enabling/disabling of hardware support for the execution of these
  instructions. Using hardware execution generally provides better
  performance, but at the loss of ability to gather runtime statistics
  about the use of the deprecated instructions.

The default mode depends on the status of the instruction in the
architecture. Deprecated instructions should default to emulation
while obsolete instructions must be undefined by default.

Note: Instruction emulation may not be possible in all cases. See
individual instruction notes for further information.

Supported legacy instructions
-----------------------------
* SWP{B}

:Node: /proc/sys/abi/swp
:Status: Obsolete
:Default: Undef (0)

* CP15 Barriers

:Node: /proc/sys/abi/cp15_barrier
:Status: Deprecated
:Default: Emulate (1)

* SETEND

:Node: /proc/sys/abi/setend
:Status: Deprecated
:Default: Emulate (1)*

  Note: All the cpus on the system must have mixed endian support at EL0
  for this feature to be enabled. If a new CPU - which doesn't support mixed
  endian - is hotplugged in after this feature has been enabled, there could
  be unexpected results in the application.


code:
===============================================
Memory Tagging Extension (MTE) in AArch64 Linux
===============================================

Authors: Vincenzo Frascino <vincenzo.frascino@arm.com>
         Catalin Marinas <catalin.marinas@arm.com>

Date: 2020-02-25

This document describes the provision of the Memory Tagging Extension
functionality in AArch64 Linux.

Introduction
============

ARMv8.5 based processors introduce the Memory Tagging Extension (MTE)
feature. MTE is built on top of the ARMv8.0 virtual address tagging TBI
(Top Byte Ignore) feature and allows software to access a 4-bit
allocation tag for each 16-byte granule in the physical address space.
Such memory range must be mapped with the Normal-Tagged memory
attribute. A logical tag is derived from bits 59-56 of the virtual
address used for the memory access. A CPU with MTE enabled will compare
the logical tag against the allocation tag and potentially raise an
exception on mismatch, subject to system registers configuration.

Userspace Support
=================

When ``CONFIG_ARM64_MTE`` is selected and Memory Tagging Extension is
supported by the hardware, the kernel advertises the feature to
userspace via ``HWCAP2_MTE``.

PROT_MTE
--------

To access the allocation tags, a user process must enable the Tagged
memory attribute on an address range using a new ``prot`` flag for
``mmap()`` and ``mprotect()``:

``PROT_MTE`` - Pages allow access to the MTE allocation tags.

The allocation tag is set to 0 when such pages are first mapped in the
user address space and preserved on copy-on-write. ``MAP_SHARED`` is
supported and the allocation tags can be shared between processes.

**Note**: ``PROT_MTE`` is only supported on ``MAP_ANONYMOUS`` and
RAM-based file mappings (``tmpfs``, ``memfd``). Passing it to other
types of mapping will result in ``-EINVAL`` returned by these system
calls.

**Note**: The ``PROT_MTE`` flag (and corresponding memory type) cannot
be cleared by ``mprotect()``.

**Note**: ``madvise()`` memory ranges with ``MADV_DONTNEED`` and
``MADV_FREE`` may have the allocation tags cleared (set to 0) at any
point after the system call.

Tag Check Faults
----------------

When ``PROT_MTE`` is enabled on an address range and a mismatch between
the logical and allocation tags occurs on access, there are three
configurable behaviours:

- *Ignore* - This is the default mode. The CPU (and kernel) ignores the
  tag check fault.

- *Synchronous* - The kernel raises a ``SIGSEGV`` synchronously, with
  ``.si_code = SEGV_MTESERR`` and ``.si_addr = <fault-address>``. The
  memory access is not performed. If ``SIGSEGV`` is ignored or blocked
  by the offending thread, the containing process is terminated with a
  ``coredump``.

- *Asynchronous* - The kernel raises a ``SIGSEGV``, in the offending
  thread, asynchronously following one or multiple tag check faults,
  with ``.si_code = SEGV_MTEAERR`` and ``.si_addr = 0`` (the faulting
  address is unknown).

- *Asymmetric* - Reads are handled as for synchronous mode while writes
  are handled as for asynchronous mode.

The user can select the above modes, per thread, using the
``prctl(PR_SET_TAGGED_ADDR_CTRL, flags, 0, 0, 0)`` system call where ``flags``
contains any number of the following values in the ``PR_MTE_TCF_MASK``
bit-field:

- ``PR_MTE_TCF_NONE``  - *Ignore* tag check faults
                         (ignored if combined with other options)
- ``PR_MTE_TCF_SYNC``  - *Synchronous* tag check fault mode
- ``PR_MTE_TCF_ASYNC`` - *Asynchronous* tag check fault mode

If no modes are specified, tag check faults are ignored. If a single
mode is specified, the program will run in that mode. If multiple
modes are specified, the mode is selected as described in the "Per-CPU
preferred tag checking modes" section below.

The current tag check fault configuration can be read using the
``prctl(PR_GET_TAGGED_ADDR_CTRL, 0, 0, 0, 0)`` system call. If
multiple modes were requested then all will be reported.

Tag checking can also be disabled for a user thread by setting the
``PSTATE.TCO`` bit with ``MSR TCO, #1``.

**Note**: Signal handlers are always invoked with ``PSTATE.TCO = 0``,
irrespective of the interrupted context. ``PSTATE.TCO`` is restored on
``sigreturn()``.

**Note**: There are no *match-all* logical tags available for user
applications.

**Note**: Kernel accesses to the user address space (e.g. ``read()``
system call) are not checked if the user thread tag checking mode is
``PR_MTE_TCF_NONE`` or ``PR_MTE_TCF_ASYNC``. If the tag checking mode is
``PR_MTE_TCF_SYNC``, the kernel makes a best effort to check its user
address accesses, however it cannot always guarantee it. Kernel accesses
to user addresses are always performed with an effective ``PSTATE.TCO``
value of zero, regardless of the user configuration.

Excluding Tags in the ``IRG``, ``ADDG`` and ``SUBG`` instructions
-----------------------------------------------------------------

The architecture allows excluding certain tags to be randomly generated
via the ``GCR_EL1.Exclude`` register bit-field. By default, Linux
excludes all tags other than 0. A user thread can enable specific tags
in the randomly generated set using the ``prctl(PR_SET_TAGGED_ADDR_CTRL,
flags, 0, 0, 0)`` system call where ``flags`` contains the tags bitmap
in the ``PR_MTE_TAG_MASK`` bit-field.

**Note**: The hardware uses an exclude mask but the ``prctl()``
interface provides an include mask. An include mask of ``0`` (exclusion
mask ``0xffff``) results in the CPU always generating tag ``0``.

Per-CPU preferred tag checking mode
-----------------------------------

On some CPUs the performance of MTE in stricter tag checking modes
is similar to that of less strict tag checking modes. This makes it
worthwhile to enable stricter checks on those CPUs when a less strict
checking mode is requested, in order to gain the error detection
benefits of the stricter checks without the performance downsides. To
support this scenario, a privileged user may configure a stricter
tag checking mode as the CPU's preferred tag checking mode.

The preferred tag checking mode for each CPU is controlled by
``/sys/devices/system/cpu/cpu<N>/mte_tcf_preferred``, to which a
privileged user may write the value ``async``, ``sync`` or ``asymm``.  The
default preferred mode for each CPU is ``async``.

To allow a program to potentially run in the CPU's preferred tag
checking mode, the user program may set multiple tag check fault mode
bits in the ``flags`` argument to the ``prctl(PR_SET_TAGGED_ADDR_CTRL,
flags, 0, 0, 0)`` system call. If both synchronous and asynchronous
modes are requested then asymmetric mode may also be selected by the
kernel. If the CPU's preferred tag checking mode is in the task's set
of provided tag checking modes, that mode will be selected. Otherwise,
one of the modes in the task's mode will be selected by the kernel
from the task's mode set using the preference order:

	1. Asynchronous
	2. Asymmetric
	3. Synchronous

Note that there is no way for userspace to request multiple modes and
also disable asymmetric mode.

Initial process state
---------------------

On ``execve()``, the new process has the following configuration:

- ``PR_TAGGED_ADDR_ENABLE`` set to 0 (disabled)
- No tag checking modes are selected (tag check faults ignored)
- ``PR_MTE_TAG_MASK`` set to 0 (all tags excluded)
- ``PSTATE.TCO`` set to 0
- ``PROT_MTE`` not set on any of the initial memory maps

On ``fork()``, the new process inherits the parent's configuration and
memory map attributes with the exception of the ``madvise()`` ranges
with ``MADV_WIPEONFORK`` which will have the data and tags cleared (set
to 0).

The ``ptrace()`` interface
--------------------------

``PTRACE_PEEKMTETAGS`` and ``PTRACE_POKEMTETAGS`` allow a tracer to read
the tags from or set the tags to a tracee's address space. The
``ptrace()`` system call is invoked as ``ptrace(request, pid, addr,
data)`` where:

- ``request`` - one of ``PTRACE_PEEKMTETAGS`` or ``PTRACE_POKEMTETAGS``.
- ``pid`` - the tracee's PID.
- ``addr`` - address in the tracee's address space.
- ``data`` - pointer to a ``struct iovec`` where ``iov_base`` points to
  a buffer of ``iov_len`` length in the tracer's address space.

The tags in the tracer's ``iov_base`` buffer are represented as one
4-bit tag per byte and correspond to a 16-byte MTE tag granule in the
tracee's address space.

**Note**: If ``addr`` is not aligned to a 16-byte granule, the kernel
will use the corresponding aligned address.

``ptrace()`` return value:

- 0 - tags were copied, the tracer's ``iov_len`` was updated to the
  number of tags transferred. This may be smaller than the requested
  ``iov_len`` if the requested address range in the tracee's or the
  tracer's space cannot be accessed or does not have valid tags.
- ``-EPERM`` - the specified process cannot be traced.
- ``-EIO`` - the tracee's address range cannot be accessed (e.g. invalid
  address) and no tags copied. ``iov_len`` not updated.
- ``-EFAULT`` - fault on accessing the tracer's memory (``struct iovec``
  or ``iov_base`` buffer) and no tags copied. ``iov_len`` not updated.
- ``-EOPNOTSUPP`` - the tracee's address does not have valid tags (never
  mapped with the ``PROT_MTE`` flag). ``iov_len`` not updated.

**Note**: There are no transient errors for the requests above, so user
programs should not retry in case of a non-zero system call return.

``PTRACE_GETREGSET`` and ``PTRACE_SETREGSET`` with ``addr ==
``NT_ARM_TAGGED_ADDR_CTRL`` allow ``ptrace()`` access to the tagged
address ABI control and MTE configuration of a process as per the
``prctl()`` options described in
Documentation/arm64/tagged-address-abi.rst and above. The corresponding
``regset`` is 1 element of 8 bytes (``sizeof(long))``).

Core dump support
-----------------

The allocation tags for user memory mapped with ``PROT_MTE`` are dumped
in the core file as additional ``PT_AARCH64_MEMTAG_MTE`` segments. The
program header for such segment is defined as:

:``p_type``: ``PT_AARCH64_MEMTAG_MTE``
:``p_flags``: 0
:``p_offset``: segment file offset
:``p_vaddr``: segment virtual address, same as the corresponding
  ``PT_LOAD`` segment
:``p_paddr``: 0
:``p_filesz``: segment size in file, calculated as ``p_mem_sz / 32``
  (two 4-bit tags cover 32 bytes of memory)
:``p_memsz``: segment size in memory, same as the corresponding
  ``PT_LOAD`` segment
:``p_align``: 0

The tags are stored in the core file at ``p_offset`` as two 4-bit tags
in a byte. With the tag granule of 16 bytes, a 4K page requires 128
bytes in the core file.

Example of correct usage
========================

*MTE Example code*

.. code-block:: c

    /*
     * To be compiled with -march=armv8.5-a+memtag
     */
    #include <errno.h>
    #include <stdint.h>
    #include <stdio.h>
    #include <stdlib.h>
    #include <unistd.h>
    #include <sys/auxv.h>
    #include <sys/mman.h>
    #include <sys/prctl.h>

    /*
     * From arch/arm64/include/uapi/asm/hwcap.h
     */
    #define HWCAP2_MTE              (1 << 18)

    /*
     * From arch/arm64/include/uapi/asm/mman.h
     */
    #define PROT_MTE                 0x20

    /*
     * From include/uapi/linux/prctl.h
     */
    #define PR_SET_TAGGED_ADDR_CTRL 55
    #define PR_GET_TAGGED_ADDR_CTRL 56
    # define PR_TAGGED_ADDR_ENABLE  (1UL << 0)
    # define PR_MTE_TCF_SHIFT       1
    # define PR_MTE_TCF_NONE        (0UL << PR_MTE_TCF_SHIFT)
    # define PR_MTE_TCF_SYNC        (1UL << PR_MTE_TCF_SHIFT)
    # define PR_MTE_TCF_ASYNC       (2UL << PR_MTE_TCF_SHIFT)
    # define PR_MTE_TCF_MASK        (3UL << PR_MTE_TCF_SHIFT)
    # define PR_MTE_TAG_SHIFT       3
    # define PR_MTE_TAG_MASK        (0xffffUL << PR_MTE_TAG_SHIFT)

    /*
     * Insert a random logical tag into the given pointer.
     */
    #define insert_random_tag(ptr) ({                       \
            uint64_t __val;                                 \
            asm("irg %0, %1" : "=r" (__val) : "r" (ptr));   \
            __val;                                          \
    })

    /*
     * Set the allocation tag on the destination address.
     */
    #define set_tag(tagged_addr) do {                                      \
            asm volatile("stg %0, [%0]" : : "r" (tagged_addr) : "memory"); \
    } while (0)

    int main()
    {
            unsigned char *a;
            unsigned long page_sz = sysconf(_SC_PAGESIZE);
            unsigned long hwcap2 = getauxval(AT_HWCAP2);

            /* check if MTE is present */
            if (!(hwcap2 & HWCAP2_MTE))
                    return EXIT_FAILURE;

            /*
             * Enable the tagged address ABI, synchronous or asynchronous MTE
             * tag check faults (based on per-CPU preference) and allow all
             * non-zero tags in the randomly generated set.
             */
            if (prctl(PR_SET_TAGGED_ADDR_CTRL,
                      PR_TAGGED_ADDR_ENABLE | PR_MTE_TCF_SYNC | PR_MTE_TCF_ASYNC |
                      (0xfffe << PR_MTE_TAG_SHIFT),
                      0, 0, 0)) {
                    perror("prctl() failed");
                    return EXIT_FAILURE;
            }

            a = mmap(0, page_sz, PROT_READ | PROT_WRITE,
                     MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
            if (a == MAP_FAILED) {
                    perror("mmap() failed");
                    return EXIT_FAILURE;
            }

            /*
             * Enable MTE on the above anonymous mmap. The flag could be passed
             * directly to mmap() and skip this step.
             */
            if (mprotect(a, page_sz, PROT_READ | PROT_WRITE | PROT_MTE)) {
                    perror("mprotect() failed");
                    return EXIT_FAILURE;
            }

            /* access with the default tag (0) */
            a[0] = 1;
            a[1] = 2;

            printf("a[0] = %hhu a[1] = %hhu\n", a[0], a[1]);

            /* set the logical and allocation tags */
            a = (unsigned char *)insert_random_tag(a);
            set_tag(a);

            printf("%p\n", a);

            /* non-zero tag access */
            a[0] = 3;
            printf("a[0] = %hhu a[1] = %hhu\n", a[0], a[1]);

            /*
             * If MTE is enabled correctly the next instruction will generate an
             * exception.
             */
            printf("Expecting SIGSEGV...\n");
            a[16] = 0xdd;

            /* this should not be printed in the PR_MTE_TCF_SYNC mode */
            printf("...haven't got one\n");

            return EXIT_FAILURE;
    }


code:
==============================
Memory Layout on AArch64 Linux
==============================

Author: Catalin Marinas <catalin.marinas@arm.com>

This document describes the virtual memory layout used by the AArch64
Linux kernel. The architecture allows up to 4 levels of translation
tables with a 4KB page size and up to 3 levels with a 64KB page size.

AArch64 Linux uses either 3 levels or 4 levels of translation tables
with the 4KB page configuration, allowing 39-bit (512GB) or 48-bit
(256TB) virtual addresses, respectively, for both user and kernel. With
64KB pages, only 2 levels of translation tables, allowing 42-bit (4TB)
virtual address, are used but the memory layout is the same.

ARMv8.2 adds optional support for Large Virtual Address space. This is
only available when running with a 64KB page size and expands the
number of descriptors in the first level of translation.

User addresses have bits 63:48 set to 0 while the kernel addresses have
the same bits set to 1. TTBRx selection is given by bit 63 of the
virtual address. The swapper_pg_dir contains only kernel (global)
mappings while the user pgd contains only user (non-global) mappings.
The swapper_pg_dir address is written to TTBR1 and never written to
TTBR0.

AArch64 Linux memory layout with 4KB pages + 4 levels (48-bit)::

  Start			End			Size		Use
  -----------------------------------------------------------------------
  0000000000000000	0000ffffffffffff	 256TB		user
  ffff000000000000	ffff7fffffffffff	 128TB		kernel logical memory map
 [ffff600000000000	ffff7fffffffffff]	  32TB		[kasan shadow region]
  ffff800000000000	ffff800007ffffff	 128MB		modules
  ffff800008000000	fffffbffefffffff	 124TB		vmalloc
  fffffbfff0000000	fffffbfffdffffff	 224MB		fixed mappings (top down)
  fffffbfffe000000	fffffbfffe7fffff	   8MB		[guard region]
  fffffbfffe800000	fffffbffff7fffff	  16MB		PCI I/O space
  fffffbffff800000	fffffbffffffffff	   8MB		[guard region]
  fffffc0000000000	fffffdffffffffff	   2TB		vmemmap
  fffffe0000000000	ffffffffffffffff	   2TB		[guard region]

AArch64 Linux memory layout with 64KB pages + 3 levels (52-bit with HW support)::

  Start			End			Size		Use
  -----------------------------------------------------------------------
  0000000000000000	000fffffffffffff	   4PB		user
  fff0000000000000	ffff7fffffffffff	  ~4PB		kernel logical memory map
 [fffd800000000000	ffff7fffffffffff]	 512TB		[kasan shadow region]
  ffff800000000000	ffff800007ffffff	 128MB		modules
  ffff800008000000	fffffbffefffffff	 124TB		vmalloc
  fffffbfff0000000	fffffbfffdffffff	 224MB		fixed mappings (top down)
  fffffbfffe000000	fffffbfffe7fffff	   8MB		[guard region]
  fffffbfffe800000	fffffbffff7fffff	  16MB		PCI I/O space
  fffffbffff800000	fffffbffffffffff	   8MB		[guard region]
  fffffc0000000000	ffffffdfffffffff	  ~4TB		vmemmap
  ffffffe000000000	ffffffffffffffff	 128GB		[guard region]

Translation table lookup with 4KB pages::

  +--------+--------+--------+--------+--------+--------+--------+--------+
  |63    56|55    48|47    40|39    32|31    24|23    16|15     8|7      0|
  +--------+--------+--------+--------+--------+--------+--------+--------+
   |                 |         |         |         |         |
   |                 |         |         |         |         v
   |                 |         |         |         |   [11:0]  in-page offset
   |                 |         |         |         +-> [20:12] L3 index
   |                 |         |         +-----------> [29:21] L2 index
   |                 |         +---------------------> [38:30] L1 index
   |                 +-------------------------------> [47:39] L0 index
   +-------------------------------------------------> [63] TTBR0/1

Translation table lookup with 64KB pages::

  +--------+--------+--------+--------+--------+--------+--------+--------+
  |63    56|55    48|47    40|39    32|31    24|23    16|15     8|7      0|
  +--------+--------+--------+--------+--------+--------+--------+--------+
   |                 |    |               |              |
   |                 |    |               |              v
   |                 |    |               |            [15:0]  in-page offset
   |                 |    |               +----------> [28:16] L3 index
   |                 |    +--------------------------> [41:29] L2 index
   |                 +-------------------------------> [47:42] L1 index (48-bit)
   |                                                   [51:42] L1 index (52-bit)
   +-------------------------------------------------> [63] TTBR0/1

When using KVM without the Virtualization Host Extensions, the
hypervisor maps kernel pages in EL2 at a fixed (and potentially
random) offset from the linear mapping. See the kern_hyp_va macro and
kvm_update_va_mask function for more details. MMIO devices such as
GICv2 gets mapped next to the HYP idmap page, as do vectors when
ARM64_SPECTRE_V3A is enabled for particular CPUs.

When using KVM with the Virtualization Host Extensions, no additional
mappings are created, since the host kernel runs directly in EL2.

52-bit VA support in the kernel
-------------------------------
If the ARMv8.2-LVA optional feature is present, and we are running
with a 64KB page size; then it is possible to use 52-bits of address
space for both userspace and kernel addresses. However, any kernel
binary that supports 52-bit must also be able to fall back to 48-bit
at early boot time if the hardware feature is not present.

This fallback mechanism necessitates the kernel .text to be in the
higher addresses such that they are invariant to 48/52-bit VAs. Due
to the kasan shadow being a fraction of the entire kernel VA space,
the end of the kasan shadow must also be in the higher half of the
kernel VA space for both 48/52-bit. (Switching from 48-bit to 52-bit,
the end of the kasan shadow is invariant and dependent on ~0UL,
whilst the start address will "grow" towards the lower addresses).

In order to optimise phys_to_virt and virt_to_phys, the PAGE_OFFSET
is kept constant at 0xFFF0000000000000 (corresponding to 52-bit),
this obviates the need for an extra variable read. The physvirt
offset and vmemmap offsets are computed at early boot to enable
this logic.

As a single binary will need to support both 48-bit and 52-bit VA
spaces, the VMEMMAP must be sized large enough for 52-bit VAs and
also must be sized large enough to accommodate a fixed PAGE_OFFSET.

Most code in the kernel should not need to consider the VA_BITS, for
code that does need to know the VA size the variables are
defined as follows:

VA_BITS		constant	the *maximum* VA space size

VA_BITS_MIN	constant	the *minimum* VA space size

vabits_actual	variable	the *actual* VA space size

Maximum and minimum sizes can be useful to ensure that buffers are
sized large enough or that addresses are positioned close enough for
the "worst" case.

52-bit userspace VAs
--------------------
To maintain compatibility with software that relies on the ARMv8.0
VA space maximum size of 48-bits, the kernel will, by default,
return virtual addresses to userspace from a 48-bit range.

Software can "opt-in" to receiving VAs from a 52-bit space by
specifying an mmap hint parameter that is larger than 48-bit.

For example:

.. code-block:: c

   maybe_high_address = mmap(~0UL, size, prot, flags,...);

It is also possible to build a debug kernel that returns addresses
from a 52-bit space by enabling the following kernel config options:

.. code-block:: sh

   CONFIG_EXPERT=y && CONFIG_ARM64_FORCE_52BIT=y

Note that this option is only intended for debugging applications
and should not be used in production.


code:
.. SPDX-License-Identifier: GPL-2.0

.. _perf_index:

====
Perf
====

Perf Event Attributes
=====================

:Author: Andrew Murray <andrew.murray@arm.com>
:Date: 2019-03-06

exclude_user
------------

This attribute excludes userspace.

Userspace always runs at EL0 and thus this attribute will exclude EL0.

exclude_kernel
--------------

This attribute excludes the kernel.

The kernel runs at EL2 with VHE and EL1 without. Guest kernels always run
at EL1.

For the host this attribute will exclude EL1 and additionally EL2 on a VHE
system.

For the guest this attribute will exclude EL1. Please note that EL2 is
never counted within a guest.

exclude_hv
----------

This attribute excludes the hypervisor.

For a VHE host this attribute is ignored as we consider the host kernel to
be the hypervisor.

For a non-VHE host this attribute will exclude EL2 as we consider the
hypervisor to be any code that runs at EL2 which is predominantly used for
guest/host transitions.

For the guest this attribute has no effect. Please note that EL2 is
never counted within a guest.

exclude_host / exclude_guest
----------------------------

These attributes exclude the KVM host and guest, respectively.

The KVM host may run at EL0 (userspace), EL1 (non-VHE kernel) and EL2 (VHE
kernel or non-VHE hypervisor).

The KVM guest may run at EL0 (userspace) and EL1 (kernel).

Due to the overlapping exception levels between host and guests we cannot
exclusively rely on the PMU's hardware exception filtering - therefore we
must enable/disable counting on the entry and exit to the guest. This is
performed differently on VHE and non-VHE systems.

For non-VHE systems we exclude EL2 for exclude_host - upon entering and
exiting the guest we disable/enable the event as appropriate based on the
exclude_host and exclude_guest attributes.

For VHE systems we exclude EL1 for exclude_guest and exclude both EL0,EL2
for exclude_host. Upon entering and exiting the guest we modify the event
to include/exclude EL0 as appropriate based on the exclude_host and
exclude_guest attributes.

The statements above also apply when these attributes are used within a
non-VHE guest however please note that EL2 is never counted within a guest.

Accuracy
--------

On non-VHE hosts we enable/disable counters on the entry/exit of host/guest
transition at EL2 - however there is a period of time between
enabling/disabling the counters and entering/exiting the guest. We are
able to eliminate counters counting host events on the boundaries of guest
entry/exit when counting guest events by filtering out EL2 for
exclude_host. However when using !exclude_hv there is a small blackout
window at the guest entry/exit where host events are not captured.

On VHE systems there are no blackout windows.

Perf Userspace PMU Hardware Counter Access
==========================================

Overview
--------
The perf userspace tool relies on the PMU to monitor events. It offers an
abstraction layer over the hardware counters since the underlying
implementation is cpu-dependent.
Arm64 allows userspace tools to have access to the registers storing the
hardware counters' values directly.

This targets specifically self-monitoring tasks in order to reduce the overhead
by directly accessing the registers without having to go through the kernel.

How-to
------
The focus is set on the armv8 PMUv3 which makes sure that the access to the pmu
registers is enabled and that the userspace has access to the relevant
information in order to use them.

In order to have access to the hardware counters, the global sysctl
kernel/perf_user_access must first be enabled:

.. code-block:: sh

  echo 1 > /proc/sys/kernel/perf_user_access

It is necessary to open the event using the perf tool interface with config1:1
attr bit set: the sys_perf_event_open syscall returns a fd which can
subsequently be used with the mmap syscall in order to retrieve a page of memory
containing information about the event. The PMU driver uses this page to expose
to the user the hardware counter's index and other necessary data. Using this
index enables the user to access the PMU registers using the `mrs` instruction.
Access to the PMU registers is only valid while the sequence lock is unchanged.
In particular, the PMSELR_EL0 register is zeroed each time the sequence lock is
changed.

The userspace access is supported in libperf using the perf_evsel__mmap()
and perf_evsel__read() functions. See `tools/lib/perf/tests/test-evsel.c`_ for
an example.

About heterogeneous systems
---------------------------
On heterogeneous systems such as big.LITTLE, userspace PMU counter access can
only be enabled when the tasks are pinned to a homogeneous subset of cores and
the corresponding PMU instance is opened by specifying the 'type' attribute.
The use of generic event types is not supported in this case.

Have a look at `tools/perf/arch/arm64/tests/user-events.c`_ for an example. It
can be run using the perf tool to check that the access to the registers works
correctly from userspace:

.. code-block:: sh

  perf test -v user

About chained events and counter sizes
--------------------------------------
The user can request either a 32-bit (config1:0 == 0) or 64-bit (config1:0 == 1)
counter along with userspace access. The sys_perf_event_open syscall will fail
if a 64-bit counter is requested and the hardware doesn't support 64-bit
counters. Chained events are not supported in conjunction with userspace counter
access. If a 32-bit counter is requested on hardware with 64-bit counters, then
userspace must treat the upper 32-bits read from the counter as UNKNOWN. The
'pmc_width' field in the user page will indicate the valid width of the counter
and should be used to mask the upper bits as needed.

.. Links
.. _tools/perf/arch/arm64/tests/user-events.c:
   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/perf/arch/arm64/tests/user-events.c
.. _tools/lib/perf/tests/test-evsel.c:
   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/lib/perf/tests/test-evsel.c


code:
=======================================
Pointer authentication in AArch64 Linux
=======================================

Author: Mark Rutland <mark.rutland@arm.com>

Date: 2017-07-19

This document briefly describes the provision of pointer authentication
functionality in AArch64 Linux.

Architecture overview
---------------------

The ARMv8.3 Pointer Authentication extension adds primitives that can be
used to mitigate certain classes of attack where an attacker can corrupt
the contents of some memory (e.g. the stack).

The extension uses a Pointer Authentication Code (PAC) to determine
whether pointers have been modified unexpectedly. A PAC is derived from
a pointer, another value (such as the stack pointer), and a secret key
held in system registers.

The extension adds instructions to insert a valid PAC into a pointer,
and to verify/remove the PAC from a pointer. The PAC occupies a number
of high-order bits of the pointer, which varies dependent on the
configured virtual address size and whether pointer tagging is in use.

A subset of these instructions have been allocated from the HINT
encoding space. In the absence of the extension (or when disabled),
these instructions behave as NOPs. Applications and libraries using
these instructions operate correctly regardless of the presence of the
extension.

The extension provides five separate keys to generate PACs - two for
instruction addresses (APIAKey, APIBKey), two for data addresses
(APDAKey, APDBKey), and one for generic authentication (APGAKey).

Basic support
-------------

When CONFIG_ARM64_PTR_AUTH is selected, and relevant HW support is
present, the kernel will assign random key values to each process at
exec*() time. The keys are shared by all threads within the process, and
are preserved across fork().

Presence of address authentication functionality is advertised via
HWCAP_PACA, and generic authentication functionality via HWCAP_PACG.

The number of bits that the PAC occupies in a pointer is 55 minus the
virtual address size configured by the kernel. For example, with a
virtual address size of 48, the PAC is 7 bits wide.

When ARM64_PTR_AUTH_KERNEL is selected, the kernel will be compiled
with HINT space pointer authentication instructions protecting
function returns. Kernels built with this option will work on hardware
with or without pointer authentication support.

In addition to exec(), keys can also be reinitialized to random values
using the PR_PAC_RESET_KEYS prctl. A bitmask of PR_PAC_APIAKEY,
PR_PAC_APIBKEY, PR_PAC_APDAKEY, PR_PAC_APDBKEY and PR_PAC_APGAKEY
specifies which keys are to be reinitialized; specifying 0 means "all
keys".

Debugging
---------

When CONFIG_ARM64_PTR_AUTH is selected, and HW support for address
authentication is present, the kernel will expose the position of TTBR0
PAC bits in the NT_ARM_PAC_MASK regset (struct user_pac_mask), which
userspace can acquire via PTRACE_GETREGSET.

The regset is exposed only when HWCAP_PACA is set. Separate masks are
exposed for data pointers and instruction pointers, as the set of PAC
bits can vary between the two. Note that the masks apply to TTBR0
addresses, and are not valid to apply to TTBR1 addresses (e.g. kernel
pointers).

Additionally, when CONFIG_CHECKPOINT_RESTORE is also set, the kernel
will expose the NT_ARM_PACA_KEYS and NT_ARM_PACG_KEYS regsets (struct
user_pac_address_keys and struct user_pac_generic_keys). These can be
used to get and set the keys for a thread.

Virtualization
--------------

Pointer authentication is enabled in KVM guest when each virtual cpu is
initialised by passing flags KVM_ARM_VCPU_PTRAUTH_[ADDRESS/GENERIC] and
requesting these two separate cpu features to be enabled. The current KVM
guest implementation works by enabling both features together, so both
these userspace flags are checked before enabling pointer authentication.
The separate userspace flag will allow to have no userspace ABI changes
if support is added in the future to allow these two features to be
enabled independently of one another.

As Arm Architecture specifies that Pointer Authentication feature is
implemented along with the VHE feature so KVM arm64 ptrauth code relies
on VHE mode to be present.

Additionally, when these vcpu feature flags are not set then KVM will
filter out the Pointer Authentication system key registers from
KVM_GET/SET_REG_* ioctls and mask those features from cpufeature ID
register. Any attempt to use the Pointer Authentication instructions will
result in an UNDEFINED exception being injected into the guest.

Enabling and disabling keys
---------------------------

The prctl PR_PAC_SET_ENABLED_KEYS allows the user program to control which
PAC keys are enabled in a particular task. It takes two arguments, the
first being a bitmask of PR_PAC_APIAKEY, PR_PAC_APIBKEY, PR_PAC_APDAKEY
and PR_PAC_APDBKEY specifying which keys shall be affected by this prctl,
and the second being a bitmask of the same bits specifying whether the key
should be enabled or disabled. For example::

  prctl(PR_PAC_SET_ENABLED_KEYS,
        PR_PAC_APIAKEY | PR_PAC_APIBKEY | PR_PAC_APDAKEY | PR_PAC_APDBKEY,
        PR_PAC_APIBKEY, 0, 0);

disables all keys except the IB key.

The main reason why this is useful is to enable a userspace ABI that uses PAC
instructions to sign and authenticate function pointers and other pointers
exposed outside of the function, while still allowing binaries conforming to
the ABI to interoperate with legacy binaries that do not sign or authenticate
pointers.

The idea is that a dynamic loader or early startup code would issue this
prctl very early after establishing that a process may load legacy binaries,
but before executing any PAC instructions.

For compatibility with previous kernel versions, processes start up with IA,
IB, DA and DB enabled, and are reset to this state on exec(). Processes created
via fork() and clone() inherit the key enabled state from the calling process.

It is recommended to avoid disabling the IA key, as this has higher performance
overhead than disabling any of the other keys.


code:
=======================================
Silicon Errata and Software Workarounds
=======================================

Author: Will Deacon <will.deacon@arm.com>

Date  : 27 November 2015

It is an unfortunate fact of life that hardware is often produced with
so-called "errata", which can cause it to deviate from the architecture
under specific circumstances.  For hardware produced by ARM, these
errata are broadly classified into the following categories:

  ==========  ========================================================
  Category A  A critical error without a viable workaround.
  Category B  A significant or critical error with an acceptable
              workaround.
  Category C  A minor error that is not expected to occur under normal
              operation.
  ==========  ========================================================

For more information, consult one of the "Software Developers Errata
Notice" documents available on infocenter.arm.com (registration
required).

As far as Linux is concerned, Category B errata may require some special
treatment in the operating system. For example, avoiding a particular
sequence of code, or configuring the processor in a particular way. A
less common situation may require similar actions in order to declassify
a Category A erratum into a Category C erratum. These are collectively
known as "software workarounds" and are only required in the minority of
cases (e.g. those cases that both require a non-secure workaround *and*
can be triggered by Linux).

For software workarounds that may adversely impact systems unaffected by
the erratum in question, a Kconfig entry is added under "Kernel
Features" -> "ARM errata workarounds via the alternatives framework".
These are enabled by default and patched in at runtime when an affected
CPU is detected. For less-intrusive workarounds, a Kconfig option is not
available and the code is structured (preferably with a comment) in such
a way that the erratum will not be hit.

This approach can make it slightly onerous to determine exactly which
errata are worked around in an arbitrary kernel source tree, so this
file acts as a registry of software workarounds in the Linux Kernel and
will be updated when new workarounds are committed and backported to
stable kernels.

+----------------+-----------------+-----------------+-----------------------------+
| Implementor    | Component       | Erratum ID      | Kconfig                     |
+================+=================+=================+=============================+
| Allwinner      | A64/R18         | UNKNOWN1        | SUN50I_ERRATUM_UNKNOWN1     |
+----------------+-----------------+-----------------+-----------------------------+
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-A510     | #2457168        | ARM64_ERRATUM_2457168       |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-A510     | #2064142        | ARM64_ERRATUM_2064142       |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-A510     | #2038923        | ARM64_ERRATUM_2038923       |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-A510     | #1902691        | ARM64_ERRATUM_1902691       |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-A53      | #826319         | ARM64_ERRATUM_826319        |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-A53      | #827319         | ARM64_ERRATUM_827319        |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-A53      | #824069         | ARM64_ERRATUM_824069        |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-A53      | #819472         | ARM64_ERRATUM_819472        |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-A53      | #845719         | ARM64_ERRATUM_845719        |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-A53      | #843419         | ARM64_ERRATUM_843419        |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-A55      | #1024718        | ARM64_ERRATUM_1024718       |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-A55      | #1530923        | ARM64_ERRATUM_1530923       |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-A55      | #2441007        | ARM64_ERRATUM_2441007       |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-A57      | #832075         | ARM64_ERRATUM_832075        |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-A57      | #852523         | N/A                         |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-A57      | #834220         | ARM64_ERRATUM_834220        |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-A57      | #1319537        | ARM64_ERRATUM_1319367       |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-A57      | #1742098        | ARM64_ERRATUM_1742098       |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-A72      | #853709         | N/A                         |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-A72      | #1319367        | ARM64_ERRATUM_1319367       |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-A72      | #1655431        | ARM64_ERRATUM_1742098       |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-A73      | #858921         | ARM64_ERRATUM_858921        |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-A76      | #1188873,1418040| ARM64_ERRATUM_1418040       |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-A76      | #1165522        | ARM64_ERRATUM_1165522       |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-A76      | #1286807        | ARM64_ERRATUM_1286807       |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-A76      | #1463225        | ARM64_ERRATUM_1463225       |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-A77      | #1508412        | ARM64_ERRATUM_1508412       |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-A510     | #2051678        | ARM64_ERRATUM_2051678       |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-A510     | #2077057        | ARM64_ERRATUM_2077057       |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-A510     | #2441009        | ARM64_ERRATUM_2441009       |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-A510     | #2658417        | ARM64_ERRATUM_2658417       |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-A710     | #2119858        | ARM64_ERRATUM_2119858       |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-A710     | #2054223        | ARM64_ERRATUM_2054223       |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-A710     | #2224489        | ARM64_ERRATUM_2224489       |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-X2       | #2119858        | ARM64_ERRATUM_2119858       |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Cortex-X2       | #2224489        | ARM64_ERRATUM_2224489       |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Neoverse-N1     | #1188873,1418040| ARM64_ERRATUM_1418040       |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Neoverse-N1     | #1349291        | N/A                         |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Neoverse-N1     | #1542419        | ARM64_ERRATUM_1542419       |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Neoverse-N2     | #2139208        | ARM64_ERRATUM_2139208       |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Neoverse-N2     | #2067961        | ARM64_ERRATUM_2067961       |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | Neoverse-N2     | #2253138        | ARM64_ERRATUM_2253138       |
+----------------+-----------------+-----------------+-----------------------------+
| ARM            | MMU-500         | #841119,826419  | N/A                         |
+----------------+-----------------+-----------------+-----------------------------+
+----------------+-----------------+-----------------+-----------------------------+
| Broadcom       | Brahma-B53      | N/A             | ARM64_ERRATUM_845719        |
+----------------+-----------------+-----------------+-----------------------------+
| Broadcom       | Brahma-B53      | N/A             | ARM64_ERRATUM_843419        |
+----------------+-----------------+-----------------+-----------------------------+
+----------------+-----------------+-----------------+-----------------------------+
| Cavium         | ThunderX ITS    | #22375,24313    | CAVIUM_ERRATUM_22375        |
+----------------+-----------------+-----------------+-----------------------------+
| Cavium         | ThunderX ITS    | #23144          | CAVIUM_ERRATUM_23144        |
+----------------+-----------------+-----------------+-----------------------------+
| Cavium         | ThunderX GICv3  | #23154,38545    | CAVIUM_ERRATUM_23154        |
+----------------+-----------------+-----------------+-----------------------------+
| Cavium         | ThunderX GICv3  | #38539          | N/A                         |
+----------------+-----------------+-----------------+-----------------------------+
| Cavium         | ThunderX Core   | #27456          | CAVIUM_ERRATUM_27456        |
+----------------+-----------------+-----------------+-----------------------------+
| Cavium         | ThunderX Core   | #30115          | CAVIUM_ERRATUM_30115        |
+----------------+-----------------+-----------------+-----------------------------+
| Cavium         | ThunderX SMMUv2 | #27704          | N/A                         |
+----------------+-----------------+-----------------+-----------------------------+
| Cavium         | ThunderX2 SMMUv3| #74             | N/A                         |
+----------------+-----------------+-----------------+-----------------------------+
| Cavium         | ThunderX2 SMMUv3| #126            | N/A                         |
+----------------+-----------------+-----------------+-----------------------------+
| Cavium         | ThunderX2 Core  | #219            | CAVIUM_TX2_ERRATUM_219      |
+----------------+-----------------+-----------------+-----------------------------+
+----------------+-----------------+-----------------+-----------------------------+
| Marvell        | ARM-MMU-500     | #582743         | N/A                         |
+----------------+-----------------+-----------------+-----------------------------+
+----------------+-----------------+-----------------+-----------------------------+
| NVIDIA         | Carmel Core     | N/A             | NVIDIA_CARMEL_CNP_ERRATUM   |
+----------------+-----------------+-----------------+-----------------------------+
+----------------+-----------------+-----------------+-----------------------------+
| Freescale/NXP  | LS2080A/LS1043A | A-008585        | FSL_ERRATUM_A008585         |
+----------------+-----------------+-----------------+-----------------------------+
+----------------+-----------------+-----------------+-----------------------------+
| Hisilicon      | Hip0{5,6,7}     | #161010101      | HISILICON_ERRATUM_161010101 |
+----------------+-----------------+-----------------+-----------------------------+
| Hisilicon      | Hip0{6,7}       | #161010701      | N/A                         |
+----------------+-----------------+-----------------+-----------------------------+
| Hisilicon      | Hip0{6,7}       | #161010803      | N/A                         |
+----------------+-----------------+-----------------+-----------------------------+
| Hisilicon      | Hip07           | #161600802      | HISILICON_ERRATUM_161600802 |
+----------------+-----------------+-----------------+-----------------------------+
| Hisilicon      | Hip08 SMMU PMCG | #162001800      | N/A                         |
+----------------+-----------------+-----------------+-----------------------------+
+----------------+-----------------+-----------------+-----------------------------+
| Qualcomm Tech. | Kryo/Falkor v1  | E1003           | QCOM_FALKOR_ERRATUM_1003    |
+----------------+-----------------+-----------------+-----------------------------+
| Qualcomm Tech. | Kryo/Falkor v1  | E1009           | QCOM_FALKOR_ERRATUM_1009    |
+----------------+-----------------+-----------------+-----------------------------+
| Qualcomm Tech. | QDF2400 ITS     | E0065           | QCOM_QDF2400_ERRATUM_0065   |
+----------------+-----------------+-----------------+-----------------------------+
| Qualcomm Tech. | Falkor v{1,2}   | E1041           | QCOM_FALKOR_ERRATUM_1041    |
+----------------+-----------------+-----------------+-----------------------------+
| Qualcomm Tech. | Kryo4xx Gold    | N/A             | ARM64_ERRATUM_1463225       |
+----------------+-----------------+-----------------+-----------------------------+
| Qualcomm Tech. | Kryo4xx Gold    | N/A             | ARM64_ERRATUM_1418040       |
+----------------+-----------------+-----------------+-----------------------------+
| Qualcomm Tech. | Kryo4xx Silver  | N/A             | ARM64_ERRATUM_1530923       |
+----------------+-----------------+-----------------+-----------------------------+
| Qualcomm Tech. | Kryo4xx Silver  | N/A             | ARM64_ERRATUM_1024718       |
+----------------+-----------------+-----------------+-----------------------------+
| Qualcomm Tech. | Kryo4xx Gold    | N/A             | ARM64_ERRATUM_1286807       |
+----------------+-----------------+-----------------+-----------------------------+

+----------------+-----------------+-----------------+-----------------------------+
| Fujitsu        | A64FX           | E#010001        | FUJITSU_ERRATUM_010001      |
+----------------+-----------------+-----------------+-----------------------------+


code:
===================================================
Scalable Matrix Extension support for AArch64 Linux
===================================================

This document outlines briefly the interface provided to userspace by Linux in
order to support use of the ARM Scalable Matrix Extension (SME).

This is an outline of the most important features and issues only and not
intended to be exhaustive.  It should be read in conjunction with the SVE
documentation in sve.rst which provides details on the Streaming SVE mode
included in SME.

This document does not aim to describe the SME architecture or programmer's
model.  To aid understanding, a minimal description of relevant programmer's
model features for SME is included in Appendix A.

1.  General
-----------

* PSTATE.SM, PSTATE.ZA, the streaming mode vector length, the ZA
  register state and TPIDR2_EL0 are tracked per thread.

* The presence of SME is reported to userspace via HWCAP2_SME in the aux vector
  AT_HWCAP2 entry.  Presence of this flag implies the presence of the SME
  instructions and registers, and the Linux-specific system interfaces
  described in this document.  SME is reported in /proc/cpuinfo as "sme".

* Support for the execution of SME instructions in userspace can also be
  detected by reading the CPU ID register ID_AA64PFR1_EL1 using an MRS
  instruction, and checking that the value of the SME field is nonzero. [3]

  It does not guarantee the presence of the system interfaces described in the
  following sections: software that needs to verify that those interfaces are
  present must check for HWCAP2_SME instead.

* There are a number of optional SME features, presence of these is reported
  through AT_HWCAP2 through:

	HWCAP2_SME_I16I64
	HWCAP2_SME_F64F64
	HWCAP2_SME_I8I32
	HWCAP2_SME_F16F32
	HWCAP2_SME_B16F32
	HWCAP2_SME_F32F32
	HWCAP2_SME_FA64

  This list may be extended over time as the SME architecture evolves.

  These extensions are also reported via the CPU ID register ID_AA64SMFR0_EL1,
  which userspace can read using an MRS instruction.  See elf_hwcaps.txt and
  cpu-feature-registers.txt for details.

* Debuggers should restrict themselves to interacting with the target via the
  NT_ARM_SVE, NT_ARM_SSVE and NT_ARM_ZA regsets.  The recommended way
  of detecting support for these regsets is to connect to a target process
  first and then attempt a

	ptrace(PTRACE_GETREGSET, pid, NT_ARM_<regset>, &iov).

* Whenever ZA register values are exchanged in memory between userspace and
  the kernel, the register value is encoded in memory as a series of horizontal
  vectors from 0 to VL/8-1 stored in the same endianness invariant format as is
  used for SVE vectors.

* On thread creation TPIDR2_EL0 is preserved unless CLONE_SETTLS is specified,
  in which case it is set to 0.

2.  Vector lengths
------------------

SME defines a second vector length similar to the SVE vector length which is
controls the size of the streaming mode SVE vectors and the ZA matrix array.
The ZA matrix is square with each side having as many bytes as a streaming
mode SVE vector.

3.  Sharing of streaming and non-streaming mode SVE state
---------------------------------------------------------

It is implementation defined which if any parts of the SVE state are shared
between streaming and non-streaming modes.  When switching between modes
via software interfaces such as ptrace if no register content is provided as
part of switching no state will be assumed to be shared and everything will
be zeroed.

4.  System call behaviour
-------------------------

* On syscall PSTATE.ZA is preserved, if PSTATE.ZA==1 then the contents of the
  ZA matrix are preserved.

* On syscall PSTATE.SM will be cleared and the SVE registers will be handled
  as per the standard SVE ABI.

* Neither the SVE registers nor ZA are used to pass arguments to or receive
  results from any syscall.

* On process creation (eg, clone()) the newly created process will have
  PSTATE.SM cleared.

* All other SME state of a thread, including the currently configured vector
  length, the state of the PR_SME_VL_INHERIT flag, and the deferred vector
  length (if any), is preserved across all syscalls, subject to the specific
  exceptions for execve() described in section 6.

5.  Signal handling
-------------------

* Signal handlers are invoked with streaming mode and ZA disabled.

* A new signal frame record za_context encodes the ZA register contents on
  signal delivery. [1]

* The signal frame record for ZA always contains basic metadata, in particular
  the thread's vector length (in za_context.vl).

* The ZA matrix may or may not be included in the record, depending on
  the value of PSTATE.ZA.  The registers are present if and only if:
  za_context.head.size >= ZA_SIG_CONTEXT_SIZE(sve_vq_from_vl(za_context.vl))
  in which case PSTATE.ZA == 1.

* If matrix data is present, the remainder of the record has a vl-dependent
  size and layout.  Macros ZA_SIG_* are defined [1] to facilitate access to
  them.

* The matrix is stored as a series of horizontal vectors in the same format as
  is used for SVE vectors.

* If the ZA context is too big to fit in sigcontext.__reserved[], then extra
  space is allocated on the stack, an extra_context record is written in
  __reserved[] referencing this space.  za_context is then written in the
  extra space.  Refer to [1] for further details about this mechanism.

5.  Signal return
-----------------

When returning from a signal handler:

* If there is no za_context record in the signal frame, or if the record is
  present but contains no register data as described in the previous section,
  then ZA is disabled.

* If za_context is present in the signal frame and contains matrix data then
  PSTATE.ZA is set to 1 and ZA is populated with the specified data.

* The vector length cannot be changed via signal return.  If za_context.vl in
  the signal frame does not match the current vector length, the signal return
  attempt is treated as illegal, resulting in a forced SIGSEGV.

6.  prctl extensions
--------------------

Some new prctl() calls are added to allow programs to manage the SME vector
length:

prctl(PR_SME_SET_VL, unsigned long arg)

    Sets the vector length of the calling thread and related flags, where
    arg == vl | flags.  Other threads of the calling process are unaffected.

    vl is the desired vector length, where sve_vl_valid(vl) must be true.

    flags:

	PR_SME_VL_INHERIT

	    Inherit the current vector length across execve().  Otherwise, the
	    vector length is reset to the system default at execve().  (See
	    Section 9.)

	PR_SME_SET_VL_ONEXEC

	    Defer the requested vector length change until the next execve()
	    performed by this thread.

	    The effect is equivalent to implicit execution of the following
	    call immediately after the next execve() (if any) by the thread:

		prctl(PR_SME_SET_VL, arg & ~PR_SME_SET_VL_ONEXEC)

	    This allows launching of a new program with a different vector
	    length, while avoiding runtime side effects in the caller.

	    Without PR_SME_SET_VL_ONEXEC, the requested change takes effect
	    immediately.

    Return value: a nonnegative on success, or a negative value on error:
	EINVAL: SME not supported, invalid vector length requested, or
	    invalid flags.

    On success:

    * Either the calling thread's vector length or the deferred vector length
      to be applied at the next execve() by the thread (dependent on whether
      PR_SME_SET_VL_ONEXEC is present in arg), is set to the largest value
      supported by the system that is less than or equal to vl.  If vl ==
      SVE_VL_MAX, the value set will be the largest value supported by the
      system.

    * Any previously outstanding deferred vector length change in the calling
      thread is cancelled.

    * The returned value describes the resulting configuration, encoded as for
      PR_SME_GET_VL.  The vector length reported in this value is the new
      current vector length for this thread if PR_SME_SET_VL_ONEXEC was not
      present in arg; otherwise, the reported vector length is the deferred
      vector length that will be applied at the next execve() by the calling
      thread.

    * Changing the vector length causes all of ZA, P0..P15, FFR and all bits of
      Z0..Z31 except for Z0 bits [127:0] .. Z31 bits [127:0] to become
      unspecified, including both streaming and non-streaming SVE state.
      Calling PR_SME_SET_VL with vl equal to the thread's current vector
      length, or calling PR_SME_SET_VL with the PR_SVE_SET_VL_ONEXEC flag,
      does not constitute a change to the vector length for this purpose.

    * Changing the vector length causes PSTATE.ZA and PSTATE.SM to be cleared.
      Calling PR_SME_SET_VL with vl equal to the thread's current vector
      length, or calling PR_SME_SET_VL with the PR_SVE_SET_VL_ONEXEC flag,
      does not constitute a change to the vector length for this purpose.

prctl(PR_SME_GET_VL)

    Gets the vector length of the calling thread.

    The following flag may be OR-ed into the result:

	PR_SME_VL_INHERIT

	    Vector length will be inherited across execve().

    There is no way to determine whether there is an outstanding deferred
    vector length change (which would only normally be the case between a
    fork() or vfork() and the corresponding execve() in typical use).

    To extract the vector length from the result, bitwise and it with
    PR_SME_VL_LEN_MASK.

    Return value: a nonnegative value on success, or a negative value on error:
	EINVAL: SME not supported.

7.  ptrace extensions
---------------------

* A new regset NT_ARM_SSVE is defined for access to streaming mode SVE
  state via PTRACE_GETREGSET and  PTRACE_SETREGSET, this is documented in
  sve.rst.

* A new regset NT_ARM_ZA is defined for ZA state for access to ZA state via
  PTRACE_GETREGSET and PTRACE_SETREGSET.

  Refer to [2] for definitions.

The regset data starts with struct user_za_header, containing:

    size

	Size of the complete regset, in bytes.
	This depends on vl and possibly on other things in the future.

	If a call to PTRACE_GETREGSET requests less data than the value of
	size, the caller can allocate a larger buffer and retry in order to
	read the complete regset.

    max_size

	Maximum size in bytes that the regset can grow to for the target
	thread.  The regset won't grow bigger than this even if the target
	thread changes its vector length etc.

    vl

	Target thread's current streaming vector length, in bytes.

    max_vl

	Maximum possible streaming vector length for the target thread.

    flags

	Zero or more of the following flags, which have the same
	meaning and behaviour as the corresponding PR_SET_VL_* flags:

	    SME_PT_VL_INHERIT

	    SME_PT_VL_ONEXEC (SETREGSET only).

* The effects of changing the vector length and/or flags are equivalent to
  those documented for PR_SME_SET_VL.

  The caller must make a further GETREGSET call if it needs to know what VL is
  actually set by SETREGSET, unless is it known in advance that the requested
  VL is supported.

* The size and layout of the payload depends on the header fields.  The
  SME_PT_ZA_*() macros are provided to facilitate access to the data.

* In either case, for SETREGSET it is permissible to omit the payload, in which
  case the vector length and flags are changed and PSTATE.ZA is set to 0
  (along with any consequences of those changes).  If a payload is provided
  then PSTATE.ZA will be set to 1.

* For SETREGSET, if the requested VL is not supported, the effect will be the
  same as if the payload were omitted, except that an EIO error is reported.
  No attempt is made to translate the payload data to the correct layout
  for the vector length actually set.  It is up to the caller to translate the
  payload layout for the actual VL and retry.

* The effect of writing a partial, incomplete payload is unspecified.

8.  ELF coredump extensions
---------------------------

* NT_ARM_SSVE notes will be added to each coredump for
  each thread of the dumped process.  The contents will be equivalent to the
  data that would have been read if a PTRACE_GETREGSET of the corresponding
  type were executed for each thread when the coredump was generated.

* A NT_ARM_ZA note will be added to each coredump for each thread of the
  dumped process.  The contents will be equivalent to the data that would have
  been read if a PTRACE_GETREGSET of NT_ARM_ZA were executed for each thread
  when the coredump was generated.

* The NT_ARM_TLS note will be extended to two registers, the second register
  will contain TPIDR2_EL0 on systems that support SME and will be read as
  zero with writes ignored otherwise.

9.  System runtime configuration
--------------------------------

* To mitigate the ABI impact of expansion of the signal frame, a policy
  mechanism is provided for administrators, distro maintainers and developers
  to set the default vector length for userspace processes:

/proc/sys/abi/sme_default_vector_length

    Writing the text representation of an integer to this file sets the system
    default vector length to the specified value, unless the value is greater
    than the maximum vector length supported by the system in which case the
    default vector length is set to that maximum.

    The result can be determined by reopening the file and reading its
    contents.

    At boot, the default vector length is initially set to 32 or the maximum
    supported vector length, whichever is smaller and supported.  This
    determines the initial vector length of the init process (PID 1).

    Reading this file returns the current system default vector length.

* At every execve() call, the new vector length of the new process is set to
  the system default vector length, unless

    * PR_SME_VL_INHERIT (or equivalently SME_PT_VL_INHERIT) is set for the
      calling thread, or

    * a deferred vector length change is pending, established via the
      PR_SME_SET_VL_ONEXEC flag (or SME_PT_VL_ONEXEC).

* Modifying the system default vector length does not affect the vector length
  of any existing process or thread that does not make an execve() call.

Appendix A.  SME programmer's model (informative)
=================================================

This section provides a minimal description of the additions made by SME to the
ARMv8-A programmer's model that are relevant to this document.

Note: This section is for information only and not intended to be complete or
to replace any architectural specification.

A.1.  Registers
---------------

In A64 state, SME adds the following:

* A new mode, streaming mode, in which a subset of the normal FPSIMD and SVE
  features are available.  When supported EL0 software may enter and leave
  streaming mode at any time.

  For best system performance it is strongly encouraged for software to enable
  streaming mode only when it is actively being used.

* A new vector length controlling the size of ZA and the Z registers when in
  streaming mode, separately to the vector length used for SVE when not in
  streaming mode.  There is no requirement that either the currently selected
  vector length or the set of vector lengths supported for the two modes in
  a given system have any relationship.  The streaming mode vector length
  is referred to as SVL.

* A new ZA matrix register.  This is a square matrix of SVLxSVL bits.  Most
  operations on ZA require that streaming mode be enabled but ZA can be
  enabled without streaming mode in order to load, save and retain data.

  For best system performance it is strongly encouraged for software to enable
  ZA only when it is actively being used.

* Two new 1 bit fields in PSTATE which may be controlled via the SMSTART and
  SMSTOP instructions or by access to the SVCR system register:

  * PSTATE.ZA, if this is 1 then the ZA matrix is accessible and has valid
    data while if it is 0 then ZA can not be accessed.  When PSTATE.ZA is
    changed from 0 to 1 all bits in ZA are cleared.

  * PSTATE.SM, if this is 1 then the PE is in streaming mode.  When the value
    of PSTATE.SM is changed then it is implementation defined if the subset
    of the floating point register bits valid in both modes may be retained.
    Any other bits will be cleared.

References
==========

[1] arch/arm64/include/uapi/asm/sigcontext.h
    AArch64 Linux signal ABI definitions

[2] arch/arm64/include/uapi/asm/ptrace.h
    AArch64 Linux ptrace ABI definitions

[3] Documentation/arm64/cpu-feature-registers.rst