code:
.. SPDX-License-Identifier: GPL-2.0

TSX Async Abort (TAA) mitigation
================================

.. _tsx_async_abort:

Overview
--------

TSX Async Abort (TAA) is a side channel attack on internal buffers in some
Intel processors similar to Microachitectural Data Sampling (MDS).  In this
case certain loads may speculatively pass invalid data to dependent operations
when an asynchronous abort condition is pending in a Transactional
Synchronization Extensions (TSX) transaction.  This includes loads with no
fault or assist condition. Such loads may speculatively expose stale data from
the same uarch data structures as in MDS, with same scope of exposure i.e.
same-thread and cross-thread. This issue affects all current processors that
support TSX.

Mitigation strategy
-------------------

a) TSX disable - one of the mitigations is to disable TSX. A new MSR
IA32_TSX_CTRL will be available in future and current processors after
microcode update which can be used to disable TSX. In addition, it
controls the enumeration of the TSX feature bits (RTM and HLE) in CPUID.

b) Clear CPU buffers - similar to MDS, clearing the CPU buffers mitigates this
vulnerability. More details on this approach can be found in
:ref:`Documentation/admin-guide/hw-vuln/mds.rst <mds>`.

Kernel internal mitigation modes
--------------------------------

 =============    ============================================================
 off              Mitigation is disabled. Either the CPU is not affected or
                  tsx_async_abort=off is supplied on the kernel command line.

 tsx disabled     Mitigation is enabled. TSX feature is disabled by default at
                  bootup on processors that support TSX control.

 verw             Mitigation is enabled. CPU is affected and MD_CLEAR is
                  advertised in CPUID.

 ucode needed     Mitigation is enabled. CPU is affected and MD_CLEAR is not
                  advertised in CPUID. That is mainly for virtualization
                  scenarios where the host has the updated microcode but the
                  hypervisor does not expose MD_CLEAR in CPUID. It's a best
                  effort approach without guarantee.
 =============    ============================================================

If the CPU is affected and the "tsx_async_abort" kernel command line parameter is
not provided then the kernel selects an appropriate mitigation depending on the
status of RTM and MD_CLEAR CPUID bits.

Below tables indicate the impact of tsx=on|off|auto cmdline options on state of
TAA mitigation, VERW behavior and TSX feature for various combinations of
MSR_IA32_ARCH_CAPABILITIES bits.

1. "tsx=off"

=========  =========  ============  ============  ==============  ===================  ======================
MSR_IA32_ARCH_CAPABILITIES bits     Result with cmdline tsx=off
----------------------------------  -------------------------------------------------------------------------
TAA_NO     MDS_NO     TSX_CTRL_MSR  TSX state     VERW can clear  TAA mitigation       TAA mitigation
                                    after bootup  CPU buffers     tsx_async_abort=off  tsx_async_abort=full
=========  =========  ============  ============  ==============  ===================  ======================
    0          0           0         HW default         Yes           Same as MDS           Same as MDS
    0          0           1        Invalid case   Invalid case       Invalid case          Invalid case
    0          1           0         HW default         No         Need ucode update     Need ucode update
    0          1           1          Disabled          Yes           TSX disabled          TSX disabled
    1          X           1          Disabled           X             None needed           None needed
=========  =========  ============  ============  ==============  ===================  ======================

2. "tsx=on"

=========  =========  ============  ============  ==============  ===================  ======================
MSR_IA32_ARCH_CAPABILITIES bits     Result with cmdline tsx=on
----------------------------------  -------------------------------------------------------------------------
TAA_NO     MDS_NO     TSX_CTRL_MSR  TSX state     VERW can clear  TAA mitigation       TAA mitigation
                                    after bootup  CPU buffers     tsx_async_abort=off  tsx_async_abort=full
=========  =========  ============  ============  ==============  ===================  ======================
    0          0           0         HW default        Yes            Same as MDS          Same as MDS
    0          0           1        Invalid case   Invalid case       Invalid case         Invalid case
    0          1           0         HW default        No          Need ucode update     Need ucode update
    0          1           1          Enabled          Yes               None              Same as MDS
    1          X           1          Enabled          X              None needed          None needed
=========  =========  ============  ============  ==============  ===================  ======================

3. "tsx=auto"

=========  =========  ============  ============  ==============  ===================  ======================
MSR_IA32_ARCH_CAPABILITIES bits     Result with cmdline tsx=auto
----------------------------------  -------------------------------------------------------------------------
TAA_NO     MDS_NO     TSX_CTRL_MSR  TSX state     VERW can clear  TAA mitigation       TAA mitigation
                                    after bootup  CPU buffers     tsx_async_abort=off  tsx_async_abort=full
=========  =========  ============  ============  ==============  ===================  ======================
    0          0           0         HW default    Yes                Same as MDS           Same as MDS
    0          0           1        Invalid case  Invalid case        Invalid case          Invalid case
    0          1           0         HW default    No              Need ucode update     Need ucode update
    0          1           1          Disabled      Yes               TSX disabled          TSX disabled
    1          X           1          Enabled       X                 None needed           None needed
=========  =========  ============  ============  ==============  ===================  ======================

In the tables, TSX_CTRL_MSR is a new bit in MSR_IA32_ARCH_CAPABILITIES that
indicates whether MSR_IA32_TSX_CTRL is supported.

There are two control bits in IA32_TSX_CTRL MSR:

      Bit 0: When set it disables the Restricted Transactional Memory (RTM)
             sub-feature of TSX (will force all transactions to abort on the
             XBEGIN instruction).

      Bit 1: When set it disables the enumeration of the RTM and HLE feature
             (i.e. it will make CPUID(EAX=7).EBX{bit4} and
             CPUID(EAX=7).EBX{bit11} read as 0).


code:
.. SPDX-License-Identifier: GPL-2.0

==================
USB Legacy support
==================

:Author: Vojtech Pavlik <vojtech@suse.cz>, January 2004

Also known as "USB Keyboard" or "USB Mouse support" in the BIOS Setup is a
feature that allows one to use the USB mouse and keyboard as if they were
their classic PS/2 counterparts.  This means one can use an USB keyboard to
type in LILO for example.

It has several drawbacks, though:

1) On some machines, the emulated PS/2 mouse takes over even when no USB
   mouse is present and a real PS/2 mouse is present.  In that case the extra
   features (wheel, extra buttons, touchpad mode) of the real PS/2 mouse may
   not be available.

2) If CONFIG_HIGHMEM64G is enabled, the PS/2 mouse emulation can cause
   system crashes, because the SMM BIOS is not expecting to be in PAE mode.
   The Intel E7505 is a typical machine where this happens.

3) If AMD64 64-bit mode is enabled, again system crashes often happen,
   because the SMM BIOS isn't expecting the CPU to be in 64-bit mode.  The
   BIOS manufacturers only test with Windows, and Windows doesn't do 64-bit
   yet.

Solutions:

Problem 1)
  can be solved by loading the USB drivers prior to loading the
  PS/2 mouse driver. Since the PS/2 mouse driver is in 2.6 compiled into
  the kernel unconditionally, this means the USB drivers need to be
  compiled-in, too.

Problem 2)
  can currently only be solved by either disabling HIGHMEM64G
  in the kernel config or USB Legacy support in the BIOS. A BIOS update
  could help, but so far no such update exists.

Problem 3)
  is usually fixed by a BIOS update. Check the board
  manufacturers web site. If an update is not available, disable USB
  Legacy support in the BIOS. If this alone doesn't help, try also adding
  idle=poll on the kernel command line. The BIOS may be entering the SMM
  on the HLT instruction as well.

repo_name: angery/linux


code:
.. SPDX-License-Identifier: GPL-2.0

==============
5-level paging
==============

Overview
========
Original x86-64 was limited by 4-level paging to 256 TiB of virtual address
space and 64 TiB of physical address space. We are already bumping into
this limit: some vendors offer servers with 64 TiB of memory today.

To overcome the limitation upcoming hardware will introduce support for
5-level paging. It is a straight-forward extension of the current page
table structure adding one more layer of translation.

It bumps the limits to 128 PiB of virtual address space and 4 PiB of
physical address space. This "ought to be enough for anybody" ©.

QEMU 2.9 and later support 5-level paging.

Virtual memory layout for 5-level paging is described in
Documentation/x86/x86_64/mm.rst

Enabling 5-level paging
=======================
CONFIG_X86_5LEVEL=y enables the feature.

Kernel with CONFIG_X86_5LEVEL=y still able to boot on 4-level hardware.
In this case additional page table level -- p4d -- will be folded at
runtime.

User-space and large virtual address space
==========================================
On x86, 5-level paging enables 56-bit userspace virtual address space.
Not all user space is ready to handle wide addresses. It's known that
at least some JIT compilers use higher bits in pointers to encode their
information. It collides with valid pointers with 5-level paging and
leads to crashes.

To mitigate this, we are not going to allocate virtual address space
above 47-bit by default.

But userspace can ask for allocation from full address space by
specifying hint address (with or without MAP_FIXED) above 47-bits.

If hint address set above 47-bit, but MAP_FIXED is not specified, we try
to look for unmapped area by specified address. If it's already
occupied, we look for unmapped area in *full* address space, rather than
from 47-bit window.

A high hint address would only affect the allocation in question, but not
any future mmap()s.

Specifying high hint address on older kernel or on machine without 5-level
paging support is safe. The hint will be ignored and kernel will fall back
to allocation from 47-bit address space.

This approach helps to easily make application's memory allocator aware
about large address space without manually tracking allocated virtual
address space.

One important case we need to handle here is interaction with MPX.
MPX (without MAWA extension) cannot handle addresses above 47-bit, so we
need to make sure that MPX cannot be enabled we already have VMA above
the boundary and forbid creating such VMAs once MPX is enabled.


code:
.. SPDX-License-Identifier: GPL-2.0

===========================
AMD64 Specific Boot Options
===========================

There are many others (usually documented in driver documentation), but
only the AMD64 specific ones are listed here.

Machine check
=============
Please see Documentation/x86/x86_64/machinecheck.rst for sysfs runtime tunables.

   mce=off
		Disable machine check
   mce=no_cmci
		Disable CMCI(Corrected Machine Check Interrupt) that
		Intel processor supports.  Usually this disablement is
		not recommended, but it might be handy if your hardware
		is misbehaving.
		Note that you'll get more problems without CMCI than with
		due to the shared banks, i.e. you might get duplicated
		error logs.
   mce=dont_log_ce
		Don't make logs for corrected errors.  All events reported
		as corrected are silently cleared by OS.
		This option will be useful if you have no interest in any
		of corrected errors.
   mce=ignore_ce
		Disable features for corrected errors, e.g. polling timer
		and CMCI.  All events reported as corrected are not cleared
		by OS and remained in its error banks.
		Usually this disablement is not recommended, however if
		there is an agent checking/clearing corrected errors
		(e.g. BIOS or hardware monitoring applications), conflicting
		with OS's error handling, and you cannot deactivate the agent,
		then this option will be a help.
   mce=no_lmce
		Do not opt-in to Local MCE delivery. Use legacy method
		to broadcast MCEs.
   mce=bootlog
		Enable logging of machine checks left over from booting.
		Disabled by default on AMD Fam10h and older because some BIOS
		leave bogus ones.
		If your BIOS doesn't do that it's a good idea to enable though
		to make sure you log even machine check events that result
		in a reboot. On Intel systems it is enabled by default.
   mce=nobootlog
		Disable boot machine check logging.
   mce=monarchtimeout (number)
		monarchtimeout:
		Sets the time in us to wait for other CPUs on machine checks. 0
		to disable.
   mce=bios_cmci_threshold
		Don't overwrite the bios-set CMCI threshold. This boot option
		prevents Linux from overwriting the CMCI threshold set by the
		bios. Without this option, Linux always sets the CMCI
		threshold to 1. Enabling this may make memory predictive failure
		analysis less effective if the bios sets thresholds for memory
		errors since we will not see details for all errors.
   mce=recovery
		Force-enable recoverable machine check code paths

   nomce (for compatibility with i386)
		same as mce=off

   Everything else is in sysfs now.

APICs
=====

   apic
	Use IO-APIC. Default

   noapic
	Don't use the IO-APIC.

   disableapic
	Don't use the local APIC

   nolapic
     Don't use the local APIC (alias for i386 compatibility)

   pirq=...
	See Documentation/x86/i386/IO-APIC.rst

   noapictimer
	Don't set up the APIC timer

   no_timer_check
	Don't check the IO-APIC timer. This can work around
	problems with incorrect timer initialization on some boards.

   apicpmtimer
	Do APIC timer calibration using the pmtimer. Implies
	apicmaintimer. Useful when your PIT timer is totally broken.

Timing
======

  notsc
    Deprecated, use tsc=unstable instead.

  nohpet
    Don't use the HPET timer.

Idle loop
=========

  idle=poll
    Don't do power saving in the idle loop using HLT, but poll for rescheduling
    event. This will make the CPUs eat a lot more power, but may be useful
    to get slightly better performance in multiprocessor benchmarks. It also
    makes some profiling using performance counters more accurate.
    Please note that on systems with MONITOR/MWAIT support (like Intel EM64T
    CPUs) this option has no performance advantage over the normal idle loop.
    It may also interact badly with hyperthreading.

Rebooting
=========

   reboot=b[ios] | t[riple] | k[bd] | a[cpi] | e[fi] | p[ci] [, [w]arm | [c]old]
      bios
        Use the CPU reboot vector for warm reset
      warm
        Don't set the cold reboot flag
      cold
        Set the cold reboot flag
      triple
        Force a triple fault (init)
      kbd
        Use the keyboard controller. cold reset (default)
      acpi
        Use the ACPI RESET_REG in the FADT. If ACPI is not configured or
        the ACPI reset does not work, the reboot path attempts the reset
        using the keyboard controller.
      efi
        Use efi reset_system runtime service. If EFI is not configured or
        the EFI reset does not work, the reboot path attempts the reset using
        the keyboard controller.
      pci
        Use a write to the PCI config space register 0xcf9 to trigger reboot.

   Using warm reset will be much faster especially on big memory
   systems because the BIOS will not go through the memory check.
   Disadvantage is that not all hardware will be completely reinitialized
   on reboot so there may be boot problems on some systems.

   reboot=force
     Don't stop other CPUs on reboot. This can make reboot more reliable
     in some cases.

   reboot=default
     There are some built-in platform specific "quirks" - you may see:
     "reboot: <name> series board detected. Selecting <type> for reboots."
     In the case where you think the quirk is in error (e.g. you have
     newer BIOS, or newer board) using this option will ignore the built-in
     quirk table, and use the generic default reboot actions.

NUMA
====

  numa=off
    Only set up a single NUMA node spanning all memory.

  numa=noacpi
    Don't parse the SRAT table for NUMA setup

  numa=nohmat
    Don't parse the HMAT table for NUMA setup, or soft-reserved memory
    partitioning.

  numa=fake=<size>[MG]
    If given as a memory unit, fills all system RAM with nodes of
    size interleaved over physical nodes.

  numa=fake=<N>
    If given as an integer, fills all system RAM with N fake nodes
    interleaved over physical nodes.

  numa=fake=<N>U
    If given as an integer followed by 'U', it will divide each
    physical node into N emulated nodes.

ACPI
====

  acpi=off
    Don't enable ACPI
  acpi=ht
    Use ACPI boot table parsing, but don't enable ACPI interpreter
  acpi=force
    Force ACPI on (currently not needed)
  acpi=strict
    Disable out of spec ACPI workarounds.
  acpi_sci={edge,level,high,low}
    Set up ACPI SCI interrupt.
  acpi=noirq
    Don't route interrupts
  acpi=nocmcff
    Disable firmware first mode for corrected errors. This
    disables parsing the HEST CMC error source to check if
    firmware has set the FF flag. This may result in
    duplicate corrected error reports.

PCI
===

  pci=off
    Don't use PCI
  pci=conf1
    Use conf1 access.
  pci=conf2
    Use conf2 access.
  pci=rom
    Assign ROMs.
  pci=assign-busses
    Assign busses
  pci=irqmask=MASK
    Set PCI interrupt mask to MASK
  pci=lastbus=NUMBER
    Scan up to NUMBER busses, no matter what the mptable says.
  pci=noacpi
    Don't use ACPI to set up PCI interrupt routing.

IOMMU (input/output memory management unit)
===========================================
Multiple x86-64 PCI-DMA mapping implementations exist, for example:

   1. <kernel/dma/direct.c>: use no hardware/software IOMMU at all
      (e.g. because you have < 3 GB memory).
      Kernel boot message: "PCI-DMA: Disabling IOMMU"

   2. <arch/x86/kernel/amd_gart_64.c>: AMD GART based hardware IOMMU.
      Kernel boot message: "PCI-DMA: using GART IOMMU"

   3. <arch/x86_64/kernel/pci-swiotlb.c> : Software IOMMU implementation. Used
      e.g. if there is no hardware IOMMU in the system and it is need because
      you have >3GB memory or told the kernel to us it (iommu=soft))
      Kernel boot message: "PCI-DMA: Using software bounce buffering
      for IO (SWIOTLB)"

::

  iommu=[<size>][,noagp][,off][,force][,noforce]
  [,memaper[=<order>]][,merge][,fullflush][,nomerge]
  [,noaperture]

General iommu options:

    off
      Don't initialize and use any kind of IOMMU.
    noforce
      Don't force hardware IOMMU usage when it is not needed. (default).
    force
      Force the use of the hardware IOMMU even when it is
      not actually needed (e.g. because < 3 GB memory).
    soft
      Use software bounce buffering (SWIOTLB) (default for
      Intel machines). This can be used to prevent the usage
      of an available hardware IOMMU.

iommu options only relevant to the AMD GART hardware IOMMU:

    <size>
      Set the size of the remapping area in bytes.
    allowed
      Overwrite iommu off workarounds for specific chipsets.
    fullflush
      Flush IOMMU on each allocation (default).
    nofullflush
      Don't use IOMMU fullflush.
    memaper[=<order>]
      Allocate an own aperture over RAM with size 32MB<<order.
      (default: order=1, i.e. 64MB)
    merge
      Do scatter-gather (SG) merging. Implies "force" (experimental).
    nomerge
      Don't do scatter-gather (SG) merging.
    noaperture
      Ask the IOMMU not to touch the aperture for AGP.
    noagp
      Don't initialize the AGP driver and use full aperture.
    panic
      Always panic when IOMMU overflows.

iommu options only relevant to the software bounce buffering (SWIOTLB) IOMMU
implementation:

    swiotlb=<slots>[,force,noforce]
      <slots>
        Prereserve that many 2K slots for the software IO bounce buffering.
      force
        Force all IO through the software TLB.
      noforce
        Do not initialize the software TLB.

Miscellaneous
=============

  nogbpages
    Do not use GB pages for kernel direct mappings.
  gbpages
    Use GB pages for kernel direct mappings.

AMD SEV (Secure Encrypted Virtualization)
=========================================
Options relating to AMD SEV, specified via the following format:

::

   sev=option1[,option2]

The available options are:

   debug
     Enable debug messages.


code:
.. SPDX-License-Identifier: GPL-2.0

===================================================
Firmware support for CPU hotplug under Linux/x86-64
===================================================

Linux/x86-64 supports CPU hotplug now. For various reasons Linux wants to
know in advance of boot time the maximum number of CPUs that could be plugged
into the system. ACPI 3.0 currently has no official way to supply
this information from the firmware to the operating system.

In ACPI each CPU needs an LAPIC object in the MADT table (5.2.11.5 in the
ACPI 3.0 specification).  ACPI already has the concept of disabled LAPIC
objects by setting the Enabled bit in the LAPIC object to zero.

For CPU hotplug Linux/x86-64 expects now that any possible future hotpluggable
CPU is already available in the MADT. If the CPU is not available yet
it should have its LAPIC Enabled bit set to 0. Linux will use the number
of disabled LAPICs to compute the maximum number of future CPUs.

In the worst case the user can overwrite this choice using a command line
option (additional_cpus=...), but it is recommended to supply the correct
number (or a reasonable approximation of it, with erring towards more not less)
in the MADT to avoid manual configuration.


code:
.. SPDX-License-Identifier: GPL-2.0

=====================
Fake NUMA For CPUSets
=====================

:Author: David Rientjes <rientjes@cs.washington.edu>

Using numa=fake and CPUSets for Resource Management

This document describes how the numa=fake x86_64 command-line option can be used
in conjunction with cpusets for coarse memory management.  Using this feature,
you can create fake NUMA nodes that represent contiguous chunks of memory and
assign them to cpusets and their attached tasks.  This is a way of limiting the
amount of system memory that are available to a certain class of tasks.

For more information on the features of cpusets, see
Documentation/admin-guide/cgroup-v1/cpusets.rst.
There are a number of different configurations you can use for your needs.  For
more information on the numa=fake command line option and its various ways of
configuring fake nodes, see Documentation/x86/x86_64/boot-options.rst.

For the purposes of this introduction, we'll assume a very primitive NUMA
emulation setup of "numa=fake=4*512,".  This will split our system memory into
four equal chunks of 512M each that we can now use to assign to cpusets.  As
you become more familiar with using this combination for resource control,
you'll determine a better setup to minimize the number of nodes you have to deal
with.

A machine may be split as follows with "numa=fake=4*512," as reported by dmesg::

	Faking node 0 at 0000000000000000-0000000020000000 (512MB)
	Faking node 1 at 0000000020000000-0000000040000000 (512MB)
	Faking node 2 at 0000000040000000-0000000060000000 (512MB)
	Faking node 3 at 0000000060000000-0000000080000000 (512MB)
	...
	On node 0 totalpages: 130975
	On node 1 totalpages: 131072
	On node 2 totalpages: 131072
	On node 3 totalpages: 131072

Now following the instructions for mounting the cpusets filesystem from
Documentation/admin-guide/cgroup-v1/cpusets.rst, you can assign fake nodes (i.e. contiguous memory
address spaces) to individual cpusets::

	[root@xroads /]# mkdir exampleset
	[root@xroads /]# mount -t cpuset none exampleset
	[root@xroads /]# mkdir exampleset/ddset
	[root@xroads /]# cd exampleset/ddset
	[root@xroads /exampleset/ddset]# echo 0-1 > cpus
	[root@xroads /exampleset/ddset]# echo 0-1 > mems

Now this cpuset, 'ddset', will only allowed access to fake nodes 0 and 1 for
memory allocations (1G).

You can now assign tasks to these cpusets to limit the memory resources
available to them according to the fake nodes assigned as mems::

	[root@xroads /exampleset/ddset]# echo $$ > tasks
	[root@xroads /exampleset/ddset]# dd if=/dev/zero of=tmp bs=1024 count=1G
	[1] 13425

Notice the difference between the system memory usage as reported by
/proc/meminfo between the restricted cpuset case above and the unrestricted
case (i.e. running the same 'dd' command without assigning it to a fake NUMA
cpuset):

	========	============	==========
	Name		Unrestricted	Restricted
	========	============	==========
	MemTotal	3091900 kB	3091900 kB
	MemFree		42113 kB	1513236 kB
	========	============	==========

This allows for coarse memory management for the tasks you assign to particular
cpusets.  Since cpusets can form a hierarchy, you can create some pretty
interesting combinations of use-cases for various classes of tasks for your
memory management needs.


code:
.. SPDX-License-Identifier: GPL-2.0

Using FS and GS segments in user space applications
===================================================

The x86 architecture supports segmentation. Instructions which access
memory can use segment register based addressing mode. The following
notation is used to address a byte within a segment:

  Segment-register:Byte-address

The segment base address is added to the Byte-address to compute the
resulting virtual address which is accessed. This allows to access multiple
instances of data with the identical Byte-address, i.e. the same code. The
selection of a particular instance is purely based on the base-address in
the segment register.

In 32-bit mode the CPU provides 6 segments, which also support segment
limits. The limits can be used to enforce address space protections.

In 64-bit mode the CS/SS/DS/ES segments are ignored and the base address is
always 0 to provide a full 64bit address space. The FS and GS segments are
still functional in 64-bit mode.

Common FS and GS usage
------------------------------

The FS segment is commonly used to address Thread Local Storage (TLS). FS
is usually managed by runtime code or a threading library. Variables
declared with the '__thread' storage class specifier are instantiated per
thread and the compiler emits the FS: address prefix for accesses to these
variables. Each thread has its own FS base address so common code can be
used without complex address offset calculations to access the per thread
instances. Applications should not use FS for other purposes when they use
runtimes or threading libraries which manage the per thread FS.

The GS segment has no common use and can be used freely by
applications. GCC and Clang support GS based addressing via address space
identifiers.

Reading and writing the FS/GS base address
------------------------------------------

There exist two mechanisms to read and write the FS/GS base address:

 - the arch_prctl() system call

 - the FSGSBASE instruction family

Accessing FS/GS base with arch_prctl()
--------------------------------------

 The arch_prctl(2) based mechanism is available on all 64-bit CPUs and all
 kernel versions.

 Reading the base:

   arch_prctl(ARCH_GET_FS, &fsbase);
   arch_prctl(ARCH_GET_GS, &gsbase);

 Writing the base:

   arch_prctl(ARCH_SET_FS, fsbase);
   arch_prctl(ARCH_SET_GS, gsbase);

 The ARCH_SET_GS prctl may be disabled depending on kernel configuration
 and security settings.

Accessing FS/GS base with the FSGSBASE instructions
---------------------------------------------------

 With the Ivy Bridge CPU generation Intel introduced a new set of
 instructions to access the FS and GS base registers directly from user
 space. These instructions are also supported on AMD Family 17H CPUs. The
 following instructions are available:

  =============== ===========================
  RDFSBASE %reg   Read the FS base register
  RDGSBASE %reg   Read the GS base register
  WRFSBASE %reg   Write the FS base register
  WRGSBASE %reg   Write the GS base register
  =============== ===========================

 The instructions avoid the overhead of the arch_prctl() syscall and allow
 more flexible usage of the FS/GS addressing modes in user space
 applications. This does not prevent conflicts between threading libraries
 and runtimes which utilize FS and applications which want to use it for
 their own purpose.

FSGSBASE instructions enablement
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 The instructions are enumerated in CPUID leaf 7, bit 0 of EBX. If
 available /proc/cpuinfo shows 'fsgsbase' in the flag entry of the CPUs.

 The availability of the instructions does not enable them
 automatically. The kernel has to enable them explicitly in CR4. The
 reason for this is that older kernels make assumptions about the values in
 the GS register and enforce them when GS base is set via
 arch_prctl(). Allowing user space to write arbitrary values to GS base
 would violate these assumptions and cause malfunction.

 On kernels which do not enable FSGSBASE the execution of the FSGSBASE
 instructions will fault with a #UD exception.

 The kernel provides reliable information about the enabled state in the
 ELF AUX vector. If the HWCAP2_FSGSBASE bit is set in the AUX vector, the
 kernel has FSGSBASE instructions enabled and applications can use them.
 The following code example shows how this detection works::

   #include <sys/auxv.h>
   #include <elf.h>

   /* Will be eventually in asm/hwcap.h */
   #ifndef HWCAP2_FSGSBASE
   #define HWCAP2_FSGSBASE        (1 << 1)
   #endif

   ....

   unsigned val = getauxval(AT_HWCAP2);

   if (val & HWCAP2_FSGSBASE)
        printf("FSGSBASE enabled\n");

FSGSBASE instructions compiler support
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

GCC version 4.6.4 and newer provide instrinsics for the FSGSBASE
instructions. Clang 5 supports them as well.

  =================== ===========================
  _readfsbase_u64()   Read the FS base register
  _readfsbase_u64()   Read the GS base register
  _writefsbase_u64()  Write the FS base register
  _writegsbase_u64()  Write the GS base register
  =================== ===========================

To utilize these instrinsics <immintrin.h> must be included in the source
code and the compiler option -mfsgsbase has to be added.

Compiler support for FS/GS based addressing
-------------------------------------------

GCC version 6 and newer provide support for FS/GS based addressing via
Named Address Spaces. GCC implements the following address space
identifiers for x86:

  ========= ====================================
  __seg_fs  Variable is addressed relative to FS
  __seg_gs  Variable is addressed relative to GS
  ========= ====================================

The preprocessor symbols __SEG_FS and __SEG_GS are defined when these
address spaces are supported. Code which implements fallback modes should
check whether these symbols are defined. Usage example::

  #ifdef __SEG_GS

  long data0 = 0;
  long data1 = 1;

  long __seg_gs *ptr;

  /* Check whether FSGSBASE is enabled by the kernel (HWCAP2_FSGSBASE) */
  ....

  /* Set GS base to point to data0 */
  _writegsbase_u64(&data0);

  /* Access offset 0 of GS */
  ptr = 0;
  printf("data0 = %ld\n", *ptr);

  /* Set GS base to point to data1 */
  _writegsbase_u64(&data1);
  /* ptr still addresses offset 0! */
  printf("data1 = %ld\n", *ptr);

Clang does not provide the GCC address space identifiers, but it provides
address spaces via an attribute based mechanism in Clang 2.6 and newer
versions:

 ==================================== =====================================
  __attribute__((address_space(256))  Variable is addressed relative to GS
  __attribute__((address_space(257))  Variable is addressed relative to FS
 ==================================== =====================================

FS/GS based addressing with inline assembly
-------------------------------------------

In case the compiler does not support address spaces, inline assembly can
be used for FS/GS based addressing mode::

	mov %fs:offset, %reg
	mov %gs:offset, %reg

	mov %reg, %fs:offset
	mov %reg, %gs:offset


code:
.. SPDX-License-Identifier: GPL-2.0

==============
x86_64 Support
==============

.. toctree::
   :maxdepth: 2

   boot-options
   uefi
   mm
   5level-paging
   fake-numa-for-cpusets
   cpu-hotplug-spec
   machinecheck
   fsgs


code:
.. SPDX-License-Identifier: GPL-2.0

===============================================================
Configurable sysfs parameters for the x86-64 machine check code
===============================================================

Machine checks report internal hardware error conditions detected
by the CPU. Uncorrected errors typically cause a machine check
(often with panic), corrected ones cause a machine check log entry.

Machine checks are organized in banks (normally associated with
a hardware subsystem) and subevents in a bank. The exact meaning
of the banks and subevent is CPU specific.

mcelog knows how to decode them.

When you see the "Machine check errors logged" message in the system
log then mcelog should run to collect and decode machine check entries
from /dev/mcelog. Normally mcelog should be run regularly from a cronjob.

Each CPU has a directory in /sys/devices/system/machinecheck/machinecheckN
(N = CPU number).

The directory contains some configurable entries. See
Documentation/ABI/testing/sysfs-mce for more details.

TBD document entries for AMD threshold interrupt configuration

For more details about the x86 machine check architecture
see the Intel and AMD architecture manuals from their developer websites.

For more details about the architecture
see http://one.firstfloor.org/~andi/mce.pdf


code:
.. SPDX-License-Identifier: GPL-2.0

=================
Memory Management
=================

Complete virtual memory map with 4-level page tables
====================================================

.. note::

 - Negative addresses such as "-23 TB" are absolute addresses in bytes, counted down
   from the top of the 64-bit address space. It's easier to understand the layout
   when seen both in absolute addresses and in distance-from-top notation.

   For example 0xffffe90000000000 == -23 TB, it's 23 TB lower than the top of the
   64-bit address space (ffffffffffffffff).

   Note that as we get closer to the top of the address space, the notation changes
   from TB to GB and then MB/KB.

 - "16M TB" might look weird at first sight, but it's an easier way to visualize size
   notation than "16 EB", which few will recognize at first sight as 16 exabytes.
   It also shows it nicely how incredibly large 64-bit address space is.

::

  ========================================================================================================================
      Start addr    |   Offset   |     End addr     |  Size   | VM area description
  ========================================================================================================================
                    |            |                  |         |
   0000000000000000 |    0       | 00007fffffffffff |  128 TB | user-space virtual memory, different per mm
  __________________|____________|__________________|_________|___________________________________________________________
                    |            |                  |         |
   0000800000000000 | +128    TB | ffff7fffffffffff | ~16M TB | ... huge, almost 64 bits wide hole of non-canonical
                    |            |                  |         |     virtual memory addresses up to the -128 TB
                    |            |                  |         |     starting offset of kernel mappings.
  __________________|____________|__________________|_________|___________________________________________________________
                                                              |
                                                              | Kernel-space virtual memory, shared between all processes:
  ____________________________________________________________|___________________________________________________________
                    |            |                  |         |
   ffff800000000000 | -128    TB | ffff87ffffffffff |    8 TB | ... guard hole, also reserved for hypervisor
   ffff880000000000 | -120    TB | ffff887fffffffff |  0.5 TB | LDT remap for PTI
   ffff888000000000 | -119.5  TB | ffffc87fffffffff |   64 TB | direct mapping of all physical memory (page_offset_base)
   ffffc88000000000 |  -55.5  TB | ffffc8ffffffffff |  0.5 TB | ... unused hole
   ffffc90000000000 |  -55    TB | ffffe8ffffffffff |   32 TB | vmalloc/ioremap space (vmalloc_base)
   ffffe90000000000 |  -23    TB | ffffe9ffffffffff |    1 TB | ... unused hole
   ffffea0000000000 |  -22    TB | ffffeaffffffffff |    1 TB | virtual memory map (vmemmap_base)
   ffffeb0000000000 |  -21    TB | ffffebffffffffff |    1 TB | ... unused hole
   ffffec0000000000 |  -20    TB | fffffbffffffffff |   16 TB | KASAN shadow memory
  __________________|____________|__________________|_________|____________________________________________________________
                                                              |
                                                              | Identical layout to the 56-bit one from here on:
  ____________________________________________________________|____________________________________________________________
                    |            |                  |         |
   fffffc0000000000 |   -4    TB | fffffdffffffffff |    2 TB | ... unused hole
                    |            |                  |         | vaddr_end for KASLR
   fffffe0000000000 |   -2    TB | fffffe7fffffffff |  0.5 TB | cpu_entry_area mapping
   fffffe8000000000 |   -1.5  TB | fffffeffffffffff |  0.5 TB | ... unused hole
   ffffff0000000000 |   -1    TB | ffffff7fffffffff |  0.5 TB | %esp fixup stacks
   ffffff8000000000 | -512    GB | ffffffeeffffffff |  444 GB | ... unused hole
   ffffffef00000000 |  -68    GB | fffffffeffffffff |   64 GB | EFI region mapping space
   ffffffff00000000 |   -4    GB | ffffffff7fffffff |    2 GB | ... unused hole
   ffffffff80000000 |   -2    GB | ffffffff9fffffff |  512 MB | kernel text mapping, mapped to physical address 0
   ffffffff80000000 |-2048    MB |                  |         |
   ffffffffa0000000 |-1536    MB | fffffffffeffffff | 1520 MB | module mapping space
   ffffffffff000000 |  -16    MB |                  |         |
      FIXADDR_START | ~-11    MB | ffffffffff5fffff | ~0.5 MB | kernel-internal fixmap range, variable size and offset
   ffffffffff600000 |  -10    MB | ffffffffff600fff |    4 kB | legacy vsyscall ABI
   ffffffffffe00000 |   -2    MB | ffffffffffffffff |    2 MB | ... unused hole
  __________________|____________|__________________|_________|___________________________________________________________

Complete virtual memory map with 5-level page tables
====================================================

.. note::

 - With 56-bit addresses, user-space memory gets expanded by a factor of 512x,
   from 0.125 PB to 64 PB. All kernel mappings shift down to the -64 PB starting
   offset and many of the regions expand to support the much larger physical
   memory supported.

::

  ========================================================================================================================
      Start addr    |   Offset   |     End addr     |  Size   | VM area description
  ========================================================================================================================
                    |            |                  |         |
   0000000000000000 |    0       | 00ffffffffffffff |   64 PB | user-space virtual memory, different per mm
  __________________|____________|__________________|_________|___________________________________________________________
                    |            |                  |         |
   0100000000000000 |  +64    PB | feffffffffffffff | ~16K PB | ... huge, still almost 64 bits wide hole of non-canonical
                    |            |                  |         |     virtual memory addresses up to the -64 PB
                    |            |                  |         |     starting offset of kernel mappings.
  __________________|____________|__________________|_________|___________________________________________________________
                                                              |
                                                              | Kernel-space virtual memory, shared between all processes:
  ____________________________________________________________|___________________________________________________________
                    |            |                  |         |
   ff00000000000000 |  -64    PB | ff0fffffffffffff |    4 PB | ... guard hole, also reserved for hypervisor
   ff10000000000000 |  -60    PB | ff10ffffffffffff | 0.25 PB | LDT remap for PTI
   ff11000000000000 |  -59.75 PB | ff90ffffffffffff |   32 PB | direct mapping of all physical memory (page_offset_base)
   ff91000000000000 |  -27.75 PB | ff9fffffffffffff | 3.75 PB | ... unused hole
   ffa0000000000000 |  -24    PB | ffd1ffffffffffff | 12.5 PB | vmalloc/ioremap space (vmalloc_base)
   ffd2000000000000 |  -11.5  PB | ffd3ffffffffffff |  0.5 PB | ... unused hole
   ffd4000000000000 |  -11    PB | ffd5ffffffffffff |  0.5 PB | virtual memory map (vmemmap_base)
   ffd6000000000000 |  -10.5  PB | ffdeffffffffffff | 2.25 PB | ... unused hole
   ffdf000000000000 |   -8.25 PB | fffffbffffffffff |   ~8 PB | KASAN shadow memory
  __________________|____________|__________________|_________|____________________________________________________________
                                                              |
                                                              | Identical layout to the 47-bit one from here on:
  ____________________________________________________________|____________________________________________________________
                    |            |                  |         |
   fffffc0000000000 |   -4    TB | fffffdffffffffff |    2 TB | ... unused hole
                    |            |                  |         | vaddr_end for KASLR
   fffffe0000000000 |   -2    TB | fffffe7fffffffff |  0.5 TB | cpu_entry_area mapping
   fffffe8000000000 |   -1.5  TB | fffffeffffffffff |  0.5 TB | ... unused hole
   ffffff0000000000 |   -1    TB | ffffff7fffffffff |  0.5 TB | %esp fixup stacks
   ffffff8000000000 | -512    GB | ffffffeeffffffff |  444 GB | ... unused hole
   ffffffef00000000 |  -68    GB | fffffffeffffffff |   64 GB | EFI region mapping space
   ffffffff00000000 |   -4    GB | ffffffff7fffffff |    2 GB | ... unused hole
   ffffffff80000000 |   -2    GB | ffffffff9fffffff |  512 MB | kernel text mapping, mapped to physical address 0
   ffffffff80000000 |-2048    MB |                  |         |
   ffffffffa0000000 |-1536    MB | fffffffffeffffff | 1520 MB | module mapping space
   ffffffffff000000 |  -16    MB |                  |         |
      FIXADDR_START | ~-11    MB | ffffffffff5fffff | ~0.5 MB | kernel-internal fixmap range, variable size and offset
   ffffffffff600000 |  -10    MB | ffffffffff600fff |    4 kB | legacy vsyscall ABI
   ffffffffffe00000 |   -2    MB | ffffffffffffffff |    2 MB | ... unused hole
  __________________|____________|__________________|_________|___________________________________________________________

Architecture defines a 64-bit virtual address. Implementations can support
less. Currently supported are 48- and 57-bit virtual addresses. Bits 63
through to the most-significant implemented bit are sign extended.
This causes hole between user space and kernel addresses if you interpret them
as unsigned.

The direct mapping covers all memory in the system up to the highest
memory address (this means in some cases it can also include PCI memory
holes).

We map EFI runtime services in the 'efi_pgd' PGD in a 64Gb large virtual
memory window (this size is arbitrary, it can be raised later if needed).
The mappings are not part of any other kernel PGD and are only available
during EFI runtime calls.

Note that if CONFIG_RANDOMIZE_MEMORY is enabled, the direct mapping of all
physical memory, vmalloc/ioremap space and virtual memory map are randomized.
Their order is preserved but their base will be offset early at boot time.

Be very careful vs. KASLR when changing anything here. The KASLR address
range must not overlap with anything except the KASAN shadow area, which is
correct as KASAN disables KASLR.

For both 4- and 5-level layouts, the STACKLEAK_POISON value in the last 2MB
hole: ffffffffffff4111


code:
.. SPDX-License-Identifier: GPL-2.0

=====================================
General note on [U]EFI x86_64 support
=====================================

The nomenclature EFI and UEFI are used interchangeably in this document.

Although the tools below are _not_ needed for building the kernel,
the needed bootloader support and associated tools for x86_64 platforms
with EFI firmware and specifications are listed below.

1. UEFI specification:  http://www.uefi.org

2. Booting Linux kernel on UEFI x86_64 platform requires bootloader
   support. Elilo with x86_64 support can be used.

3. x86_64 platform with EFI/UEFI firmware.

Mechanics
---------

- Build the kernel with the following configuration::

	CONFIG_FB_EFI=y
	CONFIG_FRAMEBUFFER_CONSOLE=y

  If EFI runtime services are expected, the following configuration should
  be selected::

	CONFIG_EFI=y
	CONFIG_EFIVAR_FS=y or m		# optional

- Create a VFAT partition on the disk
- Copy the following to the VFAT partition:

	elilo bootloader with x86_64 support, elilo configuration file,
	kernel image built in first step and corresponding
	initrd. Instructions on building elilo and its dependencies
	can be found in the elilo sourceforge project.

- Boot to EFI shell and invoke elilo choosing the kernel image built
  in first step.
- If some or all EFI runtime services don't work, you can try following
  kernel command line parameters to turn off some or all EFI runtime
  services.

	noefi
		turn off all EFI runtime services
	reboot_type=k
		turn off EFI reboot runtime service

- If the EFI memory map has additional entries not in the E820 map,
  you can include those entries in the kernels memory map of available
  physical RAM by using the following kernel command line parameter.

	add_efi_memmap
		include EFI memory map of available physical RAM


code:
Using XSTATE features in user space applications
================================================

The x86 architecture supports floating-point extensions which are
enumerated via CPUID. Applications consult CPUID and use XGETBV to
evaluate which features have been enabled by the kernel XCR0.

Up to AVX-512 and PKRU states, these features are automatically enabled by
the kernel if available. Features like AMX TILE_DATA (XSTATE component 18)
are enabled by XCR0 as well, but the first use of related instruction is
trapped by the kernel because by default the required large XSTATE buffers
are not allocated automatically.

Using dynamically enabled XSTATE features in user space applications
--------------------------------------------------------------------

The kernel provides an arch_prctl(2) based mechanism for applications to
request the usage of such features. The arch_prctl(2) options related to
this are:

-ARCH_GET_XCOMP_SUPP

 arch_prctl(ARCH_GET_XCOMP_SUPP, &features);

 ARCH_GET_XCOMP_SUPP stores the supported features in userspace storage of
 type uint64_t. The second argument is a pointer to that storage.

-ARCH_GET_XCOMP_PERM

 arch_prctl(ARCH_GET_XCOMP_PERM, &features);

 ARCH_GET_XCOMP_PERM stores the features for which the userspace process
 has permission in userspace storage of type uint64_t. The second argument
 is a pointer to that storage.

-ARCH_REQ_XCOMP_PERM

 arch_prctl(ARCH_REQ_XCOMP_PERM, feature_nr);

 ARCH_REQ_XCOMP_PERM allows to request permission for a dynamically enabled
 feature or a feature set. A feature set can be mapped to a facility, e.g.
 AMX, and can require one or more XSTATE components to be enabled.

 The feature argument is the number of the highest XSTATE component which
 is required for a facility to work.

When requesting permission for a feature, the kernel checks the
availability. The kernel ensures that sigaltstacks in the process's tasks
are large enough to accommodate the resulting large signal frame. It
enforces this both during ARCH_REQ_XCOMP_SUPP and during any subsequent
sigaltstack(2) calls. If an installed sigaltstack is smaller than the
resulting sigframe size, ARCH_REQ_XCOMP_SUPP results in -ENOSUPP. Also,
sigaltstack(2) results in -ENOMEM if the requested altstack is too small
for the permitted features.

Permission, when granted, is valid per process. Permissions are inherited
on fork(2) and cleared on exec(3).

The first use of an instruction related to a dynamically enabled feature is
trapped by the kernel. The trap handler checks whether the process has
permission to use the feature. If the process has no permission then the
kernel sends SIGILL to the application. If the process has permission then
the handler allocates a larger xstate buffer for the task so the large
state can be context switched. In the unlikely cases that the allocation
fails, the kernel sends SIGSEGV.

Dynamic features in signal frames
---------------------------------

Dynamcally enabled features are not written to the signal frame upon signal
entry if the feature is in its initial configuration.  This differs from
non-dynamic features which are always written regardless of their
configuration.  Signal handlers can examine the XSAVE buffer's XSTATE_BV
field to determine if a features was written.


code:
.. SPDX-License-Identifier: GPL-2.0

=========
Zero Page
=========
The additional fields in struct boot_params as a part of 32-bit boot
protocol of kernel. These should be filled by bootloader or 16-bit
real-mode setup code of the kernel. References/settings to it mainly
are in::

  arch/x86/include/uapi/asm/bootparam.h

===========	=====	=======================	=================================================
Offset/Size	Proto	Name			Meaning

000/040		ALL	screen_info		Text mode or frame buffer information
						(struct screen_info)
040/014		ALL	apm_bios_info		APM BIOS information (struct apm_bios_info)
058/008		ALL	tboot_addr      	Physical address of tboot shared page
060/010		ALL	ist_info		Intel SpeedStep (IST) BIOS support information
						(struct ist_info)
070/008		ALL	acpi_rsdp_addr		Physical address of ACPI RSDP table
080/010		ALL	hd0_info		hd0 disk parameter, OBSOLETE!!
090/010		ALL	hd1_info		hd1 disk parameter, OBSOLETE!!
0A0/010		ALL	sys_desc_table		System description table (struct sys_desc_table),
						OBSOLETE!!
0B0/010		ALL	olpc_ofw_header		OLPC's OpenFirmware CIF and friends
0C0/004		ALL	ext_ramdisk_image	ramdisk_image high 32bits
0C4/004		ALL	ext_ramdisk_size	ramdisk_size high 32bits
0C8/004		ALL	ext_cmd_line_ptr	cmd_line_ptr high 32bits
13C/004		ALL	cc_blob_address		Physical address of Confidential Computing blob
140/080		ALL	edid_info		Video mode setup (struct edid_info)
1C0/020		ALL	efi_info		EFI 32 information (struct efi_info)
1E0/004		ALL	alt_mem_k		Alternative mem check, in KB
1E4/004		ALL	scratch			Scratch field for the kernel setup code
1E8/001		ALL	e820_entries		Number of entries in e820_table (below)
1E9/001		ALL	eddbuf_entries		Number of entries in eddbuf (below)
1EA/001		ALL	edd_mbr_sig_buf_entries	Number of entries in edd_mbr_sig_buffer
						(below)
1EB/001		ALL     kbd_status      	Numlock is enabled
1EC/001		ALL     secure_boot		Secure boot is enabled in the firmware
1EF/001		ALL	sentinel		Used to detect broken bootloaders
290/040		ALL	edd_mbr_sig_buffer	EDD MBR signatures
2D0/A00		ALL	e820_table		E820 memory map table
						(array of struct e820_entry)
D00/1EC		ALL	eddbuf			EDD data (array of struct edd_info)
===========	=====	=======================	=================================================


code:
===========================================
Atomic Operation Control (ATOMCTL) Register
===========================================

We Have Atomic Operation Control (ATOMCTL) Register.
This register determines the effect of using a S32C1I instruction
with various combinations of:

     1. With and without an Coherent Cache Controller which
        can do Atomic Transactions to the memory internally.

     2. With and without An Intelligent Memory Controller which
        can do Atomic Transactions itself.

The Core comes up with a default value of for the three types of cache ops::

      0x28: (WB: Internal, WT: Internal, BY:Exception)

On the FPGA Cards we typically simulate an Intelligent Memory controller
which can implement  RCW transactions. For FPGA cards with an External
Memory controller we let it to the atomic operations internally while
doing a Cached (WB) transaction and use the Memory RCW for un-cached
operations.

For systems without an coherent cache controller, non-MX, we always
use the memory controllers RCW, thought non-MX controlers likely
support the Internal Operation.

CUSTOMER-WARNING:
   Virtually all customers buy their memory controllers from vendors that
   don't support atomic RCW memory transactions and will likely want to
   configure this register to not use RCW.

Developers might find using RCW in Bypass mode convenient when testing
with the cache being bypassed; for example studying cache alias problems.

See Section 4.3.12.4 of ISA; Bits::

                             WB     WT      BY
                           5   4 | 3   2 | 1   0

=========    ==================      ==================      ===============
  2 Bit
  Field
  Values     WB - Write Back         WT - Write Thru         BY - Bypass
=========    ==================      ==================      ===============
    0        Exception               Exception               Exception
    1        RCW Transaction         RCW Transaction         RCW Transaction
    2        Internal Operation      Internal Operation      Reserved
    3        Reserved                Reserved                Reserved
=========    ==================      ==================      ===============


code:
=====================================
Passing boot parameters to the kernel
=====================================

Boot parameters are represented as a TLV list in the memory. Please see
arch/xtensa/include/asm/bootparam.h for definition of the bp_tag structure and
tag value constants. First entry in the list must have type BP_TAG_FIRST, last
entry must have type BP_TAG_LAST. The address of the first list entry is
passed to the kernel in the register a2. The address type depends on MMU type:

- For configurations without MMU, with region protection or with MPU the
  address must be the physical address.
- For configurations with region translarion MMU or with MMUv3 and CONFIG_MMU=n
  the address must be a valid address in the current mapping. The kernel will
  not change the mapping on its own.
- For configurations with MMUv2 the address must be a virtual address in the
  default virtual mapping (0xd0000000..0xffffffff).
- For configurations with MMUv3 and CONFIG_MMU=y the address may be either a
  virtual or physical address. In either case it must be within the default
  virtual mapping. It is considered physical if it is within the range of
  physical addresses covered by the default KSEG mapping (XCHAL_KSEG_PADDR..
  XCHAL_KSEG_PADDR + XCHAL_KSEG_SIZE), otherwise it is considered virtual.


code:
.. SPDX-License-Identifier: GPL-2.0

.. kernel-feat:: $srctree/Documentation/features xtensa


code:
.. SPDX-License-Identifier: GPL-2.0

===================
Xtensa Architecture
===================

.. toctree::
   :maxdepth: 1

   atomctl
   booting
   mmu

   features


code:
=============================
MMUv3 initialization sequence
=============================

The code in the initialize_mmu macro sets up MMUv3 memory mapping
identically to MMUv2 fixed memory mapping. Depending on
CONFIG_INITIALIZE_XTENSA_MMU_INSIDE_VMLINUX symbol this code is
located in addresses it was linked for (symbol undefined), or not
(symbol defined), so it needs to be position-independent.

The code has the following assumptions:

  - This code fragment is run only on an MMU v3.
  - TLBs are in their reset state.
  - ITLBCFG and DTLBCFG are zero (reset state).
  - RASID is 0x04030201 (reset state).
  - PS.RING is zero (reset state).
  - LITBASE is zero (reset state, PC-relative literals); required to be PIC.

TLB setup proceeds along the following steps.

  Legend:

    - VA = virtual address (two upper nibbles of it);
    - PA = physical address (two upper nibbles of it);
    - pc = physical range that contains this code;

After step 2, we jump to virtual address in the range 0x40000000..0x5fffffff
or 0x00000000..0x1fffffff, depending on whether the kernel was loaded below
0x40000000 or above. That address corresponds to next instruction to execute
in this code. After step 4, we jump to intended (linked) address of this code.
The scheme below assumes that the kernel is loaded below 0x40000000.

 ====== =====  =====  =====  =====   ====== =====  =====
 -      Step0  Step1  Step2  Step3          Step4  Step5

   VA      PA     PA     PA     PA     VA      PA     PA
 ====== =====  =====  =====  =====   ====== =====  =====
 E0..FF -> E0  -> E0  -> E0          F0..FF -> F0  -> F0
 C0..DF -> C0  -> C0  -> C0          E0..EF -> F0  -> F0
 A0..BF -> A0  -> A0  -> A0          D8..DF -> 00  -> 00
 80..9F -> 80  -> 80  -> 80          D0..D7 -> 00  -> 00
 60..7F -> 60  -> 60  -> 60
 40..5F -> 40         -> pc  -> pc   40..5F -> pc
 20..3F -> 20  -> 20  -> 20
 00..1F -> 00  -> 00  -> 00
 ====== =====  =====  =====  =====   ====== =====  =====

The default location of IO peripherals is above 0xf0000000. This may be changed
using a "ranges" property in a device tree simple-bus node. See the Devicetree
Specification, section 4.5 for details on the syntax and semantics of
simple-bus nodes. The following limitations apply:

1. Only top level simple-bus nodes are considered

2. Only one (first) simple-bus node is considered

3. Empty "ranges" properties are not supported

4. Only the first triplet in the "ranges" property is considered

5. The parent-bus-address value is rounded down to the nearest 256MB boundary

6. The IO area covers the entire 256MB segment of parent-bus-address; the
   "ranges" triplet length field is ignored

MMUv3 address space layouts.
============================

Default MMUv2-compatible layout::

                        Symbol                   VADDR       Size
  +------------------+
  | Userspace        |                           0x00000000  TASK_SIZE
  +------------------+                           0x40000000
  +------------------+
  | Page table       |  XCHAL_PAGE_TABLE_VADDR   0x80000000  XCHAL_PAGE_TABLE_SIZE
  +------------------+
  | KASAN shadow map |  KASAN_SHADOW_START       0x80400000  KASAN_SHADOW_SIZE
  +------------------+                           0x8e400000
  +------------------+
  | VMALLOC area     |  VMALLOC_START            0xc0000000  128MB - 64KB
  +------------------+  VMALLOC_END
  +------------------+
  | Cache aliasing   |  TLBTEMP_BASE_1           0xc8000000  DCACHE_WAY_SIZE
  | remap area 1     |
  +------------------+
  | Cache aliasing   |  TLBTEMP_BASE_2                       DCACHE_WAY_SIZE
  | remap area 2     |
  +------------------+
  +------------------+
  | KMAP area        |  PKMAP_BASE                           PTRS_PER_PTE *
  |                  |                                       DCACHE_N_COLORS *
  |                  |                                       PAGE_SIZE
  |                  |                                       (4MB * DCACHE_N_COLORS)
  +------------------+
  | Atomic KMAP area |  FIXADDR_START                        KM_TYPE_NR *
  |                  |                                       NR_CPUS *
  |                  |                                       DCACHE_N_COLORS *
  |                  |                                       PAGE_SIZE
  +------------------+  FIXADDR_TOP              0xcffff000
  +------------------+
  | Cached KSEG      |  XCHAL_KSEG_CACHED_VADDR  0xd0000000  128MB
  +------------------+
  | Uncached KSEG    |  XCHAL_KSEG_BYPASS_VADDR  0xd8000000  128MB
  +------------------+
  | Cached KIO       |  XCHAL_KIO_CACHED_VADDR   0xe0000000  256MB
  +------------------+
  | Uncached KIO     |  XCHAL_KIO_BYPASS_VADDR   0xf0000000  256MB
  +------------------+

256MB cached + 256MB uncached layout::

                        Symbol                   VADDR       Size
  +------------------+
  | Userspace        |                           0x00000000  TASK_SIZE
  +------------------+                           0x40000000
  +------------------+
  | Page table       |  XCHAL_PAGE_TABLE_VADDR   0x80000000  XCHAL_PAGE_TABLE_SIZE
  +------------------+
  | KASAN shadow map |  KASAN_SHADOW_START       0x80400000  KASAN_SHADOW_SIZE
  +------------------+                           0x8e400000
  +------------------+
  | VMALLOC area     |  VMALLOC_START            0xa0000000  128MB - 64KB
  +------------------+  VMALLOC_END
  +------------------+
  | Cache aliasing   |  TLBTEMP_BASE_1           0xa8000000  DCACHE_WAY_SIZE
  | remap area 1     |
  +------------------+
  | Cache aliasing   |  TLBTEMP_BASE_2                       DCACHE_WAY_SIZE
  | remap area 2     |
  +------------------+
  +------------------+
  | KMAP area        |  PKMAP_BASE                           PTRS_PER_PTE *
  |                  |                                       DCACHE_N_COLORS *
  |                  |                                       PAGE_SIZE
  |                  |                                       (4MB * DCACHE_N_COLORS)
  +------------------+
  | Atomic KMAP area |  FIXADDR_START                        KM_TYPE_NR *
  |                  |                                       NR_CPUS *
  |                  |                                       DCACHE_N_COLORS *
  |                  |                                       PAGE_SIZE
  +------------------+  FIXADDR_TOP              0xaffff000
  +------------------+
  | Cached KSEG      |  XCHAL_KSEG_CACHED_VADDR  0xb0000000  256MB
  +------------------+
  | Uncached KSEG    |  XCHAL_KSEG_BYPASS_VADDR  0xc0000000  256MB
  +------------------+
  +------------------+
  | Cached KIO       |  XCHAL_KIO_CACHED_VADDR   0xe0000000  256MB
  +------------------+
  | Uncached KIO     |  XCHAL_KIO_BYPASS_VADDR   0xf0000000  256MB
  +------------------+

512MB cached + 512MB uncached layout::

                        Symbol                   VADDR       Size
  +------------------+
  | Userspace        |                           0x00000000  TASK_SIZE
  +------------------+                           0x40000000
  +------------------+
  | Page table       |  XCHAL_PAGE_TABLE_VADDR   0x80000000  XCHAL_PAGE_TABLE_SIZE
  +------------------+
  | KASAN shadow map |  KASAN_SHADOW_START       0x80400000  KASAN_SHADOW_SIZE
  +------------------+                           0x8e400000
  +------------------+
  | VMALLOC area     |  VMALLOC_START            0x90000000  128MB - 64KB
  +------------------+  VMALLOC_END
  +------------------+
  | Cache aliasing   |  TLBTEMP_BASE_1           0x98000000  DCACHE_WAY_SIZE
  | remap area 1     |
  +------------------+
  | Cache aliasing   |  TLBTEMP_BASE_2                       DCACHE_WAY_SIZE
  | remap area 2     |
  +------------------+
  +------------------+
  | KMAP area        |  PKMAP_BASE                           PTRS_PER_PTE *
  |                  |                                       DCACHE_N_COLORS *
  |                  |                                       PAGE_SIZE
  |                  |                                       (4MB * DCACHE_N_COLORS)
  +------------------+
  | Atomic KMAP area |  FIXADDR_START                        KM_TYPE_NR *
  |                  |                                       NR_CPUS *
  |                  |                                       DCACHE_N_COLORS *
  |                  |                                       PAGE_SIZE
  +------------------+  FIXADDR_TOP              0x9ffff000
  +------------------+
  | Cached KSEG      |  XCHAL_KSEG_CACHED_VADDR  0xa0000000  512MB
  +------------------+
  | Uncached KSEG    |  XCHAL_KSEG_BYPASS_VADDR  0xc0000000  512MB
  +------------------+
  | Cached KIO       |  XCHAL_KIO_CACHED_VADDR   0xe0000000  256MB
  +------------------+
  | Uncached KIO     |  XCHAL_KIO_BYPASS_VADDR   0xf0000000  256MB
  +------------------+


code:
# SPDX-License-Identifier: GPL-2.0
#
# Kbuild for top-level directory of the kernel

# Prepare global headers and check sanity before descending into sub-directories
# ---------------------------------------------------------------------------

# Generate bounds.h

bounds-file := include/generated/bounds.h

targets := kernel/bounds.s

$(bounds-file): kernel/bounds.s FORCE
	$(call filechk,offsets,__LINUX_BOUNDS_H__)

# Generate timeconst.h

timeconst-file := include/generated/timeconst.h

filechk_gentimeconst = echo $(CONFIG_HZ) | bc -q $<

$(timeconst-file): kernel/time/timeconst.bc FORCE
	$(call filechk,gentimeconst)

# Generate asm-offsets.h

offsets-file := include/generated/asm-offsets.h

targets += arch/$(SRCARCH)/kernel/asm-offsets.s

arch/$(SRCARCH)/kernel/asm-offsets.s: $(timeconst-file) $(bounds-file)

$(offsets-file): arch/$(SRCARCH)/kernel/asm-offsets.s FORCE
	$(call filechk,offsets,__ASM_OFFSETS_H__)

# Check for missing system calls

quiet_cmd_syscalls = CALL    $<
      cmd_syscalls = $(CONFIG_SHELL) $< $(CC) $(c_flags) $(missing_syscalls_flags)

PHONY += missing-syscalls
missing-syscalls: scripts/checksyscalls.sh $(offsets-file)
	$(call cmd,syscalls)

# Check the manual modification of atomic headers

quiet_cmd_check_sha1 = CHKSHA1 $<
      cmd_check_sha1 = \
	if ! command -v sha1sum >/dev/null; then \
		echo "warning: cannot check the header due to sha1sum missing"; \
		exit 0; \
	fi; \
	if [ "$$(sed -n '$$s:// ::p' $<)" != \
	     "$$(sed '$$d' $< | sha1sum | sed 's/ .*//')" ]; then \
		echo "error: $< has been modified." >&2; \
		exit 1; \
	fi; \
	touch $@

atomic-checks += $(addprefix $(obj)/.checked-, \
	  atomic-arch-fallback.h \
	  atomic-instrumented.h \
	  atomic-long.h)

targets += $(atomic-checks)
$(atomic-checks): $(obj)/.checked-%: include/linux/atomic/%  FORCE
	$(call if_changed,check_sha1)

# A phony target that depends on all the preparation targets

PHONY += prepare
prepare: $(offsets-file) missing-syscalls $(atomic-checks)
	@:

# Ordinary directory descending
# ---------------------------------------------------------------------------

obj-y			+= init/
obj-y			+= usr/
obj-y			+= arch/$(SRCARCH)/
obj-y			+= $(ARCH_CORE)
obj-y			+= kernel/
obj-y			+= certs/
obj-y			+= mm/
obj-y			+= fs/
obj-y			+= ipc/
obj-y			+= security/
obj-y			+= crypto/
obj-$(CONFIG_BLOCK)	+= block/
obj-$(CONFIG_IO_URING)	+= io_uring/
obj-$(CONFIG_RUST)	+= rust/
obj-y			+= $(ARCH_LIB)
obj-y			+= drivers/
obj-y			+= sound/
obj-$(CONFIG_SAMPLES)	+= samples/
obj-$(CONFIG_NET)	+= net/
obj-y			+= virt/
obj-y			+= $(ARCH_DRIVERS)

path: Kbuild


code:
# SPDX-License-Identifier: GPL-2.0
#
# For a description of the syntax of this configuration file,
# see Documentation/kbuild/kconfig-language.rst.
#
mainmenu "Linux/$(ARCH) $(KERNELVERSION) Kernel Configuration"

source "scripts/Kconfig.include"

source "init/Kconfig"

source "kernel/Kconfig.freezer"

source "fs/Kconfig.binfmt"

source "mm/Kconfig"

source "net/Kconfig"

source "drivers/Kconfig"

source "fs/Kconfig"

source "security/Kconfig"

source "crypto/Kconfig"

source "lib/Kconfig"

source "lib/Kconfig.debug"

source "Documentation/Kconfig"

path: Kconfig


code:
Valid-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
Valid-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-only
Valid-License-Identifier: GFDL-1.1-no-invariants-or-later
Valid-License-Identifier: GFDL-1.1-no-invariants-only
SPDX-URL: https://spdx.org/licenses/GFDL-1.1-no-invariants-or-later.html
Usage-Guide:
  The GNU Free Documentation License should only be used without
  Invariant Sections, Front-Cover Texts or Back-Cover Texts.
  It should not be used for new documents, except when they are
  part of sections that are already using it (notably, the
  userspace-api media documentation).
  To use the license in source code, put the following SPDX tag/value pair
  into a comment according to the placement guidelines in the licensing
  rules documentation:
    SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
  or
    SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-only
  or
    SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
  or
    SPDX-License-Identifier: GFDL-1.1-no-invariants-only
License-Text:
                GNU Free Documentation License
                   Version 1.1, March 2000

 Copyright (C) 2000  Free Software Foundation, Inc.
     51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
 Everyone is permitted to copy and distribute verbatim copies
 of this license document, but changing it is not allowed.

0. PREAMBLE

The purpose of this License is to make a manual, textbook, or other
written document "free" in the sense of freedom: to assure everyone
the effective freedom to copy and redistribute it, with or without
modifying it, either commercially or noncommercially.  Secondarily,
this License preserves for the author and publisher a way to get
credit for their work, while not being considered responsible for
modifications made by others.

This License is a kind of "copyleft", which means that derivative
works of the document must themselves be free in the same sense.  It
complements the GNU General Public License, which is a copyleft
license designed for free software.

We have designed this License in order to use it for manuals for free
software, because free software needs free documentation: a free
program should come with manuals providing the same freedoms that the
software does.  But this License is not limited to software manuals;
it can be used for any textual work, regardless of subject matter or
whether it is published as a printed book.  We recommend this License
principally for works whose purpose is instruction or reference.

1. APPLICABILITY AND DEFINITIONS

This License applies to any manual or other work that contains a
notice placed by the copyright holder saying it can be distributed
under the terms of this License.  The "Document", below, refers to any
such manual or work.  Any member of the public is a licensee, and is
addressed as "you".

A "Modified Version" of the Document means any work containing the
Document or a portion of it, either copied verbatim, or with
modifications and/or translated into another language.

A "Secondary Section" is a named appendix or a front-matter section of
the Document that deals exclusively with the relationship of the
publishers or authors of the Document to the Document's overall subject
(or to related matters) and contains nothing that could fall directly
within that overall subject.  (For example, if the Document is in part a
textbook of mathematics, a Secondary Section may not explain any
mathematics.)  The relationship could be a matter of historical
connection with the subject or with related matters, or of legal,
commercial, philosophical, ethical or political position regarding
them.

The "Invariant Sections" are certain Secondary Sections whose titles
are designated, as being those of Invariant Sections, in the notice
that says that the Document is released under this License.

The "Cover Texts" are certain short passages of text that are listed,
as Front-Cover Texts or Back-Cover Texts, in the notice that says that
the Document is released under this License.

A "Transparent" copy of the Document means a machine-readable copy,
represented in a format whose specification is available to the
general public, whose contents can be viewed and edited directly and
straightforwardly with generic text editors or (for images composed of
pixels) generic paint programs or (for drawings) some widely available
drawing editor, and that is suitable for input to text formatters or
for automatic translation to a variety of formats suitable for input
to text formatters.  A copy made in an otherwise Transparent file
format whose markup has been designed to thwart or discourage
subsequent modification by readers is not Transparent.  A copy that is
not "Transparent" is called "Opaque".

Examples of suitable formats for Transparent copies include plain
ASCII without markup, Texinfo input format, LaTeX input format, SGML
or XML using a publicly available DTD, and standard-conforming simple
HTML designed for human modification.  Opaque formats include
PostScript, PDF, proprietary formats that can be read and edited only
by proprietary word processors, SGML or XML for which the DTD and/or
processing tools are not generally available, and the
machine-generated HTML produced by some word processors for output
purposes only.

The "Title Page" means, for a printed book, the title page itself,
plus such following pages as are needed to hold, legibly, the material
this License requires to appear in the title page.  For works in
formats which do not have any title page as such, "Title Page" means
the text near the most prominent appearance of the work's title,
preceding the beginning of the body of the text.

2. VERBATIM COPYING

You may copy and distribute the Document in any medium, either
commercially or noncommercially, provided that this License, the
copyright notices, and the license notice saying this License applies
to the Document are reproduced in all copies, and that you add no other
conditions whatsoever to those of this License.  You may not use
technical measures to obstruct or control the reading or further
copying of the copies you make or distribute.  However, you may accept
compensation in exchange for copies.  If you distribute a large enough
number of copies you must also follow the conditions in section 3.

You may also lend copies, under the same conditions stated above, and
you may publicly display copies.

3. COPYING IN QUANTITY

If you publish printed copies of the Document numbering more than 100,
and the Document's license notice requires Cover Texts, you must enclose
the copies in covers that carry, clearly and legibly, all these Cover
Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on
the back cover.  Both covers must also clearly and legibly identify
you as the publisher of these copies.  The front cover must present
the full title with all words of the title equally prominent and
visible.  You may add other material on the covers in addition.
Copying with changes limited to the covers, as long as they preserve
the title of the Document and satisfy these conditions, can be treated
as verbatim copying in other respects.

If the required texts for either cover are too voluminous to fit
legibly, you should put the first ones listed (as many as fit
reasonably) on the actual cover, and continue the rest onto adjacent
pages.

If you publish or distribute Opaque copies of the Document numbering
more than 100, you must either include a machine-readable Transparent
copy along with each Opaque copy, or state in or with each Opaque copy
a publicly-accessible computer-network location containing a complete
Transparent copy of the Document, free of added material, which the
general network-using public has access to download anonymously at no
charge using public-standard network protocols.  If you use the latter
option, you must take reasonably prudent steps, when you begin
distribution of Opaque copies in quantity, to ensure that this
Transparent copy will remain thus accessible at the stated location
until at least one year after the last time you distribute an Opaque
copy (directly or through your agents or retailers) of that edition to
the public.

It is requested, but not required, that you contact the authors of the
Document well before redistributing any large number of copies, to give
them a chance to provide you with an updated version of the Document.

4. MODIFICATIONS

You may copy and distribute a Modified Version of the Document under
the conditions of sections 2 and 3 above, provided that you release
the Modified Version under precisely this License, with the Modified
Version filling the role of the Document, thus licensing distribution
and modification of the Modified Version to whoever possesses a copy
of it.  In addition, you must do these things in the Modified Version:

A. Use in the Title Page (and on the covers, if any) a title distinct
   from that of the Document, and from those of previous versions
   (which should, if there were any, be listed in the History section
   of the Document).  You may use the same title as a previous version
   if the original publisher of that version gives permission.
B. List on the Title Page, as authors, one or more persons or entities
   responsible for authorship of the modifications in the Modified
   Version, together with at least five of the principal authors of the
   Document (all of its principal authors, if it has less than five).
C. State on the Title page the name of the publisher of the
   Modified Version, as the publisher.
D. Preserve all the copyright notices of the Document.
E. Add an appropriate copyright notice for your modifications
   adjacent to the other copyright notices.
F. Include, immediately after the copyright notices, a license notice
   giving the public permission to use the Modified Version under the
   terms of this License, in the form shown in the Addendum below.
G. Preserve in that license notice the full lists of Invariant Sections
   and required Cover Texts given in the Document's license notice.
H. Include an unaltered copy of this License.
I. Preserve the section entitled "History", and its title, and add to
   it an item stating at least the title, year, new authors, and
   publisher of the Modified Version as given on the Title Page.  If
   there is no section entitled "History" in the Document, create one
   stating the title, year, authors, and publisher of the Document as
   given on its Title Page, then add an item describing the Modified
   Version as stated in the previous sentence.
J. Preserve the network location, if any, given in the Document for
   public access to a Transparent copy of the Document, and likewise
   the network locations given in the Document for previous versions
   it was based on.  These may be placed in the "History" section.
   You may omit a network location for a work that was published at
   least four years before the Document itself, or if the original
   publisher of the version it refers to gives permission.
K. In any section entitled "Acknowledgements" or "Dedications",
   preserve the section's title, and preserve in the section all the
   substance and tone of each of the contributor acknowledgements
   and/or dedications given therein.
L. Preserve all the Invariant Sections of the Document,
   unaltered in their text and in their titles.  Section numbers
   or the equivalent are not considered part of the section titles.
M. Delete any section entitled "Endorsements".  Such a section
   may not be included in the Modified Version.
N. Do not retitle any existing section as "Endorsements"
   or to conflict in title with any Invariant Section.

If the Modified Version includes new front-matter sections or
appendices that qualify as Secondary Sections and contain no material
copied from the Document, you may at your option designate some or all
of these sections as invariant.  To do this, add their titles to the
list of Invariant Sections in the Modified Version's license notice.
These titles must be distinct from any other section titles.

You may add a section entitled "Endorsements", provided it contains
nothing but endorsements of your Modified Version by various
parties--for example, statements of peer review or that the text has
been approved by an organization as the authoritative definition of a
standard.

You may add a passage of up to five words as a Front-Cover Text, and a
passage of up to 25 words as a Back-Cover Text, to the end of the list
of Cover Texts in the Modified Version.  Only one passage of
Front-Cover Text and one of Back-Cover Text may be added by (or
through arrangements made by) any one entity.  If the Document already
includes a cover text for the same cover, previously added by you or
by arrangement made by the same entity you are acting on behalf of,
you may not add another; but you may replace the old one, on explicit
permission from the previous publisher that added the old one.

The author(s) and publisher(s) of the Document do not by this License
give permission to use their names for publicity for or to assert or
imply endorsement of any Modified Version.

5. COMBINING DOCUMENTS

You may combine the Document with other documents released under this
License, under the terms defined in section 4 above for modified
versions, provided that you include in the combination all of the
Invariant Sections of all of the original documents, unmodified, and
list them all as Invariant Sections of your combined work in its
license notice.

The combined work need only contain one copy of this License, and
multiple identical Invariant Sections may be replaced with a single
copy.  If there are multiple Invariant Sections with the same name but
different contents, make the title of each such section unique by
adding at the end of it, in parentheses, the name of the original
author or publisher of that section if known, or else a unique number.
Make the same adjustment to the section titles in the list of
Invariant Sections in the license notice of the combined work.

In the combination, you must combine any sections entitled "History"
in the various original documents, forming one section entitled
"History"; likewise combine any sections entitled "Acknowledgements",
and any sections entitled "Dedications".  You must delete all sections
entitled "Endorsements."

6. COLLECTIONS OF DOCUMENTS

You may make a collection consisting of the Document and other documents
released under this License, and replace the individual copies of this
License in the various documents with a single copy that is included in
the collection, provided that you follow the rules of this License for
verbatim copying of each of the documents in all other respects.

You may extract a single document from such a collection, and distribute
it individually under this License, provided you insert a copy of this
License into the extracted document, and follow this License in all
other respects regarding verbatim copying of that document.

7. AGGREGATION WITH INDEPENDENT WORKS

A compilation of the Document or its derivatives with other separate
and independent documents or works, in or on a volume of a storage or
distribution medium, does not as a whole count as a Modified Version
of the Document, provided no compilation copyright is claimed for the
compilation.  Such a compilation is called an "aggregate", and this
License does not apply to the other self-contained works thus compiled
with the Document, on account of their being thus compiled, if they
are not themselves derivative works of the Document.

If the Cover Text requirement of section 3 is applicable to these
copies of the Document, then if the Document is less than one quarter
of the entire aggregate, the Document's Cover Texts may be placed on
covers that surround only the Document within the aggregate.
Otherwise they must appear on covers around the whole aggregate.

8. TRANSLATION

Translation is considered a kind of modification, so you may
distribute translations of the Document under the terms of section 4.
Replacing Invariant Sections with translations requires special
permission from their copyright holders, but you may include
translations of some or all Invariant Sections in addition to the
original versions of these Invariant Sections.  You may include a
translation of this License provided that you also include the
original English version of this License.  In case of a disagreement
between the translation and the original English version of this
License, the original English version will prevail.

9. TERMINATION

You may not copy, modify, sublicense, or distribute the Document except
as expressly provided for under this License.  Any other attempt to
copy, modify, sublicense or distribute the Document is void, and will
automatically terminate your rights under this License.  However,
parties who have received copies, or rights, from you under this
License will not have their licenses terminated so long as such
parties remain in full compliance.

10. FUTURE REVISIONS OF THIS LICENSE

The Free Software Foundation may publish new, revised versions
of the GNU Free Documentation License from time to time.  Such new
versions will be similar in spirit to the present version, but may
differ in detail to address new problems or concerns.  See
https://www.gnu.org/licenses/.

Each version of the License is given a distinguishing version number.
If the Document specifies that a particular numbered version of this
License "or any later version" applies to it, you have the option of
following the terms and conditions either of that specified version or
of any later version that has been published (not as a draft) by the
Free Software Foundation.  If the Document does not specify a version
number of this License, you may choose any version ever published (not
as a draft) by the Free Software Foundation.

ADDENDUM: How to use this License for your documents

To use this License in a document you have written, include a copy of
the License in the document and put the following copyright and
license notices just after the title page:

      Copyright (c)  YEAR  YOUR NAME.
      Permission is granted to copy, distribute and/or modify this document
      under the terms of the GNU Free Documentation License, Version 1.1
      or any later version published by the Free Software Foundation;
      with the Invariant Sections being LIST THEIR TITLES, with the
      Front-Cover Texts being LIST, and with the Back-Cover Texts being LIST.
      A copy of the license is included in the section entitled "GNU
      Free Documentation License".

If you have no Invariant Sections, write "with no Invariant Sections"
instead of saying which ones are invariant.  If you have no
Front-Cover Texts, write "no Front-Cover Texts" instead of
"Front-Cover Texts being LIST"; likewise for Back-Cover Texts.

If your document contains nontrivial examples of program code, we
recommend releasing these examples in parallel under your choice of
free software license, such as the GNU General Public License,
to permit their use in free software.