code:
.. SPDX-License-Identifier: GPL-2.0
.. include:: <isonum.txt>

===========================================
User Interface for Resource Control feature
===========================================

:Copyright: |copy| 2016 Intel Corporation
:Authors: - Fenghua Yu <fenghua.yu@intel.com>
          - Tony Luck <tony.luck@intel.com>
          - Vikas Shivappa <vikas.shivappa@intel.com>

Intel refers to this feature as Intel Resource Director Technology(Intel(R) RDT).
AMD refers to this feature as AMD Platform Quality of Service(AMD QoS).

This feature is enabled by the CONFIG_X86_CPU_RESCTRL and the x86 /proc/cpuinfo
flag bits:

=============================================	================================
RDT (Resource Director Technology) Allocation	"rdt_a"
CAT (Cache Allocation Technology)		"cat_l3", "cat_l2"
CDP (Code and Data Prioritization)		"cdp_l3", "cdp_l2"
CQM (Cache QoS Monitoring)			"cqm_llc", "cqm_occup_llc"
MBM (Memory Bandwidth Monitoring)		"cqm_mbm_total", "cqm_mbm_local"
MBA (Memory Bandwidth Allocation)		"mba"
=============================================	================================

To use the feature mount the file system::

 # mount -t resctrl resctrl [-o cdp[,cdpl2][,mba_MBps]] /sys/fs/resctrl

mount options are:

"cdp":
	Enable code/data prioritization in L3 cache allocations.
"cdpl2":
	Enable code/data prioritization in L2 cache allocations.
"mba_MBps":
	Enable the MBA Software Controller(mba_sc) to specify MBA
	bandwidth in MBps

L2 and L3 CDP are controlled separately.

RDT features are orthogonal. A particular system may support only
monitoring, only control, or both monitoring and control.  Cache
pseudo-locking is a unique way of using cache control to "pin" or
"lock" data in the cache. Details can be found in
"Cache Pseudo-Locking".

The mount succeeds if either of allocation or monitoring is present, but
only those files and directories supported by the system will be created.
For more details on the behavior of the interface during monitoring
and allocation, see the "Resource alloc and monitor groups" section.

Info directory
==============

The 'info' directory contains information about the enabled
resources. Each resource has its own subdirectory. The subdirectory
names reflect the resource names.

Each subdirectory contains the following files with respect to
allocation:

Cache resource(L3/L2)  subdirectory contains the following files
related to allocation:

"num_closids":
		The number of CLOSIDs which are valid for this
		resource. The kernel uses the smallest number of
		CLOSIDs of all enabled resources as limit.
"cbm_mask":
		The bitmask which is valid for this resource.
		This mask is equivalent to 100%.
"min_cbm_bits":
		The minimum number of consecutive bits which
		must be set when writing a mask.

"shareable_bits":
		Bitmask of shareable resource with other executing
		entities (e.g. I/O). User can use this when
		setting up exclusive cache partitions. Note that
		some platforms support devices that have their
		own settings for cache use which can over-ride
		these bits.
"bit_usage":
		Annotated capacity bitmasks showing how all
		instances of the resource are used. The legend is:

			"0":
			      Corresponding region is unused. When the system's
			      resources have been allocated and a "0" is found
			      in "bit_usage" it is a sign that resources are
			      wasted.

			"H":
			      Corresponding region is used by hardware only
			      but available for software use. If a resource
			      has bits set in "shareable_bits" but not all
			      of these bits appear in the resource groups'
			      schematas then the bits appearing in
			      "shareable_bits" but no resource group will
			      be marked as "H".
			"X":
			      Corresponding region is available for sharing and
			      used by hardware and software. These are the
			      bits that appear in "shareable_bits" as
			      well as a resource group's allocation.
			"S":
			      Corresponding region is used by software
			      and available for sharing.
			"E":
			      Corresponding region is used exclusively by
			      one resource group. No sharing allowed.
			"P":
			      Corresponding region is pseudo-locked. No
			      sharing allowed.

Memory bandwidth(MB) subdirectory contains the following files
with respect to allocation:

"min_bandwidth":
		The minimum memory bandwidth percentage which
		user can request.

"bandwidth_gran":
		The granularity in which the memory bandwidth
		percentage is allocated. The allocated
		b/w percentage is rounded off to the next
		control step available on the hardware. The
		available bandwidth control steps are:
		min_bandwidth + N * bandwidth_gran.

"delay_linear":
		Indicates if the delay scale is linear or
		non-linear. This field is purely informational
		only.

"thread_throttle_mode":
		Indicator on Intel systems of how tasks running on threads
		of a physical core are throttled in cases where they
		request different memory bandwidth percentages:

		"max":
			the smallest percentage is applied
			to all threads
		"per-thread":
			bandwidth percentages are directly applied to
			the threads running on the core

If RDT monitoring is available there will be an "L3_MON" directory
with the following files:

"num_rmids":
		The number of RMIDs available. This is the
		upper bound for how many "CTRL_MON" + "MON"
		groups can be created.

"mon_features":
		Lists the monitoring events if
		monitoring is enabled for the resource.

"max_threshold_occupancy":
		Read/write file provides the largest value (in
		bytes) at which a previously used LLC_occupancy
		counter can be considered for re-use.

Finally, in the top level of the "info" directory there is a file
named "last_cmd_status". This is reset with every "command" issued
via the file system (making new directories or writing to any of the
control files). If the command was successful, it will read as "ok".
If the command failed, it will provide more information that can be
conveyed in the error returns from file operations. E.g.
::

	# echo L3:0=f7 > schemata
	bash: echo: write error: Invalid argument
	# cat info/last_cmd_status
	mask f7 has non-consecutive 1-bits

Resource alloc and monitor groups
=================================

Resource groups are represented as directories in the resctrl file
system.  The default group is the root directory which, immediately
after mounting, owns all the tasks and cpus in the system and can make
full use of all resources.

On a system with RDT control features additional directories can be
created in the root directory that specify different amounts of each
resource (see "schemata" below). The root and these additional top level
directories are referred to as "CTRL_MON" groups below.

On a system with RDT monitoring the root directory and other top level
directories contain a directory named "mon_groups" in which additional
directories can be created to monitor subsets of tasks in the CTRL_MON
group that is their ancestor. These are called "MON" groups in the rest
of this document.

Removing a directory will move all tasks and cpus owned by the group it
represents to the parent. Removing one of the created CTRL_MON groups
will automatically remove all MON groups below it.

All groups contain the following files:

"tasks":
	Reading this file shows the list of all tasks that belong to
	this group. Writing a task id to the file will add a task to the
	group. If the group is a CTRL_MON group the task is removed from
	whichever previous CTRL_MON group owned the task and also from
	any MON group that owned the task. If the group is a MON group,
	then the task must already belong to the CTRL_MON parent of this
	group. The task is removed from any previous MON group.

"cpus":
	Reading this file shows a bitmask of the logical CPUs owned by
	this group. Writing a mask to this file will add and remove
	CPUs to/from this group. As with the tasks file a hierarchy is
	maintained where MON groups may only include CPUs owned by the
	parent CTRL_MON group.
	When the resource group is in pseudo-locked mode this file will
	only be readable, reflecting the CPUs associated with the
	pseudo-locked region.

"cpus_list":
	Just like "cpus", only using ranges of CPUs instead of bitmasks.

When control is enabled all CTRL_MON groups will also contain:

"schemata":
	A list of all the resources available to this group.
	Each resource has its own line and format - see below for details.

"size":
	Mirrors the display of the "schemata" file to display the size in
	bytes of each allocation instead of the bits representing the
	allocation.

"mode":
	The "mode" of the resource group dictates the sharing of its
	allocations. A "shareable" resource group allows sharing of its
	allocations while an "exclusive" resource group does not. A
	cache pseudo-locked region is created by first writing
	"pseudo-locksetup" to the "mode" file before writing the cache
	pseudo-locked region's schemata to the resource group's "schemata"
	file. On successful pseudo-locked region creation the mode will
	automatically change to "pseudo-locked".

When monitoring is enabled all MON groups will also contain:

"mon_data":
	This contains a set of files organized by L3 domain and by
	RDT event. E.g. on a system with two L3 domains there will
	be subdirectories "mon_L3_00" and "mon_L3_01".	Each of these
	directories have one file per event (e.g. "llc_occupancy",
	"mbm_total_bytes", and "mbm_local_bytes"). In a MON group these
	files provide a read out of the current value of the event for
	all tasks in the group. In CTRL_MON groups these files provide
	the sum for all tasks in the CTRL_MON group and all tasks in
	MON groups. Please see example section for more details on usage.

Resource allocation rules
-------------------------

When a task is running the following rules define which resources are
available to it:

1) If the task is a member of a non-default group, then the schemata
   for that group is used.

2) Else if the task belongs to the default group, but is running on a
   CPU that is assigned to some specific group, then the schemata for the
   CPU's group is used.

3) Otherwise the schemata for the default group is used.

Resource monitoring rules
-------------------------
1) If a task is a member of a MON group, or non-default CTRL_MON group
   then RDT events for the task will be reported in that group.

2) If a task is a member of the default CTRL_MON group, but is running
   on a CPU that is assigned to some specific group, then the RDT events
   for the task will be reported in that group.

3) Otherwise RDT events for the task will be reported in the root level
   "mon_data" group.

Notes on cache occupancy monitoring and control
===============================================
When moving a task from one group to another you should remember that
this only affects *new* cache allocations by the task. E.g. you may have
a task in a monitor group showing 3 MB of cache occupancy. If you move
to a new group and immediately check the occupancy of the old and new
groups you will likely see that the old group is still showing 3 MB and
the new group zero. When the task accesses locations still in cache from
before the move, the h/w does not update any counters. On a busy system
you will likely see the occupancy in the old group go down as cache lines
are evicted and re-used while the occupancy in the new group rises as
the task accesses memory and loads into the cache are counted based on
membership in the new group.

The same applies to cache allocation control. Moving a task to a group
with a smaller cache partition will not evict any cache lines. The
process may continue to use them from the old partition.

Hardware uses CLOSid(Class of service ID) and an RMID(Resource monitoring ID)
to identify a control group and a monitoring group respectively. Each of
the resource groups are mapped to these IDs based on the kind of group. The
number of CLOSid and RMID are limited by the hardware and hence the creation of
a "CTRL_MON" directory may fail if we run out of either CLOSID or RMID
and creation of "MON" group may fail if we run out of RMIDs.

max_threshold_occupancy - generic concepts
------------------------------------------

Note that an RMID once freed may not be immediately available for use as
the RMID is still tagged the cache lines of the previous user of RMID.
Hence such RMIDs are placed on limbo list and checked back if the cache
occupancy has gone down. If there is a time when system has a lot of
limbo RMIDs but which are not ready to be used, user may see an -EBUSY
during mkdir.

max_threshold_occupancy is a user configurable value to determine the
occupancy at which an RMID can be freed.

Schemata files - general concepts
---------------------------------
Each line in the file describes one resource. The line starts with
the name of the resource, followed by specific values to be applied
in each of the instances of that resource on the system.

Cache IDs
---------
On current generation systems there is one L3 cache per socket and L2
caches are generally just shared by the hyperthreads on a core, but this
isn't an architectural requirement. We could have multiple separate L3
caches on a socket, multiple cores could share an L2 cache. So instead
of using "socket" or "core" to define the set of logical cpus sharing
a resource we use a "Cache ID". At a given cache level this will be a
unique number across the whole system (but it isn't guaranteed to be a
contiguous sequence, there may be gaps).  To find the ID for each logical
CPU look in /sys/devices/system/cpu/cpu*/cache/index*/id

Cache Bit Masks (CBM)
---------------------
For cache resources we describe the portion of the cache that is available
for allocation using a bitmask. The maximum value of the mask is defined
by each cpu model (and may be different for different cache levels). It
is found using CPUID, but is also provided in the "info" directory of
the resctrl file system in "info/{resource}/cbm_mask". Intel hardware
requires that these masks have all the '1' bits in a contiguous block. So
0x3, 0x6 and 0xC are legal 4-bit masks with two bits set, but 0x5, 0x9
and 0xA are not.  On a system with a 20-bit mask each bit represents 5%
of the capacity of the cache. You could partition the cache into four
equal parts with masks: 0x1f, 0x3e0, 0x7c00, 0xf8000.

Memory bandwidth Allocation and monitoring
==========================================

For Memory bandwidth resource, by default the user controls the resource
by indicating the percentage of total memory bandwidth.

The minimum bandwidth percentage value for each cpu model is predefined
and can be looked up through "info/MB/min_bandwidth". The bandwidth
granularity that is allocated is also dependent on the cpu model and can
be looked up at "info/MB/bandwidth_gran". The available bandwidth
control steps are: min_bw + N * bw_gran. Intermediate values are rounded
to the next control step available on the hardware.

The bandwidth throttling is a core specific mechanism on some of Intel
SKUs. Using a high bandwidth and a low bandwidth setting on two threads
sharing a core may result in both threads being throttled to use the
low bandwidth (see "thread_throttle_mode").

The fact that Memory bandwidth allocation(MBA) may be a core
specific mechanism where as memory bandwidth monitoring(MBM) is done at
the package level may lead to confusion when users try to apply control
via the MBA and then monitor the bandwidth to see if the controls are
effective. Below are such scenarios:

1. User may *not* see increase in actual bandwidth when percentage
   values are increased:

This can occur when aggregate L2 external bandwidth is more than L3
external bandwidth. Consider an SKL SKU with 24 cores on a package and
where L2 external  is 10GBps (hence aggregate L2 external bandwidth is
240GBps) and L3 external bandwidth is 100GBps. Now a workload with '20
threads, having 50% bandwidth, each consuming 5GBps' consumes the max L3
bandwidth of 100GBps although the percentage value specified is only 50%
<< 100%. Hence increasing the bandwidth percentage will not yield any
more bandwidth. This is because although the L2 external bandwidth still
has capacity, the L3 external bandwidth is fully used. Also note that
this would be dependent on number of cores the benchmark is run on.

2. Same bandwidth percentage may mean different actual bandwidth
   depending on # of threads:

For the same SKU in #1, a 'single thread, with 10% bandwidth' and '4
thread, with 10% bandwidth' can consume upto 10GBps and 40GBps although
they have same percentage bandwidth of 10%. This is simply because as
threads start using more cores in an rdtgroup, the actual bandwidth may
increase or vary although user specified bandwidth percentage is same.

In order to mitigate this and make the interface more user friendly,
resctrl added support for specifying the bandwidth in MBps as well.  The
kernel underneath would use a software feedback mechanism or a "Software
Controller(mba_sc)" which reads the actual bandwidth using MBM counters
and adjust the memory bandwidth percentages to ensure::

	"actual bandwidth < user specified bandwidth".

By default, the schemata would take the bandwidth percentage values
where as user can switch to the "MBA software controller" mode using
a mount option 'mba_MBps'. The schemata format is specified in the below
sections.

L3 schemata file details (code and data prioritization disabled)
----------------------------------------------------------------
With CDP disabled the L3 schemata format is::

	L3:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...

L3 schemata file details (CDP enabled via mount option to resctrl)
------------------------------------------------------------------
When CDP is enabled L3 control is split into two separate resources
so you can specify independent masks for code and data like this::

	L3DATA:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
	L3CODE:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...

L2 schemata file details
------------------------
CDP is supported at L2 using the 'cdpl2' mount option. The schemata
format is either::

	L2:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...

or

	L2DATA:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
	L2CODE:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...

Memory bandwidth Allocation (default mode)
------------------------------------------

Memory b/w domain is L3 cache.
::

	MB:<cache_id0>=bandwidth0;<cache_id1>=bandwidth1;...

Memory bandwidth Allocation specified in MBps
---------------------------------------------

Memory bandwidth domain is L3 cache.
::

	MB:<cache_id0>=bw_MBps0;<cache_id1>=bw_MBps1;...

Reading/writing the schemata file
---------------------------------
Reading the schemata file will show the state of all resources
on all domains. When writing you only need to specify those values
which you wish to change.  E.g.
::

  # cat schemata
  L3DATA:0=fffff;1=fffff;2=fffff;3=fffff
  L3CODE:0=fffff;1=fffff;2=fffff;3=fffff
  # echo "L3DATA:2=3c0;" > schemata
  # cat schemata
  L3DATA:0=fffff;1=fffff;2=3c0;3=fffff
  L3CODE:0=fffff;1=fffff;2=fffff;3=fffff

Cache Pseudo-Locking
====================
CAT enables a user to specify the amount of cache space that an
application can fill. Cache pseudo-locking builds on the fact that a
CPU can still read and write data pre-allocated outside its current
allocated area on a cache hit. With cache pseudo-locking, data can be
preloaded into a reserved portion of cache that no application can
fill, and from that point on will only serve cache hits. The cache
pseudo-locked memory is made accessible to user space where an
application can map it into its virtual address space and thus have
a region of memory with reduced average read latency.

The creation of a cache pseudo-locked region is triggered by a request
from the user to do so that is accompanied by a schemata of the region
to be pseudo-locked. The cache pseudo-locked region is created as follows:

- Create a CAT allocation CLOSNEW with a CBM matching the schemata
  from the user of the cache region that will contain the pseudo-locked
  memory. This region must not overlap with any current CAT allocation/CLOS
  on the system and no future overlap with this cache region is allowed
  while the pseudo-locked region exists.
- Create a contiguous region of memory of the same size as the cache
  region.
- Flush the cache, disable hardware prefetchers, disable preemption.
- Make CLOSNEW the active CLOS and touch the allocated memory to load
  it into the cache.
- Set the previous CLOS as active.
- At this point the closid CLOSNEW can be released - the cache
  pseudo-locked region is protected as long as its CBM does not appear in
  any CAT allocation. Even though the cache pseudo-locked region will from
  this point on not appear in any CBM of any CLOS an application running with
  any CLOS will be able to access the memory in the pseudo-locked region since
  the region continues to serve cache hits.
- The contiguous region of memory loaded into the cache is exposed to
  user-space as a character device.

Cache pseudo-locking increases the probability that data will remain
in the cache via carefully configuring the CAT feature and controlling
application behavior. There is no guarantee that data is placed in
cache. Instructions like INVD, WBINVD, CLFLUSH, etc. can still evict
“locked” data from cache. Power management C-states may shrink or
power off cache. Deeper C-states will automatically be restricted on
pseudo-locked region creation.

It is required that an application using a pseudo-locked region runs
with affinity to the cores (or a subset of the cores) associated
with the cache on which the pseudo-locked region resides. A sanity check
within the code will not allow an application to map pseudo-locked memory
unless it runs with affinity to cores associated with the cache on which the
pseudo-locked region resides. The sanity check is only done during the
initial mmap() handling, there is no enforcement afterwards and the
application self needs to ensure it remains affine to the correct cores.

Pseudo-locking is accomplished in two stages:

1) During the first stage the system administrator allocates a portion
   of cache that should be dedicated to pseudo-locking. At this time an
   equivalent portion of memory is allocated, loaded into allocated
   cache portion, and exposed as a character device.
2) During the second stage a user-space application maps (mmap()) the
   pseudo-locked memory into its address space.

Cache Pseudo-Locking Interface
------------------------------
A pseudo-locked region is created using the resctrl interface as follows:

1) Create a new resource group by creating a new directory in /sys/fs/resctrl.
2) Change the new resource group's mode to "pseudo-locksetup" by writing
   "pseudo-locksetup" to the "mode" file.
3) Write the schemata of the pseudo-locked region to the "schemata" file. All
   bits within the schemata should be "unused" according to the "bit_usage"
   file.

On successful pseudo-locked region creation the "mode" file will contain
"pseudo-locked" and a new character device with the same name as the resource
group will exist in /dev/pseudo_lock. This character device can be mmap()'ed
by user space in order to obtain access to the pseudo-locked memory region.

An example of cache pseudo-locked region creation and usage can be found below.

Cache Pseudo-Locking Debugging Interface
----------------------------------------
The pseudo-locking debugging interface is enabled by default (if
CONFIG_DEBUG_FS is enabled) and can be found in /sys/kernel/debug/resctrl.

There is no explicit way for the kernel to test if a provided memory
location is present in the cache. The pseudo-locking debugging interface uses
the tracing infrastructure to provide two ways to measure cache residency of
the pseudo-locked region:

1) Memory access latency using the pseudo_lock_mem_latency tracepoint. Data
   from these measurements are best visualized using a hist trigger (see
   example below). In this test the pseudo-locked region is traversed at
   a stride of 32 bytes while hardware prefetchers and preemption
   are disabled. This also provides a substitute visualization of cache
   hits and misses.
2) Cache hit and miss measurements using model specific precision counters if
   available. Depending on the levels of cache on the system the pseudo_lock_l2
   and pseudo_lock_l3 tracepoints are available.

When a pseudo-locked region is created a new debugfs directory is created for
it in debugfs as /sys/kernel/debug/resctrl/<newdir>. A single
write-only file, pseudo_lock_measure, is present in this directory. The
measurement of the pseudo-locked region depends on the number written to this
debugfs file:

1:
     writing "1" to the pseudo_lock_measure file will trigger the latency
     measurement captured in the pseudo_lock_mem_latency tracepoint. See
     example below.
2:
     writing "2" to the pseudo_lock_measure file will trigger the L2 cache
     residency (cache hits and misses) measurement captured in the
     pseudo_lock_l2 tracepoint. See example below.
3:
     writing "3" to the pseudo_lock_measure file will trigger the L3 cache
     residency (cache hits and misses) measurement captured in the
     pseudo_lock_l3 tracepoint.

All measurements are recorded with the tracing infrastructure. This requires
the relevant tracepoints to be enabled before the measurement is triggered.

Example of latency debugging interface
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In this example a pseudo-locked region named "newlock" was created. Here is
how we can measure the latency in cycles of reading from this region and
visualize this data with a histogram that is available if CONFIG_HIST_TRIGGERS
is set::

  # :> /sys/kernel/debug/tracing/trace
  # echo 'hist:keys=latency' > /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_mem_latency/trigger
  # echo 1 > /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_mem_latency/enable
  # echo 1 > /sys/kernel/debug/resctrl/newlock/pseudo_lock_measure
  # echo 0 > /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_mem_latency/enable
  # cat /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_mem_latency/hist

  # event histogram
  #
  # trigger info: hist:keys=latency:vals=hitcount:sort=hitcount:size=2048 [active]
  #

  { latency:        456 } hitcount:          1
  { latency:         50 } hitcount:         83
  { latency:         36 } hitcount:         96
  { latency:         44 } hitcount:        174
  { latency:         48 } hitcount:        195
  { latency:         46 } hitcount:        262
  { latency:         42 } hitcount:        693
  { latency:         40 } hitcount:       3204
  { latency:         38 } hitcount:       3484

  Totals:
      Hits: 8192
      Entries: 9
    Dropped: 0

Example of cache hits/misses debugging
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In this example a pseudo-locked region named "newlock" was created on the L2
cache of a platform. Here is how we can obtain details of the cache hits
and misses using the platform's precision counters.
::

  # :> /sys/kernel/debug/tracing/trace
  # echo 1 > /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_l2/enable
  # echo 2 > /sys/kernel/debug/resctrl/newlock/pseudo_lock_measure
  # echo 0 > /sys/kernel/debug/tracing/events/resctrl/pseudo_lock_l2/enable
  # cat /sys/kernel/debug/tracing/trace

  # tracer: nop
  #
  #                              _-----=> irqs-off
  #                             / _----=> need-resched
  #                            | / _---=> hardirq/softirq
  #                            || / _--=> preempt-depth
  #                            ||| /     delay
  #           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
  #              | |       |   ||||       |         |
  pseudo_lock_mea-1672  [002] ....  3132.860500: pseudo_lock_l2: hits=4097 miss=0

Examples for RDT allocation usage
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

1) Example 1

On a two socket machine (one L3 cache per socket) with just four bits
for cache bit masks, minimum b/w of 10% with a memory bandwidth
granularity of 10%.
::

  # mount -t resctrl resctrl /sys/fs/resctrl
  # cd /sys/fs/resctrl
  # mkdir p0 p1
  # echo "L3:0=3;1=c\nMB:0=50;1=50" > /sys/fs/resctrl/p0/schemata
  # echo "L3:0=3;1=3\nMB:0=50;1=50" > /sys/fs/resctrl/p1/schemata

The default resource group is unmodified, so we have access to all parts
of all caches (its schemata file reads "L3:0=f;1=f").

Tasks that are under the control of group "p0" may only allocate from the
"lower" 50% on cache ID 0, and the "upper" 50% of cache ID 1.
Tasks in group "p1" use the "lower" 50% of cache on both sockets.

Similarly, tasks that are under the control of group "p0" may use a
maximum memory b/w of 50% on socket0 and 50% on socket 1.
Tasks in group "p1" may also use 50% memory b/w on both sockets.
Note that unlike cache masks, memory b/w cannot specify whether these
allocations can overlap or not. The allocations specifies the maximum
b/w that the group may be able to use and the system admin can configure
the b/w accordingly.

If resctrl is using the software controller (mba_sc) then user can enter the
max b/w in MB rather than the percentage values.
::

  # echo "L3:0=3;1=c\nMB:0=1024;1=500" > /sys/fs/resctrl/p0/schemata
  # echo "L3:0=3;1=3\nMB:0=1024;1=500" > /sys/fs/resctrl/p1/schemata

In the above example the tasks in "p1" and "p0" on socket 0 would use a max b/w
of 1024MB where as on socket 1 they would use 500MB.

2) Example 2

Again two sockets, but this time with a more realistic 20-bit mask.

Two real time tasks pid=1234 running on processor 0 and pid=5678 running on
processor 1 on socket 0 on a 2-socket and dual core machine. To avoid noisy
neighbors, each of the two real-time tasks exclusively occupies one quarter
of L3 cache on socket 0.
::

  # mount -t resctrl resctrl /sys/fs/resctrl
  # cd /sys/fs/resctrl

First we reset the schemata for the default group so that the "upper"
50% of the L3 cache on socket 0 and 50% of memory b/w cannot be used by
ordinary tasks::

  # echo "L3:0=3ff;1=fffff\nMB:0=50;1=100" > schemata

Next we make a resource group for our first real time task and give
it access to the "top" 25% of the cache on socket 0.
::

  # mkdir p0
  # echo "L3:0=f8000;1=fffff" > p0/schemata

Finally we move our first real time task into this resource group. We
also use taskset(1) to ensure the task always runs on a dedicated CPU
on socket 0. Most uses of resource groups will also constrain which
processors tasks run on.
::

  # echo 1234 > p0/tasks
  # taskset -cp 1 1234

Ditto for the second real time task (with the remaining 25% of cache)::

  # mkdir p1
  # echo "L3:0=7c00;1=fffff" > p1/schemata
  # echo 5678 > p1/tasks
  # taskset -cp 2 5678

For the same 2 socket system with memory b/w resource and CAT L3 the
schemata would look like(Assume min_bandwidth 10 and bandwidth_gran is
10):

For our first real time task this would request 20% memory b/w on socket 0.
::

  # echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata

For our second real time task this would request an other 20% memory b/w
on socket 0.
::

  # echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata

3) Example 3

A single socket system which has real-time tasks running on core 4-7 and
non real-time workload assigned to core 0-3. The real-time tasks share text
and data, so a per task association is not required and due to interaction
with the kernel it's desired that the kernel on these cores shares L3 with
the tasks.
::

  # mount -t resctrl resctrl /sys/fs/resctrl
  # cd /sys/fs/resctrl

First we reset the schemata for the default group so that the "upper"
50% of the L3 cache on socket 0, and 50% of memory bandwidth on socket 0
cannot be used by ordinary tasks::

  # echo "L3:0=3ff\nMB:0=50" > schemata

Next we make a resource group for our real time cores and give it access
to the "top" 50% of the cache on socket 0 and 50% of memory bandwidth on
socket 0.
::

  # mkdir p0
  # echo "L3:0=ffc00\nMB:0=50" > p0/schemata

Finally we move core 4-7 over to the new group and make sure that the
kernel and the tasks running there get 50% of the cache. They should
also get 50% of memory bandwidth assuming that the cores 4-7 are SMT
siblings and only the real time threads are scheduled on the cores 4-7.
::

  # echo F0 > p0/cpus

4) Example 4

The resource groups in previous examples were all in the default "shareable"
mode allowing sharing of their cache allocations. If one resource group
configures a cache allocation then nothing prevents another resource group
to overlap with that allocation.

In this example a new exclusive resource group will be created on a L2 CAT
system with two L2 cache instances that can be configured with an 8-bit
capacity bitmask. The new exclusive resource group will be configured to use
25% of each cache instance.
::

  # mount -t resctrl resctrl /sys/fs/resctrl/
  # cd /sys/fs/resctrl

First, we observe that the default group is configured to allocate to all L2
cache::

  # cat schemata
  L2:0=ff;1=ff

We could attempt to create the new resource group at this point, but it will
fail because of the overlap with the schemata of the default group::

  # mkdir p0
  # echo 'L2:0=0x3;1=0x3' > p0/schemata
  # cat p0/mode
  shareable
  # echo exclusive > p0/mode
  -sh: echo: write error: Invalid argument
  # cat info/last_cmd_status
  schemata overlaps

To ensure that there is no overlap with another resource group the default
resource group's schemata has to change, making it possible for the new
resource group to become exclusive.
::

  # echo 'L2:0=0xfc;1=0xfc' > schemata
  # echo exclusive > p0/mode
  # grep . p0/*
  p0/cpus:0
  p0/mode:exclusive
  p0/schemata:L2:0=03;1=03
  p0/size:L2:0=262144;1=262144

A new resource group will on creation not overlap with an exclusive resource
group::

  # mkdir p1
  # grep . p1/*
  p1/cpus:0
  p1/mode:shareable
  p1/schemata:L2:0=fc;1=fc
  p1/size:L2:0=786432;1=786432

The bit_usage will reflect how the cache is used::

  # cat info/L2/bit_usage
  0=SSSSSSEE;1=SSSSSSEE

A resource group cannot be forced to overlap with an exclusive resource group::

  # echo 'L2:0=0x1;1=0x1' > p1/schemata
  -sh: echo: write error: Invalid argument
  # cat info/last_cmd_status
  overlaps with exclusive group

Example of Cache Pseudo-Locking
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Lock portion of L2 cache from cache id 1 using CBM 0x3. Pseudo-locked
region is exposed at /dev/pseudo_lock/newlock that can be provided to
application for argument to mmap().
::

  # mount -t resctrl resctrl /sys/fs/resctrl/
  # cd /sys/fs/resctrl

Ensure that there are bits available that can be pseudo-locked, since only
unused bits can be pseudo-locked the bits to be pseudo-locked needs to be
removed from the default resource group's schemata::

  # cat info/L2/bit_usage
  0=SSSSSSSS;1=SSSSSSSS
  # echo 'L2:1=0xfc' > schemata
  # cat info/L2/bit_usage
  0=SSSSSSSS;1=SSSSSS00

Create a new resource group that will be associated with the pseudo-locked
region, indicate that it will be used for a pseudo-locked region, and
configure the requested pseudo-locked region capacity bitmask::

  # mkdir newlock
  # echo pseudo-locksetup > newlock/mode
  # echo 'L2:1=0x3' > newlock/schemata

On success the resource group's mode will change to pseudo-locked, the
bit_usage will reflect the pseudo-locked region, and the character device
exposing the pseudo-locked region will exist::

  # cat newlock/mode
  pseudo-locked
  # cat info/L2/bit_usage
  0=SSSSSSSS;1=SSSSSSPP
  # ls -l /dev/pseudo_lock/newlock
  crw------- 1 root root 243, 0 Apr  3 05:01 /dev/pseudo_lock/newlock

::

  /*
  * Example code to access one page of pseudo-locked cache region
  * from user space.
  */
  #define _GNU_SOURCE
  #include <fcntl.h>
  #include <sched.h>
  #include <stdio.h>
  #include <stdlib.h>
  #include <unistd.h>
  #include <sys/mman.h>

  /*
  * It is required that the application runs with affinity to only
  * cores associated with the pseudo-locked region. Here the cpu
  * is hardcoded for convenience of example.
  */
  static int cpuid = 2;

  int main(int argc, char *argv[])
  {
    cpu_set_t cpuset;
    long page_size;
    void *mapping;
    int dev_fd;
    int ret;

    page_size = sysconf(_SC_PAGESIZE);

    CPU_ZERO(&cpuset);
    CPU_SET(cpuid, &cpuset);
    ret = sched_setaffinity(0, sizeof(cpuset), &cpuset);
    if (ret < 0) {
      perror("sched_setaffinity");
      exit(EXIT_FAILURE);
    }

    dev_fd = open("/dev/pseudo_lock/newlock", O_RDWR);
    if (dev_fd < 0) {
      perror("open");
      exit(EXIT_FAILURE);
    }

    mapping = mmap(0, page_size, PROT_READ | PROT_WRITE, MAP_SHARED,
            dev_fd, 0);
    if (mapping == MAP_FAILED) {
      perror("mmap");
      close(dev_fd);
      exit(EXIT_FAILURE);
    }

    /* Application interacts with pseudo-locked memory @mapping */

    ret = munmap(mapping, page_size);
    if (ret < 0) {
      perror("munmap");
      close(dev_fd);
      exit(EXIT_FAILURE);
    }

    close(dev_fd);
    exit(EXIT_SUCCESS);
  }

Locking between applications
----------------------------

Certain operations on the resctrl filesystem, composed of read/writes
to/from multiple files, must be atomic.

As an example, the allocation of an exclusive reservation of L3 cache
involves:

  1. Read the cbmmasks from each directory or the per-resource "bit_usage"
  2. Find a contiguous set of bits in the global CBM bitmask that is clear
     in any of the directory cbmmasks
  3. Create a new directory
  4. Set the bits found in step 2 to the new directory "schemata" file

If two applications attempt to allocate space concurrently then they can
end up allocating the same bits so the reservations are shared instead of
exclusive.

To coordinate atomic operations on the resctrlfs and to avoid the problem
above, the following locking procedure is recommended:

Locking is based on flock, which is available in libc and also as a shell
script command

Write lock:

 A) Take flock(LOCK_EX) on /sys/fs/resctrl
 B) Read/write the directory structure.
 C) funlock

Read lock:

 A) Take flock(LOCK_SH) on /sys/fs/resctrl
 B) If success read the directory structure.
 C) funlock

Example with bash::

  # Atomically read directory structure
  $ flock -s /sys/fs/resctrl/ find /sys/fs/resctrl

  # Read directory contents and create new subdirectory

  $ cat create-dir.sh
  find /sys/fs/resctrl/ > output.txt
  mask = function-of(output.txt)
  mkdir /sys/fs/resctrl/newres/
  echo mask > /sys/fs/resctrl/newres/schemata

  $ flock /sys/fs/resctrl/ ./create-dir.sh

Example with C::

  /*
  * Example code do take advisory locks
  * before accessing resctrl filesystem
  */
  #include <sys/file.h>
  #include <stdlib.h>

  void resctrl_take_shared_lock(int fd)
  {
    int ret;

    /* take shared lock on resctrl filesystem */
    ret = flock(fd, LOCK_SH);
    if (ret) {
      perror("flock");
      exit(-1);
    }
  }

  void resctrl_take_exclusive_lock(int fd)
  {
    int ret;

    /* release lock on resctrl filesystem */
    ret = flock(fd, LOCK_EX);
    if (ret) {
      perror("flock");
      exit(-1);
    }
  }

  void resctrl_release_lock(int fd)
  {
    int ret;

    /* take shared lock on resctrl filesystem */
    ret = flock(fd, LOCK_UN);
    if (ret) {
      perror("flock");
      exit(-1);
    }
  }

  void main(void)
  {
    int fd, ret;

    fd = open("/sys/fs/resctrl", O_DIRECTORY);
    if (fd == -1) {
      perror("open");
      exit(-1);
    }
    resctrl_take_shared_lock(fd);
    /* code to read directory contents */
    resctrl_release_lock(fd);

    resctrl_take_exclusive_lock(fd);
    /* code to read and write directory contents */
    resctrl_release_lock(fd);
  }

Examples for RDT Monitoring along with allocation usage
=======================================================
Reading monitored data
----------------------
Reading an event file (for ex: mon_data/mon_L3_00/llc_occupancy) would
show the current snapshot of LLC occupancy of the corresponding MON
group or CTRL_MON group.

Example 1 (Monitor CTRL_MON group and subset of tasks in CTRL_MON group)
------------------------------------------------------------------------
On a two socket machine (one L3 cache per socket) with just four bits
for cache bit masks::

  # mount -t resctrl resctrl /sys/fs/resctrl
  # cd /sys/fs/resctrl
  # mkdir p0 p1
  # echo "L3:0=3;1=c" > /sys/fs/resctrl/p0/schemata
  # echo "L3:0=3;1=3" > /sys/fs/resctrl/p1/schemata
  # echo 5678 > p1/tasks
  # echo 5679 > p1/tasks

The default resource group is unmodified, so we have access to all parts
of all caches (its schemata file reads "L3:0=f;1=f").

Tasks that are under the control of group "p0" may only allocate from the
"lower" 50% on cache ID 0, and the "upper" 50% of cache ID 1.
Tasks in group "p1" use the "lower" 50% of cache on both sockets.

Create monitor groups and assign a subset of tasks to each monitor group.
::

  # cd /sys/fs/resctrl/p1/mon_groups
  # mkdir m11 m12
  # echo 5678 > m11/tasks
  # echo 5679 > m12/tasks

fetch data (data shown in bytes)
::

  # cat m11/mon_data/mon_L3_00/llc_occupancy
  16234000
  # cat m11/mon_data/mon_L3_01/llc_occupancy
  14789000
  # cat m12/mon_data/mon_L3_00/llc_occupancy
  16789000

The parent ctrl_mon group shows the aggregated data.
::

  # cat /sys/fs/resctrl/p1/mon_data/mon_l3_00/llc_occupancy
  31234000

Example 2 (Monitor a task from its creation)
--------------------------------------------
On a two socket machine (one L3 cache per socket)::

  # mount -t resctrl resctrl /sys/fs/resctrl
  # cd /sys/fs/resctrl
  # mkdir p0 p1

An RMID is allocated to the group once its created and hence the <cmd>
below is monitored from its creation.
::

  # echo $$ > /sys/fs/resctrl/p1/tasks
  # <cmd>

Fetch the data::

  # cat /sys/fs/resctrl/p1/mon_data/mon_l3_00/llc_occupancy
  31789000

Example 3 (Monitor without CAT support or before creating CAT groups)
---------------------------------------------------------------------

Assume a system like HSW has only CQM and no CAT support. In this case
the resctrl will still mount but cannot create CTRL_MON directories.
But user can create different MON groups within the root group thereby
able to monitor all tasks including kernel threads.

This can also be used to profile jobs cache size footprint before being
able to allocate them to different allocation groups.
::

  # mount -t resctrl resctrl /sys/fs/resctrl
  # cd /sys/fs/resctrl
  # mkdir mon_groups/m01
  # mkdir mon_groups/m02

  # echo 3478 > /sys/fs/resctrl/mon_groups/m01/tasks
  # echo 2467 > /sys/fs/resctrl/mon_groups/m02/tasks

Monitor the groups separately and also get per domain data. From the
below its apparent that the tasks are mostly doing work on
domain(socket) 0.
::

  # cat /sys/fs/resctrl/mon_groups/m01/mon_L3_00/llc_occupancy
  31234000
  # cat /sys/fs/resctrl/mon_groups/m01/mon_L3_01/llc_occupancy
  34555
  # cat /sys/fs/resctrl/mon_groups/m02/mon_L3_00/llc_occupancy
  31234000
  # cat /sys/fs/resctrl/mon_groups/m02/mon_L3_01/llc_occupancy
  32789

Example 4 (Monitor real time tasks)
-----------------------------------

A single socket system which has real time tasks running on cores 4-7
and non real time tasks on other cpus. We want to monitor the cache
occupancy of the real time threads on these cores.
::

  # mount -t resctrl resctrl /sys/fs/resctrl
  # cd /sys/fs/resctrl
  # mkdir p1

Move the cpus 4-7 over to p1::

  # echo f0 > p1/cpus

View the llc occupancy snapshot::

  # cat /sys/fs/resctrl/p1/mon_data/mon_L3_00/llc_occupancy
  11234000

Intel RDT Errata
================

Intel MBM Counters May Report System Memory Bandwidth Incorrectly
-----------------------------------------------------------------

Errata SKX99 for Skylake server and BDF102 for Broadwell server.

Problem: Intel Memory Bandwidth Monitoring (MBM) counters track metrics
according to the assigned Resource Monitor ID (RMID) for that logical
core. The IA32_QM_CTR register (MSR 0xC8E), used to report these
metrics, may report incorrect system bandwidth for certain RMID values.

Implication: Due to the errata, system memory bandwidth may not match
what is reported.

Workaround: MBM total and local readings are corrected according to the
following correction factor table:

+---------------+---------------+---------------+-----------------+
|core count	|rmid count	|rmid threshold	|correction factor|
+---------------+---------------+---------------+-----------------+
|1		|8		|0		|1.000000	  |
+---------------+---------------+---------------+-----------------+
|2		|16		|0		|1.000000	  |
+---------------+---------------+---------------+-----------------+
|3		|24		|15		|0.969650	  |
+---------------+---------------+---------------+-----------------+
|4		|32		|0		|1.000000	  |
+---------------+---------------+---------------+-----------------+
|6		|48		|31		|0.969650	  |
+---------------+---------------+---------------+-----------------+
|7		|56		|47		|1.142857	  |
+---------------+---------------+---------------+-----------------+
|8		|64		|0		|1.000000	  |
+---------------+---------------+---------------+-----------------+
|9		|72		|63		|1.185115	  |
+---------------+---------------+---------------+-----------------+
|10		|80		|63		|1.066553	  |
+---------------+---------------+---------------+-----------------+
|11		|88		|79		|1.454545	  |
+---------------+---------------+---------------+-----------------+
|12		|96		|0		|1.000000	  |
+---------------+---------------+---------------+-----------------+
|13		|104		|95		|1.230769	  |
+---------------+---------------+---------------+-----------------+
|14		|112		|95		|1.142857	  |
+---------------+---------------+---------------+-----------------+
|15		|120		|95		|1.066667	  |
+---------------+---------------+---------------+-----------------+
|16		|128		|0		|1.000000	  |
+---------------+---------------+---------------+-----------------+
|17		|136		|127		|1.254863	  |
+---------------+---------------+---------------+-----------------+
|18		|144		|127		|1.185255	  |
+---------------+---------------+---------------+-----------------+
|19		|152		|0		|1.000000	  |
+---------------+---------------+---------------+-----------------+
|20		|160		|127		|1.066667	  |
+---------------+---------------+---------------+-----------------+
|21		|168		|0		|1.000000	  |
+---------------+---------------+---------------+-----------------+
|22		|176		|159		|1.454334	  |
+---------------+---------------+---------------+-----------------+
|23		|184		|0		|1.000000	  |
+---------------+---------------+---------------+-----------------+
|24		|192		|127		|0.969744	  |
+---------------+---------------+---------------+-----------------+
|25		|200		|191		|1.280246	  |
+---------------+---------------+---------------+-----------------+
|26		|208		|191		|1.230921	  |
+---------------+---------------+---------------+-----------------+
|27		|216		|0		|1.000000	  |
+---------------+---------------+---------------+-----------------+
|28		|224		|191		|1.143118	  |
+---------------+---------------+---------------+-----------------+

If rmid > rmid threshold, MBM total and local values should be multiplied
by the correction factor.

See:

1. Erratum SKX99 in Intel Xeon Processor Scalable Family Specification Update:
http://web.archive.org/web/20200716124958/https://www.intel.com/content/www/us/en/processors/xeon/scalable/xeon-scalable-spec-update.html

2. Erratum BDF102 in Intel Xeon E5-2600 v4 Processor Product Family Specification Update:
http://web.archive.org/web/20191125200531/https://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/xeon-e5-v4-spec-update.pdf

3. The errata in Intel Resource Director Technology (Intel RDT) on 2nd Generation Intel Xeon Scalable Processors Reference Manual:
https://software.intel.com/content/www/us/en/develop/articles/intel-resource-director-technology-rdt-reference-manual.html

for further information.


code:
.. SPDX-License-Identifier: GPL-2.0

===============================
Software Guard eXtensions (SGX)
===============================

Overview
========

Software Guard eXtensions (SGX) hardware enables for user space applications
to set aside private memory regions of code and data:

* Privileged (ring-0) ENCLS functions orchestrate the construction of the
  regions.
* Unprivileged (ring-3) ENCLU functions allow an application to enter and
  execute inside the regions.

These memory regions are called enclaves. An enclave can be only entered at a
fixed set of entry points. Each entry point can hold a single hardware thread
at a time.  While the enclave is loaded from a regular binary file by using
ENCLS functions, only the threads inside the enclave can access its memory. The
region is denied from outside access by the CPU, and encrypted before it leaves
from LLC.

The support can be determined by

	``grep sgx /proc/cpuinfo``

SGX must both be supported in the processor and enabled by the BIOS.  If SGX
appears to be unsupported on a system which has hardware support, ensure
support is enabled in the BIOS.  If a BIOS presents a choice between "Enabled"
and "Software Enabled" modes for SGX, choose "Enabled".

Enclave Page Cache
==================

SGX utilizes an *Enclave Page Cache (EPC)* to store pages that are associated
with an enclave. It is contained in a BIOS-reserved region of physical memory.
Unlike pages used for regular memory, pages can only be accessed from outside of
the enclave during enclave construction with special, limited SGX instructions.

Only a CPU executing inside an enclave can directly access enclave memory.
However, a CPU executing inside an enclave may access normal memory outside the
enclave.

The kernel manages enclave memory similar to how it treats device memory.

Enclave Page Types
------------------

**SGX Enclave Control Structure (SECS)**
   Enclave's address range, attributes and other global data are defined
   by this structure.

**Regular (REG)**
   Regular EPC pages contain the code and data of an enclave.

**Thread Control Structure (TCS)**
   Thread Control Structure pages define the entry points to an enclave and
   track the execution state of an enclave thread.

**Version Array (VA)**
   Version Array pages contain 512 slots, each of which can contain a version
   number for a page evicted from the EPC.

Enclave Page Cache Map
----------------------

The processor tracks EPC pages in a hardware metadata structure called the
*Enclave Page Cache Map (EPCM)*.  The EPCM contains an entry for each EPC page
which describes the owning enclave, access rights and page type among the other
things.

EPCM permissions are separate from the normal page tables.  This prevents the
kernel from, for instance, allowing writes to data which an enclave wishes to
remain read-only.  EPCM permissions may only impose additional restrictions on
top of normal x86 page permissions.

For all intents and purposes, the SGX architecture allows the processor to
invalidate all EPCM entries at will.  This requires that software be prepared to
handle an EPCM fault at any time.  In practice, this can happen on events like
power transitions when the ephemeral key that encrypts enclave memory is lost.

Application interface
=====================

Enclave build functions
-----------------------

In addition to the traditional compiler and linker build process, SGX has a
separate enclave “build” process.  Enclaves must be built before they can be
executed (entered). The first step in building an enclave is opening the
**/dev/sgx_enclave** device.  Since enclave memory is protected from direct
access, special privileged instructions are then used to copy data into enclave
pages and establish enclave page permissions.

.. kernel-doc:: arch/x86/kernel/cpu/sgx/ioctl.c
   :functions: sgx_ioc_enclave_create
               sgx_ioc_enclave_add_pages
               sgx_ioc_enclave_init
               sgx_ioc_enclave_provision

Enclave runtime management
--------------------------

Systems supporting SGX2 additionally support changes to initialized
enclaves: modifying enclave page permissions and type, and dynamically
adding and removing of enclave pages. When an enclave accesses an address
within its address range that does not have a backing page then a new
regular page will be dynamically added to the enclave. The enclave is
still required to run EACCEPT on the new page before it can be used.

.. kernel-doc:: arch/x86/kernel/cpu/sgx/ioctl.c
   :functions: sgx_ioc_enclave_restrict_permissions
               sgx_ioc_enclave_modify_types
               sgx_ioc_enclave_remove_pages

Enclave vDSO
------------

Entering an enclave can only be done through SGX-specific EENTER and ERESUME
functions, and is a non-trivial process.  Because of the complexity of
transitioning to and from an enclave, enclaves typically utilize a library to
handle the actual transitions.  This is roughly analogous to how glibc
implementations are used by most applications to wrap system calls.

Another crucial characteristic of enclaves is that they can generate exceptions
as part of their normal operation that need to be handled in the enclave or are
unique to SGX.

Instead of the traditional signal mechanism to handle these exceptions, SGX
can leverage special exception fixup provided by the vDSO.  The kernel-provided
vDSO function wraps low-level transitions to/from the enclave like EENTER and
ERESUME.  The vDSO function intercepts exceptions that would otherwise generate
a signal and return the fault information directly to its caller.  This avoids
the need to juggle signal handlers.

.. kernel-doc:: arch/x86/include/uapi/asm/sgx.h
   :functions: vdso_sgx_enter_enclave_t

ksgxd
=====

SGX support includes a kernel thread called *ksgxd*.

EPC sanitization
----------------

ksgxd is started when SGX initializes.  Enclave memory is typically ready
for use when the processor powers on or resets.  However, if SGX has been in
use since the reset, enclave pages may be in an inconsistent state.  This might
occur after a crash and kexec() cycle, for instance.  At boot, ksgxd
reinitializes all enclave pages so that they can be allocated and re-used.

The sanitization is done by going through EPC address space and applying the
EREMOVE function to each physical page. Some enclave pages like SECS pages have
hardware dependencies on other pages which prevents EREMOVE from functioning.
Executing two EREMOVE passes removes the dependencies.

Page reclaimer
--------------

Similar to the core kswapd, ksgxd, is responsible for managing the
overcommitment of enclave memory.  If the system runs out of enclave memory,
*ksgxd* “swaps” enclave memory to normal memory.

Launch Control
==============

SGX provides a launch control mechanism. After all enclave pages have been
copied, kernel executes EINIT function, which initializes the enclave. Only after
this the CPU can execute inside the enclave.

EINIT function takes an RSA-3072 signature of the enclave measurement.  The function
checks that the measurement is correct and signature is signed with the key
hashed to the four **IA32_SGXLEPUBKEYHASH{0, 1, 2, 3}** MSRs representing the
SHA256 of a public key.

Those MSRs can be configured by the BIOS to be either readable or writable.
Linux supports only writable configuration in order to give full control to the
kernel on launch control policy. Before calling EINIT function, the driver sets
the MSRs to match the enclave's signing key.

Encryption engines
==================

In order to conceal the enclave data while it is out of the CPU package, the
memory controller has an encryption engine to transparently encrypt and decrypt
enclave memory.

In CPUs prior to Ice Lake, the Memory Encryption Engine (MEE) is used to
encrypt pages leaving the CPU caches. MEE uses a n-ary Merkle tree with root in
SRAM to maintain integrity of the encrypted data. This provides integrity and
anti-replay protection but does not scale to large memory sizes because the time
required to update the Merkle tree grows logarithmically in relation to the
memory size.

CPUs starting from Icelake use Total Memory Encryption (TME) in the place of
MEE. TME-based SGX implementations do not have an integrity Merkle tree, which
means integrity and replay-attacks are not mitigated.  B, it includes
additional changes to prevent cipher text from being returned and SW memory
aliases from being created.

DMA to enclave memory is blocked by range registers on both MEE and TME systems
(SDM section 41.10).

Usage Models
============

Shared Library
--------------

Sensitive data and the code that acts on it is partitioned from the application
into a separate library. The library is then linked as a DSO which can be loaded
into an enclave. The application can then make individual function calls into
the enclave through special SGX instructions. A run-time within the enclave is
configured to marshal function parameters into and out of the enclave and to
call the correct library function.

Application Container
---------------------

An application may be loaded into a container enclave which is specially
configured with a library OS and run-time which permits the application to run.
The enclave run-time and library OS work together to execute the application
when a thread enters the enclave.

Impact of Potential Kernel SGX Bugs
===================================

EPC leaks
---------

When EPC page leaks happen, a WARNING like this is shown in dmesg:

"EREMOVE returned ... and an EPC page was leaked.  SGX may become unusable..."

This is effectively a kernel use-after-free of an EPC page, and due
to the way SGX works, the bug is detected at freeing. Rather than
adding the page back to the pool of available EPC pages, the kernel
intentionally leaks the page to avoid additional errors in the future.

When this happens, the kernel will likely soon leak more EPC pages, and
SGX will likely become unusable because the memory available to SGX is
limited. However, while this may be fatal to SGX, the rest of the kernel
is unlikely to be impacted and should continue to work.

As a result, when this happpens, user should stop running any new
SGX workloads, (or just any new workloads), and migrate all valuable
workloads. Although a machine reboot can recover all EPC memory, the bug
should be reported to Linux developers.

Virtual EPC
===========

The implementation has also a virtual EPC driver to support SGX enclaves
in guests. Unlike the SGX driver, an EPC page allocated by the virtual
EPC driver doesn't have a specific enclave associated with it. This is
because KVM doesn't track how a guest uses EPC pages.

As a result, the SGX core page reclaimer doesn't support reclaiming EPC
pages allocated to KVM guests through the virtual EPC driver. If the
user wants to deploy SGX applications both on the host and in guests
on the same machine, the user should reserve enough EPC (by taking out
total virtual EPC size of all SGX VMs from the physical EPC size) for
host SGX applications so they can run with acceptable performance.

Architectural behavior is to restore all EPC pages to an uninitialized
state also after a guest reboot.  Because this state can be reached only
through the privileged ``ENCLS[EREMOVE]`` instruction, ``/dev/sgx_vepc``
provides the ``SGX_IOC_VEPC_REMOVE_ALL`` ioctl to execute the instruction
on all pages in the virtual EPC.

``EREMOVE`` can fail for three reasons.  Userspace must pay attention
to expected failures and handle them as follows:

1. Page removal will always fail when any thread is running in the
   enclave to which the page belongs.  In this case the ioctl will
   return ``EBUSY`` independent of whether it has successfully removed
   some pages; userspace can avoid these failures by preventing execution
   of any vcpu which maps the virtual EPC.

2. Page removal will cause a general protection fault if two calls to
   ``EREMOVE`` happen concurrently for pages that refer to the same
   "SECS" metadata pages.  This can happen if there are concurrent
   invocations to ``SGX_IOC_VEPC_REMOVE_ALL``, or if a ``/dev/sgx_vepc``
   file descriptor in the guest is closed at the same time as
   ``SGX_IOC_VEPC_REMOVE_ALL``; it will also be reported as ``EBUSY``.
   This can be avoided in userspace by serializing calls to the ioctl()
   and to close(), but in general it should not be a problem.

3. Finally, page removal will fail for SECS metadata pages which still
   have child pages.  Child pages can be removed by executing
   ``SGX_IOC_VEPC_REMOVE_ALL`` on all ``/dev/sgx_vepc`` file descriptors
   mapped into the guest.  This means that the ioctl() must be called
   twice: an initial set of calls to remove child pages and a subsequent
   set of calls to remove SECS pages.  The second set of calls is only
   required for those mappings that returned a nonzero value from the
   first call.  It indicates a bug in the kernel or the userspace client
   if any of the second round of ``SGX_IOC_VEPC_REMOVE_ALL`` calls has
   a return code other than 0.

repo_name: angery/linux


code:
.. SPDX-License-Identifier: GPL-2.0

===========================================
Shared Virtual Addressing (SVA) with ENQCMD
===========================================

Background
==========

Shared Virtual Addressing (SVA) allows the processor and device to use the
same virtual addresses avoiding the need for software to translate virtual
addresses to physical addresses. SVA is what PCIe calls Shared Virtual
Memory (SVM).

In addition to the convenience of using application virtual addresses
by the device, it also doesn't require pinning pages for DMA.
PCIe Address Translation Services (ATS) along with Page Request Interface
(PRI) allow devices to function much the same way as the CPU handling
application page-faults. For more information please refer to the PCIe
specification Chapter 10: ATS Specification.

Use of SVA requires IOMMU support in the platform. IOMMU is also
required to support the PCIe features ATS and PRI. ATS allows devices
to cache translations for virtual addresses. The IOMMU driver uses the
mmu_notifier() support to keep the device TLB cache and the CPU cache in
sync. When an ATS lookup fails for a virtual address, the device should
use the PRI in order to request the virtual address to be paged into the
CPU page tables. The device must use ATS again in order the fetch the
translation before use.

Shared Hardware Workqueues
==========================

Unlike Single Root I/O Virtualization (SR-IOV), Scalable IOV (SIOV) permits
the use of Shared Work Queues (SWQ) by both applications and Virtual
Machines (VM's). This allows better hardware utilization vs. hard
partitioning resources that could result in under utilization. In order to
allow the hardware to distinguish the context for which work is being
executed in the hardware by SWQ interface, SIOV uses Process Address Space
ID (PASID), which is a 20-bit number defined by the PCIe SIG.

PASID value is encoded in all transactions from the device. This allows the
IOMMU to track I/O on a per-PASID granularity in addition to using the PCIe
Resource Identifier (RID) which is the Bus/Device/Function.

ENQCMD
======

ENQCMD is a new instruction on Intel platforms that atomically submits a
work descriptor to a device. The descriptor includes the operation to be
performed, virtual addresses of all parameters, virtual address of a completion
record, and the PASID (process address space ID) of the current process.

ENQCMD works with non-posted semantics and carries a status back if the
command was accepted by hardware. This allows the submitter to know if the
submission needs to be retried or other device specific mechanisms to
implement fairness or ensure forward progress should be provided.

ENQCMD is the glue that ensures applications can directly submit commands
to the hardware and also permits hardware to be aware of application context
to perform I/O operations via use of PASID.

Process Address Space Tagging
=============================

A new thread-scoped MSR (IA32_PASID) provides the connection between
user processes and the rest of the hardware. When an application first
accesses an SVA-capable device, this MSR is initialized with a newly
allocated PASID. The driver for the device calls an IOMMU-specific API
that sets up the routing for DMA and page-requests.

For example, the Intel Data Streaming Accelerator (DSA) uses
iommu_sva_bind_device(), which will do the following:

- Allocate the PASID, and program the process page-table (%cr3 register) in the
  PASID context entries.
- Register for mmu_notifier() to track any page-table invalidations to keep
  the device TLB in sync. For example, when a page-table entry is invalidated,
  the IOMMU propagates the invalidation to the device TLB. This will force any
  future access by the device to this virtual address to participate in
  ATS. If the IOMMU responds with proper response that a page is not
  present, the device would request the page to be paged in via the PCIe PRI
  protocol before performing I/O.

This MSR is managed with the XSAVE feature set as "supervisor state" to
ensure the MSR is updated during context switch.

PASID Management
================

The kernel must allocate a PASID on behalf of each process which will use
ENQCMD and program it into the new MSR to communicate the process identity to
platform hardware.  ENQCMD uses the PASID stored in this MSR to tag requests
from this process.  When a user submits a work descriptor to a device using the
ENQCMD instruction, the PASID field in the descriptor is auto-filled with the
value from MSR_IA32_PASID. Requests for DMA from the device are also tagged
with the same PASID. The platform IOMMU uses the PASID in the transaction to
perform address translation. The IOMMU APIs setup the corresponding PASID
entry in IOMMU with the process address used by the CPU (e.g. %cr3 register in
x86).

The MSR must be configured on each logical CPU before any application
thread can interact with a device. Threads that belong to the same
process share the same page tables, thus the same MSR value.

PASID Life Cycle Management
===========================

PASID is initialized as INVALID_IOASID (-1) when a process is created.

Only processes that access SVA-capable devices need to have a PASID
allocated. This allocation happens when a process opens/binds an SVA-capable
device but finds no PASID for this process. Subsequent binds of the same, or
other devices will share the same PASID.

Although the PASID is allocated to the process by opening a device,
it is not active in any of the threads of that process. It's loaded to the
IA32_PASID MSR lazily when a thread tries to submit a work descriptor
to a device using the ENQCMD.

That first access will trigger a #GP fault because the IA32_PASID MSR
has not been initialized with the PASID value assigned to the process
when the device was opened. The Linux #GP handler notes that a PASID has
been allocated for the process, and so initializes the IA32_PASID MSR
and returns so that the ENQCMD instruction is re-executed.

On fork(2) or exec(2) the PASID is removed from the process as it no
longer has the same address space that it had when the device was opened.

On clone(2) the new task shares the same address space, so will be
able to use the PASID allocated to the process. The IA32_PASID is not
preemptively initialized as the PASID value might not be allocated yet or
the kernel does not know whether this thread is going to access the device
and the cleared IA32_PASID MSR reduces context switch overhead by xstate
init optimization. Since #GP faults have to be handled on any threads that
were created before the PASID was assigned to the mm of the process, newly
created threads might as well be treated in a consistent way.

Due to complexity of freeing the PASID and clearing all IA32_PASID MSRs in
all threads in unbind, free the PASID lazily only on mm exit.

If a process does a close(2) of the device file descriptor and munmap(2)
of the device MMIO portal, then the driver will unbind the device. The
PASID is still marked VALID in the PASID_MSR for any threads in the
process that accessed the device. But this is harmless as without the
MMIO portal they cannot submit new work to the device.

Relationships
=============

 * Each process has many threads, but only one PASID.
 * Devices have a limited number (~10's to 1000's) of hardware workqueues.
   The device driver manages allocating hardware workqueues.
 * A single mmap() maps a single hardware workqueue as a "portal" and
   each portal maps down to a single workqueue.
 * For each device with which a process interacts, there must be
   one or more mmap()'d portals.
 * Many threads within a process can share a single portal to access
   a single device.
 * Multiple processes can separately mmap() the same portal, in
   which case they still share one device hardware workqueue.
 * The single process-wide PASID is used by all threads to interact
   with all devices.  There is not, for instance, a PASID for each
   thread or each thread<->device pair.

FAQ
===

* What is SVA/SVM?

Shared Virtual Addressing (SVA) permits I/O hardware and the processor to
work in the same address space, i.e., to share it. Some call it Shared
Virtual Memory (SVM), but Linux community wanted to avoid confusing it with
POSIX Shared Memory and Secure Virtual Machines which were terms already in
circulation.

* What is a PASID?

A Process Address Space ID (PASID) is a PCIe-defined Transaction Layer Packet
(TLP) prefix. A PASID is a 20-bit number allocated and managed by the OS.
PASID is included in all transactions between the platform and the device.

* How are shared workqueues different?

Traditionally, in order for userspace applications to interact with hardware,
there is a separate hardware instance required per process. For example,
consider doorbells as a mechanism of informing hardware about work to process.
Each doorbell is required to be spaced 4k (or page-size) apart for process
isolation. This requires hardware to provision that space and reserve it in
MMIO. This doesn't scale as the number of threads becomes quite large. The
hardware also manages the queue depth for Shared Work Queues (SWQ), and
consumers don't need to track queue depth. If there is no space to accept
a command, the device will return an error indicating retry.

A user should check Deferrable Memory Write (DMWr) capability on the device
and only submits ENQCMD when the device supports it. In the new DMWr PCIe
terminology, devices need to support DMWr completer capability. In addition,
it requires all switch ports to support DMWr routing and must be enabled by
the PCIe subsystem, much like how PCIe atomic operations are managed for
instance.

SWQ allows hardware to provision just a single address in the device. When
used with ENQCMD to submit work, the device can distinguish the process
submitting the work since it will include the PASID assigned to that
process. This helps the device scale to a large number of processes.

* Is this the same as a user space device driver?

Communicating with the device via the shared workqueue is much simpler
than a full blown user space driver. The kernel driver does all the
initialization of the hardware. User space only needs to worry about
submitting work and processing completions.

* Is this the same as SR-IOV?

Single Root I/O Virtualization (SR-IOV) focuses on providing independent
hardware interfaces for virtualizing hardware. Hence, it's required to be
almost fully functional interface to software supporting the traditional
BARs, space for interrupts via MSI-X, its own register layout.
Virtual Functions (VFs) are assisted by the Physical Function (PF)
driver.

Scalable I/O Virtualization builds on the PASID concept to create device
instances for virtualization. SIOV requires host software to assist in
creating virtual devices; each virtual device is represented by a PASID
along with the bus/device/function of the device.  This allows device
hardware to optimize device resource creation and can grow dynamically on
demand. SR-IOV creation and management is very static in nature. Consult
references below for more details.

* Why not just create a virtual function for each app?

Creating PCIe SR-IOV type Virtual Functions (VF) is expensive. VFs require
duplicated hardware for PCI config space and interrupts such as MSI-X.
Resources such as interrupts have to be hard partitioned between VFs at
creation time, and cannot scale dynamically on demand. The VFs are not
completely independent from the Physical Function (PF). Most VFs require
some communication and assistance from the PF driver. SIOV, in contrast,
creates a software-defined device where all the configuration and control
aspects are mediated via the slow path. The work submission and completion
happen without any mediation.

* Does this support virtualization?

ENQCMD can be used from within a guest VM. In these cases, the VMM helps
with setting up a translation table to translate from Guest PASID to Host
PASID. Please consult the ENQCMD instruction set reference for more
details.

* Does memory need to be pinned?

When devices support SVA along with platform hardware such as IOMMU
supporting such devices, there is no need to pin memory for DMA purposes.
Devices that support SVA also support other PCIe features that remove the
pinning requirement for memory.

Device TLB support - Device requests the IOMMU to lookup an address before
use via Address Translation Service (ATS) requests.  If the mapping exists
but there is no page allocated by the OS, IOMMU hardware returns that no
mapping exists.

Device requests the virtual address to be mapped via Page Request
Interface (PRI). Once the OS has successfully completed the mapping, it
returns the response back to the device. The device requests again for
a translation and continues.

IOMMU works with the OS in managing consistency of page-tables with the
device. When removing pages, it interacts with the device to remove any
device TLB entry that might have been cached before removing the mappings from
the OS.

References
==========

VT-D:
https://01.org/blogs/ashokraj/2018/recent-enhancements-intel-virtualization-technology-directed-i/o-intel-vt-d

SIOV:
https://01.org/blogs/2019/assignable-interfaces-intel-scalable-i/o-virtualization-linux

ENQCMD in ISE:
https://software.intel.com/sites/default/files/managed/c5/15/architecture-instruction-set-extensions-programming-reference.pdf

DSA spec:
https://software.intel.com/sites/default/files/341204-intel-data-streaming-accelerator-spec.pdf


code:
.. SPDX-License-Identifier: GPL-2.0

=====================================
Intel Trust Domain Extensions (TDX)
=====================================

Intel's Trust Domain Extensions (TDX) protect confidential guest VMs from
the host and physical attacks by isolating the guest register state and by
encrypting the guest memory. In TDX, a special module running in a special
mode sits between the host and the guest and manages the guest/host
separation.

Since the host cannot directly access guest registers or memory, much
normal functionality of a hypervisor must be moved into the guest. This is
implemented using a Virtualization Exception (#VE) that is handled by the
guest kernel. A #VE is handled entirely inside the guest kernel, but some
require the hypervisor to be consulted.

TDX includes new hypercall-like mechanisms for communicating from the
guest to the hypervisor or the TDX module.

New TDX Exceptions
==================

TDX guests behave differently from bare-metal and traditional VMX guests.
In TDX guests, otherwise normal instructions or memory accesses can cause
#VE or #GP exceptions.

Instructions marked with an '*' conditionally cause exceptions.  The
details for these instructions are discussed below.

Instruction-based #VE
---------------------

- Port I/O (INS, OUTS, IN, OUT)
- HLT
- MONITOR, MWAIT
- WBINVD, INVD
- VMCALL
- RDMSR*,WRMSR*
- CPUID*

Instruction-based #GP
---------------------

- All VMX instructions: INVEPT, INVVPID, VMCLEAR, VMFUNC, VMLAUNCH,
  VMPTRLD, VMPTRST, VMREAD, VMRESUME, VMWRITE, VMXOFF, VMXON
- ENCLS, ENCLU
- GETSEC
- RSM
- ENQCMD
- RDMSR*,WRMSR*

RDMSR/WRMSR Behavior
--------------------

MSR access behavior falls into three categories:

- #GP generated
- #VE generated
- "Just works"

In general, the #GP MSRs should not be used in guests.  Their use likely
indicates a bug in the guest.  The guest may try to handle the #GP with a
hypercall but it is unlikely to succeed.

The #VE MSRs are typically able to be handled by the hypervisor.  Guests
can make a hypercall to the hypervisor to handle the #VE.

The "just works" MSRs do not need any special guest handling.  They might
be implemented by directly passing through the MSR to the hardware or by
trapping and handling in the TDX module.  Other than possibly being slow,
these MSRs appear to function just as they would on bare metal.

CPUID Behavior
--------------

For some CPUID leaves and sub-leaves, the virtualized bit fields of CPUID
return values (in guest EAX/EBX/ECX/EDX) are configurable by the
hypervisor. For such cases, the Intel TDX module architecture defines two
virtualization types:

- Bit fields for which the hypervisor controls the value seen by the guest
  TD.

- Bit fields for which the hypervisor configures the value such that the
  guest TD either sees their native value or a value of 0.  For these bit
  fields, the hypervisor can mask off the native values, but it can not
  turn *on* values.

A #VE is generated for CPUID leaves and sub-leaves that the TDX module does
not know how to handle. The guest kernel may ask the hypervisor for the
value with a hypercall.

#VE on Memory Accesses
======================

There are essentially two classes of TDX memory: private and shared.
Private memory receives full TDX protections.  Its content is protected
against access from the hypervisor.  Shared memory is expected to be
shared between guest and hypervisor and does not receive full TDX
protections.

A TD guest is in control of whether its memory accesses are treated as
private or shared.  It selects the behavior with a bit in its page table
entries.  This helps ensure that a guest does not place sensitive
information in shared memory, exposing it to the untrusted hypervisor.

#VE on Shared Memory
--------------------

Access to shared mappings can cause a #VE.  The hypervisor ultimately
controls whether a shared memory access causes a #VE, so the guest must be
careful to only reference shared pages it can safely handle a #VE.  For
instance, the guest should be careful not to access shared memory in the
#VE handler before it reads the #VE info structure (TDG.VP.VEINFO.GET).

Shared mapping content is entirely controlled by the hypervisor. The guest
should only use shared mappings for communicating with the hypervisor.
Shared mappings must never be used for sensitive memory content like kernel
stacks.  A good rule of thumb is that hypervisor-shared memory should be
treated the same as memory mapped to userspace.  Both the hypervisor and
userspace are completely untrusted.

MMIO for virtual devices is implemented as shared memory.  The guest must
be careful not to access device MMIO regions unless it is also prepared to
handle a #VE.

#VE on Private Pages
--------------------

An access to private mappings can also cause a #VE.  Since all kernel
memory is also private memory, the kernel might theoretically need to
handle a #VE on arbitrary kernel memory accesses.  This is not feasible, so
TDX guests ensure that all guest memory has been "accepted" before memory
is used by the kernel.

A modest amount of memory (typically 512M) is pre-accepted by the firmware
before the kernel runs to ensure that the kernel can start up without
being subjected to a #VE.

The hypervisor is permitted to unilaterally move accepted pages to a
"blocked" state. However, if it does this, page access will not generate a
#VE.  It will, instead, cause a "TD Exit" where the hypervisor is required
to handle the exception.

Linux #VE handler
=================

Just like page faults or #GP's, #VE exceptions can be either handled or be
fatal.  Typically, an unhandled userspace #VE results in a SIGSEGV.
An unhandled kernel #VE results in an oops.

Handling nested exceptions on x86 is typically nasty business.  A #VE
could be interrupted by an NMI which triggers another #VE and hilarity
ensues.  The TDX #VE architecture anticipated this scenario and includes a
feature to make it slightly less nasty.

During #VE handling, the TDX module ensures that all interrupts (including
NMIs) are blocked.  The block remains in place until the guest makes a
TDG.VP.VEINFO.GET TDCALL.  This allows the guest to control when interrupts
or a new #VE can be delivered.

However, the guest kernel must still be careful to avoid potential
#VE-triggering actions (discussed above) while this block is in place.
While the block is in place, any #VE is elevated to a double fault (#DF)
which is not recoverable.

MMIO handling
=============

In non-TDX VMs, MMIO is usually implemented by giving a guest access to a
mapping which will cause a VMEXIT on access, and then the hypervisor
emulates the access.  That is not possible in TDX guests because VMEXIT
will expose the register state to the host. TDX guests don't trust the host
and can't have their state exposed to the host.

In TDX, MMIO regions typically trigger a #VE exception in the guest.  The
guest #VE handler then emulates the MMIO instruction inside the guest and
converts it into a controlled TDCALL to the host, rather than exposing
guest state to the host.

MMIO addresses on x86 are just special physical addresses. They can
theoretically be accessed with any instruction that accesses memory.
However, the kernel instruction decoding method is limited. It is only
designed to decode instructions like those generated by io.h macros.

MMIO access via other means (like structure overlays) may result in an
oops.

Shared Memory Conversions
=========================

All TDX guest memory starts out as private at boot.  This memory can not
be accessed by the hypervisor.  However, some kernel users like device
drivers might have a need to share data with the hypervisor.  To do this,
memory must be converted between shared and private.  This can be
accomplished using some existing memory encryption helpers:

 * set_memory_decrypted() converts a range of pages to shared.
 * set_memory_encrypted() converts memory back to private.

Device drivers are the primary user of shared memory, but there's no need
to touch every driver. DMA buffers and ioremap() do the conversions
automatically.

TDX uses SWIOTLB for most DMA allocations. The SWIOTLB buffer is
converted to shared on boot.

For coherent DMA allocation, the DMA buffer gets converted on the
allocation. Check force_dma_unencrypted() for details.

Attestation
===========

Attestation is used to verify the TDX guest trustworthiness to other
entities before provisioning secrets to the guest. For example, a key
server may want to use attestation to verify that the guest is the
desired one before releasing the encryption keys to mount the encrypted
rootfs or a secondary drive.

The TDX module records the state of the TDX guest in various stages of
the guest boot process using the build time measurement register (MRTD)
and runtime measurement registers (RTMR). Measurements related to the
guest initial configuration and firmware image are recorded in the MRTD
register. Measurements related to initial state, kernel image, firmware
image, command line options, initrd, ACPI tables, etc are recorded in
RTMR registers. For more details, as an example, please refer to TDX
Virtual Firmware design specification, section titled "TD Measurement".
At TDX guest runtime, the attestation process is used to attest to these
measurements.

The attestation process consists of two steps: TDREPORT generation and
Quote generation.

TDX guest uses TDCALL[TDG.MR.REPORT] to get the TDREPORT (TDREPORT_STRUCT)
from the TDX module. TDREPORT is a fixed-size data structure generated by
the TDX module which contains guest-specific information (such as build
and boot measurements), platform security version, and the MAC to protect
the integrity of the TDREPORT. A user-provided 64-Byte REPORTDATA is used
as input and included in the TDREPORT. Typically it can be some nonce
provided by attestation service so the TDREPORT can be verified uniquely.
More details about the TDREPORT can be found in Intel TDX Module
specification, section titled "TDG.MR.REPORT Leaf".

After getting the TDREPORT, the second step of the attestation process
is to send it to the Quoting Enclave (QE) to generate the Quote. TDREPORT
by design can only be verified on the local platform as the MAC key is
bound to the platform. To support remote verification of the TDREPORT,
TDX leverages Intel SGX Quoting Enclave to verify the TDREPORT locally
and convert it to a remotely verifiable Quote. Method of sending TDREPORT
to QE is implementation specific. Attestation software can choose
whatever communication channel available (i.e. vsock or TCP/IP) to
send the TDREPORT to QE and receive the Quote.

References
==========

TDX reference material is collected here:

https://www.intel.com/content/www/us/en/developer/articles/technical/intel-trust-domain-extensions.html


code:
.. SPDX-License-Identifier: GPL-2.0

=======
The TLB
=======

When the kernel unmaps or modified the attributes of a range of
memory, it has two choices:

 1. Flush the entire TLB with a two-instruction sequence.  This is
    a quick operation, but it causes collateral damage: TLB entries
    from areas other than the one we are trying to flush will be
    destroyed and must be refilled later, at some cost.
 2. Use the invlpg instruction to invalidate a single page at a
    time.  This could potentially cost many more instructions, but
    it is a much more precise operation, causing no collateral
    damage to other TLB entries.

Which method to do depends on a few things:

 1. The size of the flush being performed.  A flush of the entire
    address space is obviously better performed by flushing the
    entire TLB than doing 2^48/PAGE_SIZE individual flushes.
 2. The contents of the TLB.  If the TLB is empty, then there will
    be no collateral damage caused by doing the global flush, and
    all of the individual flush will have ended up being wasted
    work.
 3. The size of the TLB.  The larger the TLB, the more collateral
    damage we do with a full flush.  So, the larger the TLB, the
    more attractive an individual flush looks.  Data and
    instructions have separate TLBs, as do different page sizes.
 4. The microarchitecture.  The TLB has become a multi-level
    cache on modern CPUs, and the global flushes have become more
    expensive relative to single-page flushes.

There is obviously no way the kernel can know all these things,
especially the contents of the TLB during a given flush.  The
sizes of the flush will vary greatly depending on the workload as
well.  There is essentially no "right" point to choose.

You may be doing too many individual invalidations if you see the
invlpg instruction (or instructions _near_ it) show up high in
profiles.  If you believe that individual invalidations being
called too often, you can lower the tunable::

	/sys/kernel/debug/x86/tlb_single_page_flush_ceiling

This will cause us to do the global flush for more cases.
Lowering it to 0 will disable the use of the individual flushes.
Setting it to 1 is a very conservative setting and it should
never need to be 0 under normal circumstances.

Despite the fact that a single individual flush on x86 is
guaranteed to flush a full 2MB [1]_, hugetlbfs always uses the full
flushes.  THP is treated exactly the same as normal memory.

You might see invlpg inside of flush_tlb_mm_range() show up in
profiles, or you can use the trace_tlb_flush() tracepoints. to
determine how long the flush operations are taking.

Essentially, you are balancing the cycles you spend doing invlpg
with the cycles that you spend refilling the TLB later.

You can measure how expensive TLB refills are by using
performance counters and 'perf stat', like this::

  perf stat -e
    cpu/event=0x8,umask=0x84,name=dtlb_load_misses_walk_duration/,
    cpu/event=0x8,umask=0x82,name=dtlb_load_misses_walk_completed/,
    cpu/event=0x49,umask=0x4,name=dtlb_store_misses_walk_duration/,
    cpu/event=0x49,umask=0x2,name=dtlb_store_misses_walk_completed/,
    cpu/event=0x85,umask=0x4,name=itlb_misses_walk_duration/,
    cpu/event=0x85,umask=0x2,name=itlb_misses_walk_completed/

That works on an IvyBridge-era CPU (i5-3320M).  Different CPUs
may have differently-named counters, but they should at least
be there in some form.  You can use pmu-tools 'ocperf list'
(https://github.com/andikleen/pmu-tools) to find the right
counters for a given CPU.

.. [1] A footnote in Intel's SDM "4.10.4.2 Recommended Invalidation"
   says: "One execution of INVLPG is sufficient even for a page
   with size greater than 4 KBytes."


code:
.. SPDX-License-Identifier: GPL-2.0

============
x86 Topology
============

This documents and clarifies the main aspects of x86 topology modelling and
representation in the kernel. Update/change when doing changes to the
respective code.

The architecture-agnostic topology definitions are in
Documentation/admin-guide/cputopology.rst. This file holds x86-specific
differences/specialities which must not necessarily apply to the generic
definitions. Thus, the way to read up on Linux topology on x86 is to start
with the generic one and look at this one in parallel for the x86 specifics.

Needless to say, code should use the generic functions - this file is *only*
here to *document* the inner workings of x86 topology.

Started by Thomas Gleixner <tglx@linutronix.de> and Borislav Petkov <bp@alien8.de>.

The main aim of the topology facilities is to present adequate interfaces to
code which needs to know/query/use the structure of the running system wrt
threads, cores, packages, etc.

The kernel does not care about the concept of physical sockets because a
socket has no relevance to software. It's an electromechanical component. In
the past a socket always contained a single package (see below), but with the
advent of Multi Chip Modules (MCM) a socket can hold more than one package. So
there might be still references to sockets in the code, but they are of
historical nature and should be cleaned up.

The topology of a system is described in the units of:

    - packages
    - cores
    - threads

Package
=======
Packages contain a number of cores plus shared resources, e.g. DRAM
controller, shared caches etc.

Modern systems may also use the term 'Die' for package.

AMD nomenclature for package is 'Node'.

Package-related topology information in the kernel:

  - cpuinfo_x86.x86_max_cores:

    The number of cores in a package. This information is retrieved via CPUID.

  - cpuinfo_x86.x86_max_dies:

    The number of dies in a package. This information is retrieved via CPUID.

  - cpuinfo_x86.cpu_die_id:

    The physical ID of the die. This information is retrieved via CPUID.

  - cpuinfo_x86.phys_proc_id:

    The physical ID of the package. This information is retrieved via CPUID
    and deduced from the APIC IDs of the cores in the package.

    Modern systems use this value for the socket. There may be multiple
    packages within a socket. This value may differ from cpu_die_id.

  - cpuinfo_x86.logical_proc_id:

    The logical ID of the package. As we do not trust BIOSes to enumerate the
    packages in a consistent way, we introduced the concept of logical package
    ID so we can sanely calculate the number of maximum possible packages in
    the system and have the packages enumerated linearly.

  - topology_max_packages():

    The maximum possible number of packages in the system. Helpful for per
    package facilities to preallocate per package information.

  - cpu_llc_id:

    A per-CPU variable containing:

      - On Intel, the first APIC ID of the list of CPUs sharing the Last Level
        Cache

      - On AMD, the Node ID or Core Complex ID containing the Last Level
        Cache. In general, it is a number identifying an LLC uniquely on the
        system.

Cores
=====
A core consists of 1 or more threads. It does not matter whether the threads
are SMT- or CMT-type threads.

AMDs nomenclature for a CMT core is "Compute Unit". The kernel always uses
"core".

Core-related topology information in the kernel:

  - smp_num_siblings:

    The number of threads in a core. The number of threads in a package can be
    calculated by::

	threads_per_package = cpuinfo_x86.x86_max_cores * smp_num_siblings

Threads
=======
A thread is a single scheduling unit. It's the equivalent to a logical Linux
CPU.

AMDs nomenclature for CMT threads is "Compute Unit Core". The kernel always
uses "thread".

Thread-related topology information in the kernel:

  - topology_core_cpumask():

    The cpumask contains all online threads in the package to which a thread
    belongs.

    The number of online threads is also printed in /proc/cpuinfo "siblings."

  - topology_sibling_cpumask():

    The cpumask contains all online threads in the core to which a thread
    belongs.

  - topology_logical_package_id():

    The logical package ID to which a thread belongs.

  - topology_physical_package_id():

    The physical package ID to which a thread belongs.

  - topology_core_id();

    The ID of the core to which a thread belongs. It is also printed in /proc/cpuinfo
    "core_id."

System topology examples
========================

.. note::
  The alternative Linux CPU enumeration depends on how the BIOS enumerates the
  threads. Many BIOSes enumerate all threads 0 first and then all threads 1.
  That has the "advantage" that the logical Linux CPU numbers of threads 0 stay
  the same whether threads are enabled or not. That's merely an implementation
  detail and has no practical impact.

1) Single Package, Single Core::

   [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0

2) Single Package, Dual Core

   a) One thread per core::

	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
		    -> [core 1] -> [thread 0] -> Linux CPU 1

   b) Two threads per core::

	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
				-> [thread 1] -> Linux CPU 1
		    -> [core 1] -> [thread 0] -> Linux CPU 2
				-> [thread 1] -> Linux CPU 3

      Alternative enumeration::

	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
				-> [thread 1] -> Linux CPU 2
		    -> [core 1] -> [thread 0] -> Linux CPU 1
				-> [thread 1] -> Linux CPU 3

      AMD nomenclature for CMT systems::

	[node 0] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 0
				     -> [Compute Unit Core 1] -> Linux CPU 1
		 -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 2
				     -> [Compute Unit Core 1] -> Linux CPU 3

4) Dual Package, Dual Core

   a) One thread per core::

	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
		    -> [core 1] -> [thread 0] -> Linux CPU 1

	[package 1] -> [core 0] -> [thread 0] -> Linux CPU 2
		    -> [core 1] -> [thread 0] -> Linux CPU 3

   b) Two threads per core::

	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
				-> [thread 1] -> Linux CPU 1
		    -> [core 1] -> [thread 0] -> Linux CPU 2
				-> [thread 1] -> Linux CPU 3

	[package 1] -> [core 0] -> [thread 0] -> Linux CPU 4
				-> [thread 1] -> Linux CPU 5
		    -> [core 1] -> [thread 0] -> Linux CPU 6
				-> [thread 1] -> Linux CPU 7

      Alternative enumeration::

	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
				-> [thread 1] -> Linux CPU 4
		    -> [core 1] -> [thread 0] -> Linux CPU 1
				-> [thread 1] -> Linux CPU 5

	[package 1] -> [core 0] -> [thread 0] -> Linux CPU 2
				-> [thread 1] -> Linux CPU 6
		    -> [core 1] -> [thread 0] -> Linux CPU 3
				-> [thread 1] -> Linux CPU 7

      AMD nomenclature for CMT systems::

	[node 0] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 0
				     -> [Compute Unit Core 1] -> Linux CPU 1
		 -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 2
				     -> [Compute Unit Core 1] -> Linux CPU 3

	[node 1] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 4
				     -> [Compute Unit Core 1] -> Linux CPU 5
		 -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 6
				     -> [Compute Unit Core 1] -> Linux CPU 7