code:
.. SPDX-License-Identifier: GPL-2.0

=============
Multi-Gen LRU
=============
The multi-gen LRU is an alternative LRU implementation that optimizes
page reclaim and improves performance under memory pressure. Page
reclaim decides the kernel's caching policy and ability to overcommit
memory. It directly impacts the kswapd CPU usage and RAM efficiency.

Quick start
===========
Build the kernel with the following configurations.

* ``CONFIG_LRU_GEN=y``
* ``CONFIG_LRU_GEN_ENABLED=y``

All set!

Runtime options
===============
``/sys/kernel/mm/lru_gen/`` contains stable ABIs described in the
following subsections.

Kill switch
-----------
``enabled`` accepts different values to enable or disable the
following components. Its default value depends on
``CONFIG_LRU_GEN_ENABLED``. All the components should be enabled
unless some of them have unforeseen side effects. Writing to
``enabled`` has no effect when a component is not supported by the
hardware, and valid values will be accepted even when the main switch
is off.

====== ===============================================================
Values Components
====== ===============================================================
0x0001 The main switch for the multi-gen LRU.
0x0002 Clearing the accessed bit in leaf page table entries in large
       batches, when MMU sets it (e.g., on x86). This behavior can
       theoretically worsen lock contention (mmap_lock). If it is
       disabled, the multi-gen LRU will suffer a minor performance
       degradation for workloads that contiguously map hot pages,
       whose accessed bits can be otherwise cleared by fewer larger
       batches.
0x0004 Clearing the accessed bit in non-leaf page table entries as
       well, when MMU sets it (e.g., on x86). This behavior was not
       verified on x86 varieties other than Intel and AMD. If it is
       disabled, the multi-gen LRU will suffer a negligible
       performance degradation.
[yYnN] Apply to all the components above.
====== ===============================================================

E.g.,
::

    echo y >/sys/kernel/mm/lru_gen/enabled
    cat /sys/kernel/mm/lru_gen/enabled
    0x0007
    echo 5 >/sys/kernel/mm/lru_gen/enabled
    cat /sys/kernel/mm/lru_gen/enabled
    0x0005

Thrashing prevention
--------------------
Personal computers are more sensitive to thrashing because it can
cause janks (lags when rendering UI) and negatively impact user
experience. The multi-gen LRU offers thrashing prevention to the
majority of laptop and desktop users who do not have ``oomd``.

Users can write ``N`` to ``min_ttl_ms`` to prevent the working set of
``N`` milliseconds from getting evicted. The OOM killer is triggered
if this working set cannot be kept in memory. In other words, this
option works as an adjustable pressure relief valve, and when open, it
terminates applications that are hopefully not being used.

Based on the average human detectable lag (~100ms), ``N=1000`` usually
eliminates intolerable janks due to thrashing. Larger values like
``N=3000`` make janks less noticeable at the risk of premature OOM
kills.

The default value ``0`` means disabled.

Experimental features
=====================
``/sys/kernel/debug/lru_gen`` accepts commands described in the
following subsections. Multiple command lines are supported, so does
concatenation with delimiters ``,`` and ``;``.

``/sys/kernel/debug/lru_gen_full`` provides additional stats for
debugging. ``CONFIG_LRU_GEN_STATS=y`` keeps historical stats from
evicted generations in this file.

Working set estimation
----------------------
Working set estimation measures how much memory an application needs
in a given time interval, and it is usually done with little impact on
the performance of the application. E.g., data centers want to
optimize job scheduling (bin packing) to improve memory utilizations.
When a new job comes in, the job scheduler needs to find out whether
each server it manages can allocate a certain amount of memory for
this new job before it can pick a candidate. To do so, the job
scheduler needs to estimate the working sets of the existing jobs.

When it is read, ``lru_gen`` returns a histogram of numbers of pages
accessed over different time intervals for each memcg and node.
``MAX_NR_GENS`` decides the number of bins for each histogram. The
histograms are noncumulative.
::

    memcg  memcg_id  memcg_path
       node  node_id
           min_gen_nr  age_in_ms  nr_anon_pages  nr_file_pages
           ...
           max_gen_nr  age_in_ms  nr_anon_pages  nr_file_pages

Each bin contains an estimated number of pages that have been accessed
within ``age_in_ms``. E.g., ``min_gen_nr`` contains the coldest pages
and ``max_gen_nr`` contains the hottest pages, since ``age_in_ms`` of
the former is the largest and that of the latter is the smallest.

Users can write the following command to ``lru_gen`` to create a new
generation ``max_gen_nr+1``:

    ``+ memcg_id node_id max_gen_nr [can_swap [force_scan]]``

``can_swap`` defaults to the swap setting and, if it is set to ``1``,
it forces the scan of anon pages when swap is off, and vice versa.
``force_scan`` defaults to ``1`` and, if it is set to ``0``, it
employs heuristics to reduce the overhead, which is likely to reduce
the coverage as well.

A typical use case is that a job scheduler runs this command at a
certain time interval to create new generations, and it ranks the
servers it manages based on the sizes of their cold pages defined by
this time interval.

Proactive reclaim
-----------------
Proactive reclaim induces page reclaim when there is no memory
pressure. It usually targets cold pages only. E.g., when a new job
comes in, the job scheduler wants to proactively reclaim cold pages on
the server it selected, to improve the chance of successfully landing
this new job.

Users can write the following command to ``lru_gen`` to evict
generations less than or equal to ``min_gen_nr``.

    ``- memcg_id node_id min_gen_nr [swappiness [nr_to_reclaim]]``

``min_gen_nr`` should be less than ``max_gen_nr-1``, since
``max_gen_nr`` and ``max_gen_nr-1`` are not fully aged (equivalent to
the active list) and therefore cannot be evicted. ``swappiness``
overrides the default value in ``/proc/sys/vm/swappiness``.
``nr_to_reclaim`` limits the number of pages to evict.

A typical use case is that a job scheduler runs this command before it
tries to land a new job on a server. If it fails to materialize enough
cold pages because of the overestimation, it retries on the next
server according to the ranking result obtained from the working set
estimation step. This less forceful approach limits the impacts on the
existing jobs.


code:
=============================
No-MMU memory mapping support
=============================

The kernel has limited support for memory mapping under no-MMU conditions, such
as are used in uClinux environments. From the userspace point of view, memory
mapping is made use of in conjunction with the mmap() system call, the shmat()
call and the execve() system call. From the kernel's point of view, execve()
mapping is actually performed by the binfmt drivers, which call back into the
mmap() routines to do the actual work.

Memory mapping behaviour also involves the way fork(), vfork(), clone() and
ptrace() work. Under uClinux there is no fork(), and clone() must be supplied
the CLONE_VM flag.

The behaviour is similar between the MMU and no-MMU cases, but not identical;
and it's also much more restricted in the latter case:

 (#) Anonymous mapping, MAP_PRIVATE

	In the MMU case: VM regions backed by arbitrary pages; copy-on-write
	across fork.

	In the no-MMU case: VM regions backed by arbitrary contiguous runs of
	pages.

 (#) Anonymous mapping, MAP_SHARED

	These behave very much like private mappings, except that they're
	shared across fork() or clone() without CLONE_VM in the MMU case. Since
	the no-MMU case doesn't support these, behaviour is identical to
	MAP_PRIVATE there.

 (#) File, MAP_PRIVATE, PROT_READ / PROT_EXEC, !PROT_WRITE

	In the MMU case: VM regions backed by pages read from file; changes to
	the underlying file are reflected in the mapping; copied across fork.

	In the no-MMU case:

         - If one exists, the kernel will re-use an existing mapping to the
           same segment of the same file if that has compatible permissions,
           even if this was created by another process.

         - If possible, the file mapping will be directly on the backing device
           if the backing device has the NOMMU_MAP_DIRECT capability and
           appropriate mapping protection capabilities. Ramfs, romfs, cramfs
           and mtd might all permit this.

	 - If the backing device can't or won't permit direct sharing,
           but does have the NOMMU_MAP_COPY capability, then a copy of the
           appropriate bit of the file will be read into a contiguous bit of
           memory and any extraneous space beyond the EOF will be cleared

	 - Writes to the file do not affect the mapping; writes to the mapping
	   are visible in other processes (no MMU protection), but should not
	   happen.

 (#) File, MAP_PRIVATE, PROT_READ / PROT_EXEC, PROT_WRITE

	In the MMU case: like the non-PROT_WRITE case, except that the pages in
	question get copied before the write actually happens. From that point
	on writes to the file underneath that page no longer get reflected into
	the mapping's backing pages. The page is then backed by swap instead.

	In the no-MMU case: works much like the non-PROT_WRITE case, except
	that a copy is always taken and never shared.

 (#) Regular file / blockdev, MAP_SHARED, PROT_READ / PROT_EXEC / PROT_WRITE

	In the MMU case: VM regions backed by pages read from file; changes to
	pages written back to file; writes to file reflected into pages backing
	mapping; shared across fork.

	In the no-MMU case: not supported.

 (#) Memory backed regular file, MAP_SHARED, PROT_READ / PROT_EXEC / PROT_WRITE

	In the MMU case: As for ordinary regular files.

	In the no-MMU case: The filesystem providing the memory-backed file
	(such as ramfs or tmpfs) may choose to honour an open, truncate, mmap
	sequence by providing a contiguous sequence of pages to map. In that
	case, a shared-writable memory mapping will be possible. It will work
	as for the MMU case. If the filesystem does not provide any such
	support, then the mapping request will be denied.

 (#) Memory backed blockdev, MAP_SHARED, PROT_READ / PROT_EXEC / PROT_WRITE

	In the MMU case: As for ordinary regular files.

	In the no-MMU case: As for memory backed regular files, but the
	blockdev must be able to provide a contiguous run of pages without
	truncate being called. The ramdisk driver could do this if it allocated
	all its memory as a contiguous array upfront.

 (#) Memory backed chardev, MAP_SHARED, PROT_READ / PROT_EXEC / PROT_WRITE

	In the MMU case: As for ordinary regular files.

	In the no-MMU case: The character device driver may choose to honour
	the mmap() by providing direct access to the underlying device if it
	provides memory or quasi-memory that can be accessed directly. Examples
	of such are frame buffers and flash devices. If the driver does not
	provide any such support, then the mapping request will be denied.

Further notes on no-MMU MMAP
============================

 (#) A request for a private mapping of a file may return a buffer that is not
     page-aligned.  This is because XIP may take place, and the data may not be
     paged aligned in the backing store.

 (#) A request for an anonymous mapping will always be page aligned.  If
     possible the size of the request should be a power of two otherwise some
     of the space may be wasted as the kernel must allocate a power-of-2
     granule but will only discard the excess if appropriately configured as
     this has an effect on fragmentation.

 (#) The memory allocated by a request for an anonymous mapping will normally
     be cleared by the kernel before being returned in accordance with the
     Linux man pages (ver 2.22 or later).

     In the MMU case this can be achieved with reasonable performance as
     regions are backed by virtual pages, with the contents only being mapped
     to cleared physical pages when a write happens on that specific page
     (prior to which, the pages are effectively mapped to the global zero page
     from which reads can take place).  This spreads out the time it takes to
     initialize the contents of a page - depending on the write-usage of the
     mapping.

     In the no-MMU case, however, anonymous mappings are backed by physical
     pages, and the entire map is cleared at allocation time.  This can cause
     significant delays during a userspace malloc() as the C library does an
     anonymous mapping and the kernel then does a memset for the entire map.

     However, for memory that isn't required to be precleared - such as that
     returned by malloc() - mmap() can take a MAP_UNINITIALIZED flag to
     indicate to the kernel that it shouldn't bother clearing the memory before
     returning it.  Note that CONFIG_MMAP_ALLOW_UNINITIALIZED must be enabled
     to permit this, otherwise the flag will be ignored.

     uClibc uses this to speed up malloc(), and the ELF-FDPIC binfmt uses this
     to allocate the brk and stack region.

 (#) A list of all the private copy and anonymous mappings on the system is
     visible through /proc/maps in no-MMU mode.

 (#) A list of all the mappings in use by a process is visible through
     /proc/<pid>/maps in no-MMU mode.

 (#) Supplying MAP_FIXED or a requesting a particular mapping address will
     result in an error.

 (#) Files mapped privately usually have to have a read method provided by the
     driver or filesystem so that the contents can be read into the memory
     allocated if mmap() chooses not to map the backing device directly. An
     error will result if they don't. This is most likely to be encountered
     with character device files, pipes, fifos and sockets.

Interprocess shared memory
==========================

Both SYSV IPC SHM shared memory and POSIX shared memory is supported in NOMMU
mode.  The former through the usual mechanism, the latter through files created
on ramfs or tmpfs mounts.

Futexes
=======

Futexes are supported in NOMMU mode if the arch supports them.  An error will
be given if an address passed to the futex system call lies outside the
mappings made by a process or if the mapping in which the address lies does not
support futexes (such as an I/O chardev mapping).

No-MMU mremap
=============

The mremap() function is partially supported.  It may change the size of a
mapping, and may move it [#]_ if MREMAP_MAYMOVE is specified and if the new size
of the mapping exceeds the size of the slab object currently occupied by the
memory to which the mapping refers, or if a smaller slab object could be used.

MREMAP_FIXED is not supported, though it is ignored if there's no change of
address and the object does not need to be moved.

Shared mappings may not be moved.  Shareable mappings may not be moved either,
even if they are not currently shared.

The mremap() function must be given an exact match for base address and size of
a previously mapped object.  It may not be used to create holes in existing
mappings, move parts of existing mappings or resize parts of mappings.  It must
act on a complete mapping.

.. [#] Not currently supported.

Providing shareable character device support
============================================

To provide shareable character device support, a driver must provide a
file->f_op->get_unmapped_area() operation. The mmap() routines will call this
to get a proposed address for the mapping. This may return an error if it
doesn't wish to honour the mapping because it's too long, at a weird offset,
under some unsupported combination of flags or whatever.

The driver should also provide backing device information with capabilities set
to indicate the permitted types of mapping on such devices. The default is
assumed to be readable and writable, not executable, and only shareable
directly (can't be copied).

The file->f_op->mmap() operation will be called to actually inaugurate the
mapping. It can be rejected at that point. Returning the ENOSYS error will
cause the mapping to be copied instead if NOMMU_MAP_COPY is specified.

The vm_ops->close() routine will be invoked when the last mapping on a chardev
is removed. An existing mapping will be shared, partially or not, if possible
without notifying the driver.

It is permitted also for the file->f_op->get_unmapped_area() operation to
return -ENOSYS. This will be taken to mean that this operation just doesn't
want to handle it, despite the fact it's got an operation. For instance, it
might try directing the call to a secondary driver which turns out not to
implement it. Such is the case for the framebuffer driver which attempts to
direct the call to the device-specific driver. Under such circumstances, the
mapping request will be rejected if NOMMU_MAP_COPY is not specified, and a
copy mapped otherwise.

.. important::

	Some types of device may present a different appearance to anyone
	looking at them in certain modes. Flash chips can be like this; for
	instance if they're in programming or erase mode, you might see the
	status reflected in the mapping, instead of the data.

	In such a case, care must be taken lest userspace see a shared or a
	private mapping showing such information when the driver is busy
	controlling the device. Remember especially: private executable
	mappings may still be mapped directly off the device under some
	circumstances!

Providing shareable memory-backed file support
==============================================

Provision of shared mappings on memory backed files is similar to the provision
of support for shared mapped character devices. The main difference is that the
filesystem providing the service will probably allocate a contiguous collection
of pages and permit mappings to be made on that.

It is recommended that a truncate operation applied to such a file that
increases the file size, if that file is empty, be taken as a request to gather
enough pages to honour a mapping. This is required to support POSIX shared
memory.

Memory backed devices are indicated by the mapping's backing device info having
the memory_backed flag set.

Providing shareable block device support
========================================

Provision of shared mappings on block device files is exactly the same as for
character devices. If there isn't a real device underneath, then the driver
should allocate sufficient contiguous memory to honour any supported mapping.

Adjusting page trimming behaviour
=================================

NOMMU mmap automatically rounds up to the nearest power-of-2 number of pages
when performing an allocation.  This can have adverse effects on memory
fragmentation, and as such, is left configurable.  The default behaviour is to
aggressively trim allocations and discard any excess pages back in to the page
allocator.  In order to retain finer-grained control over fragmentation, this
behaviour can either be disabled completely, or bumped up to a higher page
watermark where trimming begins.

Page trimming behaviour is configurable via the sysctl ``vm.nr_trim_pages``.

repo_name: angery/linux


code:
.. _numa_memory_policy:

==================
NUMA Memory Policy
==================

What is NUMA Memory Policy?
============================

In the Linux kernel, "memory policy" determines from which node the kernel will
allocate memory in a NUMA system or in an emulated NUMA system.  Linux has
supported platforms with Non-Uniform Memory Access architectures since 2.4.?.
The current memory policy support was added to Linux 2.6 around May 2004.  This
document attempts to describe the concepts and APIs of the 2.6 memory policy
support.

Memory policies should not be confused with cpusets
(``Documentation/admin-guide/cgroup-v1/cpusets.rst``)
which is an administrative mechanism for restricting the nodes from which
memory may be allocated by a set of processes. Memory policies are a
programming interface that a NUMA-aware application can take advantage of.  When
both cpusets and policies are applied to a task, the restrictions of the cpuset
takes priority.  See :ref:`Memory Policies and cpusets <mem_pol_and_cpusets>`
below for more details.

Memory Policy Concepts
======================

Scope of Memory Policies
------------------------

The Linux kernel supports _scopes_ of memory policy, described here from
most general to most specific:

System Default Policy
	this policy is "hard coded" into the kernel.  It is the policy
	that governs all page allocations that aren't controlled by
	one of the more specific policy scopes discussed below.  When
	the system is "up and running", the system default policy will
	use "local allocation" described below.  However, during boot
	up, the system default policy will be set to interleave
	allocations across all nodes with "sufficient" memory, so as
	not to overload the initial boot node with boot-time
	allocations.

Task/Process Policy
	this is an optional, per-task policy.  When defined for a
	specific task, this policy controls all page allocations made
	by or on behalf of the task that aren't controlled by a more
	specific scope. If a task does not define a task policy, then
	all page allocations that would have been controlled by the
	task policy "fall back" to the System Default Policy.

	The task policy applies to the entire address space of a task. Thus,
	it is inheritable, and indeed is inherited, across both fork()
	[clone() w/o the CLONE_VM flag] and exec*().  This allows a parent task
	to establish the task policy for a child task exec()'d from an
	executable image that has no awareness of memory policy.  See the
	:ref:`Memory Policy APIs <memory_policy_apis>` section,
	below, for an overview of the system call
	that a task may use to set/change its task/process policy.

	In a multi-threaded task, task policies apply only to the thread
	[Linux kernel task] that installs the policy and any threads
	subsequently created by that thread.  Any sibling threads existing
	at the time a new task policy is installed retain their current
	policy.

	A task policy applies only to pages allocated after the policy is
	installed.  Any pages already faulted in by the task when the task
	changes its task policy remain where they were allocated based on
	the policy at the time they were allocated.

.. _vma_policy:

VMA Policy
	A "VMA" or "Virtual Memory Area" refers to a range of a task's
	virtual address space.  A task may define a specific policy for a range
	of its virtual address space.   See the
	:ref:`Memory Policy APIs <memory_policy_apis>` section,
	below, for an overview of the mbind() system call used to set a VMA
	policy.

	A VMA policy will govern the allocation of pages that back
	this region of the address space.  Any regions of the task's
	address space that don't have an explicit VMA policy will fall
	back to the task policy, which may itself fall back to the
	System Default Policy.

	VMA policies have a few complicating details:

	* VMA policy applies ONLY to anonymous pages.  These include
	  pages allocated for anonymous segments, such as the task
	  stack and heap, and any regions of the address space
	  mmap()ed with the MAP_ANONYMOUS flag.  If a VMA policy is
	  applied to a file mapping, it will be ignored if the mapping
	  used the MAP_SHARED flag.  If the file mapping used the
	  MAP_PRIVATE flag, the VMA policy will only be applied when
	  an anonymous page is allocated on an attempt to write to the
	  mapping-- i.e., at Copy-On-Write.

	* VMA policies are shared between all tasks that share a
	  virtual address space--a.k.a. threads--independent of when
	  the policy is installed; and they are inherited across
	  fork().  However, because VMA policies refer to a specific
	  region of a task's address space, and because the address
	  space is discarded and recreated on exec*(), VMA policies
	  are NOT inheritable across exec().  Thus, only NUMA-aware
	  applications may use VMA policies.

	* A task may install a new VMA policy on a sub-range of a
	  previously mmap()ed region.  When this happens, Linux splits
	  the existing virtual memory area into 2 or 3 VMAs, each with
	  it's own policy.

	* By default, VMA policy applies only to pages allocated after
	  the policy is installed.  Any pages already faulted into the
	  VMA range remain where they were allocated based on the
	  policy at the time they were allocated.  However, since
	  2.6.16, Linux supports page migration via the mbind() system
	  call, so that page contents can be moved to match a newly
	  installed policy.

Shared Policy
	Conceptually, shared policies apply to "memory objects" mapped
	shared into one or more tasks' distinct address spaces.  An
	application installs shared policies the same way as VMA
	policies--using the mbind() system call specifying a range of
	virtual addresses that map the shared object.  However, unlike
	VMA policies, which can be considered to be an attribute of a
	range of a task's address space, shared policies apply
	directly to the shared object.  Thus, all tasks that attach to
	the object share the policy, and all pages allocated for the
	shared object, by any task, will obey the shared policy.

	As of 2.6.22, only shared memory segments, created by shmget() or
	mmap(MAP_ANONYMOUS|MAP_SHARED), support shared policy.  When shared
	policy support was added to Linux, the associated data structures were
	added to hugetlbfs shmem segments.  At the time, hugetlbfs did not
	support allocation at fault time--a.k.a lazy allocation--so hugetlbfs
	shmem segments were never "hooked up" to the shared policy support.
	Although hugetlbfs segments now support lazy allocation, their support
	for shared policy has not been completed.

	As mentioned above in :ref:`VMA policies <vma_policy>` section,
	allocations of page cache pages for regular files mmap()ed
	with MAP_SHARED ignore any VMA policy installed on the virtual
	address range backed by the shared file mapping.  Rather,
	shared page cache pages, including pages backing private
	mappings that have not yet been written by the task, follow
	task policy, if any, else System Default Policy.

	The shared policy infrastructure supports different policies on subset
	ranges of the shared object.  However, Linux still splits the VMA of
	the task that installs the policy for each range of distinct policy.
	Thus, different tasks that attach to a shared memory segment can have
	different VMA configurations mapping that one shared object.  This
	can be seen by examining the /proc/<pid>/numa_maps of tasks sharing
	a shared memory region, when one task has installed shared policy on
	one or more ranges of the region.

Components of Memory Policies
-----------------------------

A NUMA memory policy consists of a "mode", optional mode flags, and
an optional set of nodes.  The mode determines the behavior of the
policy, the optional mode flags determine the behavior of the mode,
and the optional set of nodes can be viewed as the arguments to the
policy behavior.

Internally, memory policies are implemented by a reference counted
structure, struct mempolicy.  Details of this structure will be
discussed in context, below, as required to explain the behavior.

NUMA memory policy supports the following 4 behavioral modes:

Default Mode--MPOL_DEFAULT
	This mode is only used in the memory policy APIs.  Internally,
	MPOL_DEFAULT is converted to the NULL memory policy in all
	policy scopes.  Any existing non-default policy will simply be
	removed when MPOL_DEFAULT is specified.  As a result,
	MPOL_DEFAULT means "fall back to the next most specific policy
	scope."

	For example, a NULL or default task policy will fall back to the
	system default policy.  A NULL or default vma policy will fall
	back to the task policy.

	When specified in one of the memory policy APIs, the Default mode
	does not use the optional set of nodes.

	It is an error for the set of nodes specified for this policy to
	be non-empty.

MPOL_BIND
	This mode specifies that memory must come from the set of
	nodes specified by the policy.  Memory will be allocated from
	the node in the set with sufficient free memory that is
	closest to the node where the allocation takes place.

MPOL_PREFERRED
	This mode specifies that the allocation should be attempted
	from the single node specified in the policy.  If that
	allocation fails, the kernel will search other nodes, in order
	of increasing distance from the preferred node based on
	information provided by the platform firmware.

	Internally, the Preferred policy uses a single node--the
	preferred_node member of struct mempolicy.  When the internal
	mode flag MPOL_F_LOCAL is set, the preferred_node is ignored
	and the policy is interpreted as local allocation.  "Local"
	allocation policy can be viewed as a Preferred policy that
	starts at the node containing the cpu where the allocation
	takes place.

	It is possible for the user to specify that local allocation
	is always preferred by passing an empty nodemask with this
	mode.  If an empty nodemask is passed, the policy cannot use
	the MPOL_F_STATIC_NODES or MPOL_F_RELATIVE_NODES flags
	described below.

MPOL_INTERLEAVED
	This mode specifies that page allocations be interleaved, on a
	page granularity, across the nodes specified in the policy.
	This mode also behaves slightly differently, based on the
	context where it is used:

	For allocation of anonymous pages and shared memory pages,
	Interleave mode indexes the set of nodes specified by the
	policy using the page offset of the faulting address into the
	segment [VMA] containing the address modulo the number of
	nodes specified by the policy.  It then attempts to allocate a
	page, starting at the selected node, as if the node had been
	specified by a Preferred policy or had been selected by a
	local allocation.  That is, allocation will follow the per
	node zonelist.

	For allocation of page cache pages, Interleave mode indexes
	the set of nodes specified by the policy using a node counter
	maintained per task.  This counter wraps around to the lowest
	specified node after it reaches the highest specified node.
	This will tend to spread the pages out over the nodes
	specified by the policy based on the order in which they are
	allocated, rather than based on any page offset into an
	address range or file.  During system boot up, the temporary
	interleaved system default policy works in this mode.

MPOL_PREFERRED_MANY
	This mode specifices that the allocation should be preferrably
	satisfied from the nodemask specified in the policy. If there is
	a memory pressure on all nodes in the nodemask, the allocation
	can fall back to all existing numa nodes. This is effectively
	MPOL_PREFERRED allowed for a mask rather than a single node.

NUMA memory policy supports the following optional mode flags:

MPOL_F_STATIC_NODES
	This flag specifies that the nodemask passed by
	the user should not be remapped if the task or VMA's set of allowed
	nodes changes after the memory policy has been defined.

	Without this flag, any time a mempolicy is rebound because of a
        change in the set of allowed nodes, the preferred nodemask (Preferred
        Many), preferred node (Preferred) or nodemask (Bind, Interleave) is
        remapped to the new set of allowed nodes.  This may result in nodes
        being used that were previously undesired.

	With this flag, if the user-specified nodes overlap with the
	nodes allowed by the task's cpuset, then the memory policy is
	applied to their intersection.  If the two sets of nodes do not
	overlap, the Default policy is used.

	For example, consider a task that is attached to a cpuset with
	mems 1-3 that sets an Interleave policy over the same set.  If
	the cpuset's mems change to 3-5, the Interleave will now occur
	over nodes 3, 4, and 5.  With this flag, however, since only node
	3 is allowed from the user's nodemask, the "interleave" only
	occurs over that node.  If no nodes from the user's nodemask are
	now allowed, the Default behavior is used.

	MPOL_F_STATIC_NODES cannot be combined with the
	MPOL_F_RELATIVE_NODES flag.  It also cannot be used for
	MPOL_PREFERRED policies that were created with an empty nodemask
	(local allocation).

MPOL_F_RELATIVE_NODES
	This flag specifies that the nodemask passed
	by the user will be mapped relative to the set of the task or VMA's
	set of allowed nodes.  The kernel stores the user-passed nodemask,
	and if the allowed nodes changes, then that original nodemask will
	be remapped relative to the new set of allowed nodes.

	Without this flag (and without MPOL_F_STATIC_NODES), anytime a
	mempolicy is rebound because of a change in the set of allowed
	nodes, the node (Preferred) or nodemask (Bind, Interleave) is
	remapped to the new set of allowed nodes.  That remap may not
	preserve the relative nature of the user's passed nodemask to its
	set of allowed nodes upon successive rebinds: a nodemask of
	1,3,5 may be remapped to 7-9 and then to 1-3 if the set of
	allowed nodes is restored to its original state.

	With this flag, the remap is done so that the node numbers from
	the user's passed nodemask are relative to the set of allowed
	nodes.  In other words, if nodes 0, 2, and 4 are set in the user's
	nodemask, the policy will be effected over the first (and in the
	Bind or Interleave case, the third and fifth) nodes in the set of
	allowed nodes.  The nodemask passed by the user represents nodes
	relative to task or VMA's set of allowed nodes.

	If the user's nodemask includes nodes that are outside the range
	of the new set of allowed nodes (for example, node 5 is set in
	the user's nodemask when the set of allowed nodes is only 0-3),
	then the remap wraps around to the beginning of the nodemask and,
	if not already set, sets the node in the mempolicy nodemask.

	For example, consider a task that is attached to a cpuset with
	mems 2-5 that sets an Interleave policy over the same set with
	MPOL_F_RELATIVE_NODES.  If the cpuset's mems change to 3-7, the
	interleave now occurs over nodes 3,5-7.  If the cpuset's mems
	then change to 0,2-3,5, then the interleave occurs over nodes
	0,2-3,5.

	Thanks to the consistent remapping, applications preparing
	nodemasks to specify memory policies using this flag should
	disregard their current, actual cpuset imposed memory placement
	and prepare the nodemask as if they were always located on
	memory nodes 0 to N-1, where N is the number of memory nodes the
	policy is intended to manage.  Let the kernel then remap to the
	set of memory nodes allowed by the task's cpuset, as that may
	change over time.

	MPOL_F_RELATIVE_NODES cannot be combined with the
	MPOL_F_STATIC_NODES flag.  It also cannot be used for
	MPOL_PREFERRED policies that were created with an empty nodemask
	(local allocation).

Memory Policy Reference Counting
================================

To resolve use/free races, struct mempolicy contains an atomic reference
count field.  Internal interfaces, mpol_get()/mpol_put() increment and
decrement this reference count, respectively.  mpol_put() will only free
the structure back to the mempolicy kmem cache when the reference count
goes to zero.

When a new memory policy is allocated, its reference count is initialized
to '1', representing the reference held by the task that is installing the
new policy.  When a pointer to a memory policy structure is stored in another
structure, another reference is added, as the task's reference will be dropped
on completion of the policy installation.

During run-time "usage" of the policy, we attempt to minimize atomic operations
on the reference count, as this can lead to cache lines bouncing between cpus
and NUMA nodes.  "Usage" here means one of the following:

1) querying of the policy, either by the task itself [using the get_mempolicy()
   API discussed below] or by another task using the /proc/<pid>/numa_maps
   interface.

2) examination of the policy to determine the policy mode and associated node
   or node lists, if any, for page allocation.  This is considered a "hot
   path".  Note that for MPOL_BIND, the "usage" extends across the entire
   allocation process, which may sleep during page reclaimation, because the
   BIND policy nodemask is used, by reference, to filter ineligible nodes.

We can avoid taking an extra reference during the usages listed above as
follows:

1) we never need to get/free the system default policy as this is never
   changed nor freed, once the system is up and running.

2) for querying the policy, we do not need to take an extra reference on the
   target task's task policy nor vma policies because we always acquire the
   task's mm's mmap_lock for read during the query.  The set_mempolicy() and
   mbind() APIs [see below] always acquire the mmap_lock for write when
   installing or replacing task or vma policies.  Thus, there is no possibility
   of a task or thread freeing a policy while another task or thread is
   querying it.

3) Page allocation usage of task or vma policy occurs in the fault path where
   we hold them mmap_lock for read.  Again, because replacing the task or vma
   policy requires that the mmap_lock be held for write, the policy can't be
   freed out from under us while we're using it for page allocation.

4) Shared policies require special consideration.  One task can replace a
   shared memory policy while another task, with a distinct mmap_lock, is
   querying or allocating a page based on the policy.  To resolve this
   potential race, the shared policy infrastructure adds an extra reference
   to the shared policy during lookup while holding a spin lock on the shared
   policy management structure.  This requires that we drop this extra
   reference when we're finished "using" the policy.  We must drop the
   extra reference on shared policies in the same query/allocation paths
   used for non-shared policies.  For this reason, shared policies are marked
   as such, and the extra reference is dropped "conditionally"--i.e., only
   for shared policies.

   Because of this extra reference counting, and because we must lookup
   shared policies in a tree structure under spinlock, shared policies are
   more expensive to use in the page allocation path.  This is especially
   true for shared policies on shared memory regions shared by tasks running
   on different NUMA nodes.  This extra overhead can be avoided by always
   falling back to task or system default policy for shared memory regions,
   or by prefaulting the entire shared memory region into memory and locking
   it down.  However, this might not be appropriate for all applications.

.. _memory_policy_apis:

Memory Policy APIs
==================

Linux supports 4 system calls for controlling memory policy.  These APIS
always affect only the calling task, the calling task's address space, or
some shared object mapped into the calling task's address space.

.. note::
   the headers that define these APIs and the parameter data types for
   user space applications reside in a package that is not part of the
   Linux kernel.  The kernel system call interfaces, with the 'sys\_'
   prefix, are defined in <linux/syscalls.h>; the mode and flag
   definitions are defined in <linux/mempolicy.h>.

Set [Task] Memory Policy::

	long set_mempolicy(int mode, const unsigned long *nmask,
					unsigned long maxnode);

Set's the calling task's "task/process memory policy" to mode
specified by the 'mode' argument and the set of nodes defined by
'nmask'.  'nmask' points to a bit mask of node ids containing at least
'maxnode' ids.  Optional mode flags may be passed by combining the
'mode' argument with the flag (for example: MPOL_INTERLEAVE |
MPOL_F_STATIC_NODES).

See the set_mempolicy(2) man page for more details

Get [Task] Memory Policy or Related Information::

	long get_mempolicy(int *mode,
			   const unsigned long *nmask, unsigned long maxnode,
			   void *addr, int flags);

Queries the "task/process memory policy" of the calling task, or the
policy or location of a specified virtual address, depending on the
'flags' argument.

See the get_mempolicy(2) man page for more details

Install VMA/Shared Policy for a Range of Task's Address Space::

	long mbind(void *start, unsigned long len, int mode,
		   const unsigned long *nmask, unsigned long maxnode,
		   unsigned flags);

mbind() installs the policy specified by (mode, nmask, maxnodes) as a
VMA policy for the range of the calling task's address space specified
by the 'start' and 'len' arguments.  Additional actions may be
requested via the 'flags' argument.

See the mbind(2) man page for more details.

Set home node for a Range of Task's Address Spacec::

	long sys_set_mempolicy_home_node(unsigned long start, unsigned long len,
					 unsigned long home_node,
					 unsigned long flags);

sys_set_mempolicy_home_node set the home node for a VMA policy present in the
task's address range. The system call updates the home node only for the existing
mempolicy range. Other address ranges are ignored. A home node is the NUMA node
closest to which page allocation will come from. Specifying the home node override
the default allocation policy to allocate memory close to the local node for an
executing CPU.

Memory Policy Command Line Interface
====================================

Although not strictly part of the Linux implementation of memory policy,
a command line tool, numactl(8), exists that allows one to:

+ set the task policy for a specified program via set_mempolicy(2), fork(2) and
  exec(2)

+ set the shared policy for a shared memory segment via mbind(2)

The numactl(8) tool is packaged with the run-time version of the library
containing the memory policy system call wrappers.  Some distributions
package the headers and compile-time libraries in a separate development
package.

.. _mem_pol_and_cpusets:

Memory Policies and cpusets
===========================

Memory policies work within cpusets as described above.  For memory policies
that require a node or set of nodes, the nodes are restricted to the set of
nodes whose memories are allowed by the cpuset constraints.  If the nodemask
specified for the policy contains nodes that are not allowed by the cpuset and
MPOL_F_RELATIVE_NODES is not used, the intersection of the set of nodes
specified for the policy and the set of nodes with memory is used.  If the
result is the empty set, the policy is considered invalid and cannot be
installed.  If MPOL_F_RELATIVE_NODES is used, the policy's nodes are mapped
onto and folded into the task's set of allowed nodes as previously described.

The interaction of memory policies and cpusets can be problematic when tasks
in two cpusets share access to a memory region, such as shared memory segments
created by shmget() of mmap() with the MAP_ANONYMOUS and MAP_SHARED flags, and
any of the tasks install shared policy on the region, only nodes whose
memories are allowed in both cpusets may be used in the policies.  Obtaining
this information requires "stepping outside" the memory policy APIs to use the
cpuset information and requires that one know in what cpusets other task might
be attaching to the shared region.  Furthermore, if the cpusets' allowed
memory sets are disjoint, "local" allocation is the only valid policy.


code:
.. _numaperf:

=============
NUMA Locality
=============

Some platforms may have multiple types of memory attached to a compute
node. These disparate memory ranges may share some characteristics, such
as CPU cache coherence, but may have different performance. For example,
different media types and buses affect bandwidth and latency.

A system supports such heterogeneous memory by grouping each memory type
under different domains, or "nodes", based on locality and performance
characteristics.  Some memory may share the same node as a CPU, and others
are provided as memory only nodes. While memory only nodes do not provide
CPUs, they may still be local to one or more compute nodes relative to
other nodes. The following diagram shows one such example of two compute
nodes with local memory and a memory only node for each of compute node::

 +------------------+     +------------------+
 | Compute Node 0   +-----+ Compute Node 1   |
 | Local Node0 Mem  |     | Local Node1 Mem  |
 +--------+---------+     +--------+---------+
          |                        |
 +--------+---------+     +--------+---------+
 | Slower Node2 Mem |     | Slower Node3 Mem |
 +------------------+     +--------+---------+

A "memory initiator" is a node containing one or more devices such as
CPUs or separate memory I/O devices that can initiate memory requests.
A "memory target" is a node containing one or more physical address
ranges accessible from one or more memory initiators.

When multiple memory initiators exist, they may not all have the same
performance when accessing a given memory target. Each initiator-target
pair may be organized into different ranked access classes to represent
this relationship. The highest performing initiator to a given target
is considered to be one of that target's local initiators, and given
the highest access class, 0. Any given target may have one or more
local initiators, and any given initiator may have multiple local
memory targets.

To aid applications matching memory targets with their initiators, the
kernel provides symlinks to each other. The following example lists the
relationship for the access class "0" memory initiators and targets::

	# symlinks -v /sys/devices/system/node/nodeX/access0/targets/
	relative: /sys/devices/system/node/nodeX/access0/targets/nodeY -> ../../nodeY

	# symlinks -v /sys/devices/system/node/nodeY/access0/initiators/
	relative: /sys/devices/system/node/nodeY/access0/initiators/nodeX -> ../../nodeX

A memory initiator may have multiple memory targets in the same access
class. The target memory's initiators in a given class indicate the
nodes' access characteristics share the same performance relative to other
linked initiator nodes. Each target within an initiator's access class,
though, do not necessarily perform the same as each other.

The access class "1" is used to allow differentiation between initiators
that are CPUs and hence suitable for generic task scheduling, and
IO initiators such as GPUs and NICs.  Unlike access class 0, only
nodes containing CPUs are considered.

================
NUMA Performance
================

Applications may wish to consider which node they want their memory to
be allocated from based on the node's performance characteristics. If
the system provides these attributes, the kernel exports them under the
node sysfs hierarchy by appending the attributes directory under the
memory node's access class 0 initiators as follows::

	/sys/devices/system/node/nodeY/access0/initiators/

These attributes apply only when accessed from nodes that have the
are linked under the this access's initiators.

The performance characteristics the kernel provides for the local initiators
are exported are as follows::

	# tree -P "read*|write*" /sys/devices/system/node/nodeY/access0/initiators/
	/sys/devices/system/node/nodeY/access0/initiators/
	|-- read_bandwidth
	|-- read_latency
	|-- write_bandwidth
	`-- write_latency

The bandwidth attributes are provided in MiB/second.

The latency attributes are provided in nanoseconds.

The values reported here correspond to the rated latency and bandwidth
for the platform.

Access class 1 takes the same form but only includes values for CPU to
memory activity.

==========
NUMA Cache
==========

System memory may be constructed in a hierarchy of elements with various
performance characteristics in order to provide large address space of
slower performing memory cached by a smaller higher performing memory. The
system physical addresses memory  initiators are aware of are provided
by the last memory level in the hierarchy. The system meanwhile uses
higher performing memory to transparently cache access to progressively
slower levels.

The term "far memory" is used to denote the last level memory in the
hierarchy. Each increasing cache level provides higher performing
initiator access, and the term "near memory" represents the fastest
cache provided by the system.

This numbering is different than CPU caches where the cache level (ex:
L1, L2, L3) uses the CPU-side view where each increased level is lower
performing. In contrast, the memory cache level is centric to the last
level memory, so the higher numbered cache level corresponds to  memory
nearer to the CPU, and further from far memory.

The memory-side caches are not directly addressable by software. When
software accesses a system address, the system will return it from the
near memory cache if it is present. If it is not present, the system
accesses the next level of memory until there is either a hit in that
cache level, or it reaches far memory.

An application does not need to know about caching attributes in order
to use the system. Software may optionally query the memory cache
attributes in order to maximize the performance out of such a setup.
If the system provides a way for the kernel to discover this information,
for example with ACPI HMAT (Heterogeneous Memory Attribute Table),
the kernel will append these attributes to the NUMA node memory target.

When the kernel first registers a memory cache with a node, the kernel
will create the following directory::

	/sys/devices/system/node/nodeX/memory_side_cache/

If that directory is not present, the system either does not provide
a memory-side cache, or that information is not accessible to the kernel.

The attributes for each level of cache is provided under its cache
level index::

	/sys/devices/system/node/nodeX/memory_side_cache/indexA/
	/sys/devices/system/node/nodeX/memory_side_cache/indexB/
	/sys/devices/system/node/nodeX/memory_side_cache/indexC/

Each cache level's directory provides its attributes. For example, the
following shows a single cache level and the attributes available for
software to query::

	# tree /sys/devices/system/node/node0/memory_side_cache/
	/sys/devices/system/node/node0/memory_side_cache/
	|-- index1
	|   |-- indexing
	|   |-- line_size
	|   |-- size
	|   `-- write_policy

The "indexing" will be 0 if it is a direct-mapped cache, and non-zero
for any other indexed based, multi-way associativity.

The "line_size" is the number of bytes accessed from the next cache
level on a miss.

The "size" is the number of bytes provided by this cache level.

The "write_policy" will be 0 for write-back, and non-zero for
write-through caching.

========
See Also
========

[1] https://www.uefi.org/sites/default/files/resources/ACPI_6_2.pdf
- Section 5.2.27


code:
.. _pagemap:

=============================
Examining Process Page Tables
=============================

pagemap is a new (as of 2.6.25) set of interfaces in the kernel that allow
userspace programs to examine the page tables and related information by
reading files in ``/proc``.

There are four components to pagemap:

 * ``/proc/pid/pagemap``.  This file lets a userspace process find out which
   physical frame each virtual page is mapped to.  It contains one 64-bit
   value for each virtual page, containing the following data (from
   ``fs/proc/task_mmu.c``, above pagemap_read):

    * Bits 0-54  page frame number (PFN) if present
    * Bits 0-4   swap type if swapped
    * Bits 5-54  swap offset if swapped
    * Bit  55    pte is soft-dirty (see
      :ref:`Documentation/admin-guide/mm/soft-dirty.rst <soft_dirty>`)
    * Bit  56    page exclusively mapped (since 4.2)
    * Bit  57    pte is uffd-wp write-protected (since 5.13) (see
      :ref:`Documentation/admin-guide/mm/userfaultfd.rst <userfaultfd>`)
    * Bits 58-60 zero
    * Bit  61    page is file-page or shared-anon (since 3.5)
    * Bit  62    page swapped
    * Bit  63    page present

   Since Linux 4.0 only users with the CAP_SYS_ADMIN capability can get PFNs.
   In 4.0 and 4.1 opens by unprivileged fail with -EPERM.  Starting from
   4.2 the PFN field is zeroed if the user does not have CAP_SYS_ADMIN.
   Reason: information about PFNs helps in exploiting Rowhammer vulnerability.

   If the page is not present but in swap, then the PFN contains an
   encoding of the swap file number and the page's offset into the
   swap. Unmapped pages return a null PFN. This allows determining
   precisely which pages are mapped (or in swap) and comparing mapped
   pages between processes.

   Efficient users of this interface will use ``/proc/pid/maps`` to
   determine which areas of memory are actually mapped and llseek to
   skip over unmapped regions.

 * ``/proc/kpagecount``.  This file contains a 64-bit count of the number of
   times each page is mapped, indexed by PFN.

The page-types tool in the tools/vm directory can be used to query the
number of times a page is mapped.

 * ``/proc/kpageflags``.  This file contains a 64-bit set of flags for each
   page, indexed by PFN.

   The flags are (from ``fs/proc/page.c``, above kpageflags_read):

    0. LOCKED
    1. ERROR
    2. REFERENCED
    3. UPTODATE
    4. DIRTY
    5. LRU
    6. ACTIVE
    7. SLAB
    8. WRITEBACK
    9. RECLAIM
    10. BUDDY
    11. MMAP
    12. ANON
    13. SWAPCACHE
    14. SWAPBACKED
    15. COMPOUND_HEAD
    16. COMPOUND_TAIL
    17. HUGE
    18. UNEVICTABLE
    19. HWPOISON
    20. NOPAGE
    21. KSM
    22. THP
    23. OFFLINE
    24. ZERO_PAGE
    25. IDLE
    26. PGTABLE

 * ``/proc/kpagecgroup``.  This file contains a 64-bit inode number of the
   memory cgroup each page is charged to, indexed by PFN. Only available when
   CONFIG_MEMCG is set.

Short descriptions to the page flags
====================================

0 - LOCKED
   The page is being locked for exclusive access, e.g. by undergoing read/write
   IO.
7 - SLAB
   The page is managed by the SLAB/SLOB/SLUB/SLQB kernel memory allocator.
   When compound page is used, SLUB/SLQB will only set this flag on the head
   page; SLOB will not flag it at all.
10 - BUDDY
    A free memory block managed by the buddy system allocator.
    The buddy system organizes free memory in blocks of various orders.
    An order N block has 2^N physically contiguous pages, with the BUDDY flag
    set for and _only_ for the first page.
15 - COMPOUND_HEAD
    A compound page with order N consists of 2^N physically contiguous pages.
    A compound page with order 2 takes the form of "HTTT", where H donates its
    head page and T donates its tail page(s).  The major consumers of compound
    pages are hugeTLB pages
    (:ref:`Documentation/admin-guide/mm/hugetlbpage.rst <hugetlbpage>`),
    the SLUB etc.  memory allocators and various device drivers.
    However in this interface, only huge/giga pages are made visible
    to end users.
16 - COMPOUND_TAIL
    A compound page tail (see description above).
17 - HUGE
    This is an integral part of a HugeTLB page.
19 - HWPOISON
    Hardware detected memory corruption on this page: don't touch the data!
20 - NOPAGE
    No page frame exists at the requested address.
21 - KSM
    Identical memory pages dynamically shared between one or more processes.
22 - THP
    Contiguous pages which construct transparent hugepages.
23 - OFFLINE
    The page is logically offline.
24 - ZERO_PAGE
    Zero page for pfn_zero or huge_zero page.
25 - IDLE
    The page has not been accessed since it was marked idle (see
    :ref:`Documentation/admin-guide/mm/idle_page_tracking.rst <idle_page_tracking>`).
    Note that this flag may be stale in case the page was accessed via
    a PTE. To make sure the flag is up-to-date one has to read
    ``/sys/kernel/mm/page_idle/bitmap`` first.
26 - PGTABLE
    The page is in use as a page table.

IO related page flags
---------------------

1 - ERROR
   IO error occurred.
3 - UPTODATE
   The page has up-to-date data.
   ie. for file backed page: (in-memory data revision >= on-disk one)
4 - DIRTY
   The page has been written to, hence contains new data.
   i.e. for file backed page: (in-memory data revision >  on-disk one)
8 - WRITEBACK
   The page is being synced to disk.

LRU related page flags
----------------------

5 - LRU
   The page is in one of the LRU lists.
6 - ACTIVE
   The page is in the active LRU list.
18 - UNEVICTABLE
   The page is in the unevictable (non-)LRU list It is somehow pinned and
   not a candidate for LRU page reclaims, e.g. ramfs pages,
   shmctl(SHM_LOCK) and mlock() memory segments.
2 - REFERENCED
   The page has been referenced since last LRU list enqueue/requeue.
9 - RECLAIM
   The page will be reclaimed soon after its pageout IO completed.
11 - MMAP
   A memory mapped page.
12 - ANON
   A memory mapped page that is not part of a file.
13 - SWAPCACHE
   The page is mapped to swap space, i.e. has an associated swap entry.
14 - SWAPBACKED
   The page is backed by swap/RAM.

The page-types tool in the tools/vm directory can be used to query the
above flags.

Using pagemap to do something useful
====================================

The general procedure for using pagemap to find out about a process' memory
usage goes like this:

 1. Read ``/proc/pid/maps`` to determine which parts of the memory space are
    mapped to what.
 2. Select the maps you are interested in -- all of them, or a particular
    library, or the stack or the heap, etc.
 3. Open ``/proc/pid/pagemap`` and seek to the pages you would like to examine.
 4. Read a u64 for each page from pagemap.
 5. Open ``/proc/kpagecount`` and/or ``/proc/kpageflags``.  For each PFN you
    just read, seek to that entry in the file, and read the data you want.

For example, to find the "unique set size" (USS), which is the amount of
memory that a process is using that is not shared with any other process,
you can go through every map in the process, find the PFNs, look those up
in kpagecount, and tally up the number of pages that are only referenced
once.

Exceptions for Shared Memory
============================

Page table entries for shared pages are cleared when the pages are zapped or
swapped out. This makes swapped out pages indistinguishable from never-allocated
ones.

In kernel space, the swap location can still be retrieved from the page cache.
However, values stored only on the normal PTE get lost irretrievably when the
page is swapped out (i.e. SOFT_DIRTY).

In user space, whether the page is present, swapped or none can be deduced with
the help of lseek and/or mincore system calls.

lseek() can differentiate between accessed pages (present or swapped out) and
holes (none/non-allocated) by specifying the SEEK_DATA flag on the file where
the pages are backed. For anonymous shared pages, the file can be found in
``/proc/pid/map_files/``.

mincore() can differentiate between pages in memory (present, including swap
cache) and out of memory (swapped out or none/non-allocated).

Other notes
===========

Reading from any of the files will return -EINVAL if you are not starting
the read on an 8-byte boundary (e.g., if you sought an odd number of bytes
into the file), or if the size of the read is not a multiple of 8 bytes.

Before Linux 3.11 pagemap bits 55-60 were used for "page-shift" (which is
always 12 at most architectures). Since Linux 3.11 their meaning changes
after first clear of soft-dirty bits. Since Linux 4.2 they are used for
flags unconditionally.


code:
.. _shrinker_debugfs:

==========================
Shrinker Debugfs Interface
==========================

Shrinker debugfs interface provides a visibility into the kernel memory
shrinkers subsystem and allows to get information about individual shrinkers
and interact with them.

For each shrinker registered in the system a directory in **<debugfs>/shrinker/**
is created. The directory's name is composed from the shrinker's name and an
unique id: e.g. *kfree_rcu-0* or *sb-xfs:vda1-36*.

Each shrinker directory contains **count** and **scan** files, which allow to
trigger *count_objects()* and *scan_objects()* callbacks for each memcg and
numa node (if applicable).

Usage:
------

1. *List registered shrinkers*

  ::

    $ cd /sys/kernel/debug/shrinker/
    $ ls
    dquota-cache-16     sb-devpts-28     sb-proc-47       sb-tmpfs-42
    mm-shadow-18        sb-devtmpfs-5    sb-proc-48       sb-tmpfs-43
    mm-zspool:zram0-34  sb-hugetlbfs-17  sb-pstore-31     sb-tmpfs-44
    rcu-kfree-0         sb-hugetlbfs-33  sb-rootfs-2      sb-tmpfs-49
    sb-aio-20           sb-iomem-12      sb-securityfs-6  sb-tracefs-13
    sb-anon_inodefs-15  sb-mqueue-21     sb-selinuxfs-22  sb-xfs:vda1-36
    sb-bdev-3           sb-nsfs-4        sb-sockfs-8      sb-zsmalloc-19
    sb-bpf-32           sb-pipefs-14     sb-sysfs-26      thp-deferred_split-10
    sb-btrfs:vda2-24    sb-proc-25       sb-tmpfs-1       thp-zero-9
    sb-cgroup2-30       sb-proc-39       sb-tmpfs-27      xfs-buf:vda1-37
    sb-configfs-23      sb-proc-41       sb-tmpfs-29      xfs-inodegc:vda1-38
    sb-dax-11           sb-proc-45       sb-tmpfs-35
    sb-debugfs-7        sb-proc-46       sb-tmpfs-40

2. *Get information about a specific shrinker*

  ::

    $ cd sb-btrfs\:vda2-24/
    $ ls
    count            scan

3. *Count objects*

  Each line in the output has the following format::

    <cgroup inode id> <nr of objects on node 0> <nr of objects on node 1> ...
    <cgroup inode id> <nr of objects on node 0> <nr of objects on node 1> ...
    ...

  If there are no objects on all numa nodes, a line is omitted. If there
  are no objects at all, the output might be empty.

  If the shrinker is not memcg-aware or CONFIG_MEMCG is off, 0 is printed
  as cgroup inode id. If the shrinker is not numa-aware, 0's are printed
  for all nodes except the first one.
  ::

    $ cat count
    1 224 2
    21 98 0
    55 818 10
    2367 2 0
    2401 30 0
    225 13 0
    599 35 0
    939 124 0
    1041 3 0
    1075 1 0
    1109 1 0
    1279 60 0
    1313 7 0
    1347 39 0
    1381 3 0
    1449 14 0
    1483 63 0
    1517 53 0
    1551 6 0
    1585 1 0
    1619 6 0
    1653 40 0
    1687 11 0
    1721 8 0
    1755 4 0
    1789 52 0
    1823 888 0
    1857 1 0
    1925 2 0
    1959 32 0
    2027 22 0
    2061 9 0
    2469 799 0
    2537 861 0
    2639 1 0
    2707 70 0
    2775 4 0
    2877 84 0
    293 1 0
    735 8 0

4. *Scan objects*

  The expected input format::

    <cgroup inode id> <numa id> <number of objects to scan>

  For a non-memcg-aware shrinker or on a system with no memory
  cgrups **0** should be passed as cgroup id.
  ::

    $ cd /sys/kernel/debug/shrinker/
    $ cd sb-btrfs\:vda2-24/

    $ cat count | head -n 5
    1 212 0
    21 97 0
    55 802 5
    2367 2 0
    225 13 0

    $ echo "55 0 200" > scan

    $ cat count | head -n 5
    1 212 0
    21 96 0
    55 752 5
    2367 2 0
    225 13 0


code:
.. _soft_dirty:

===============
Soft-Dirty PTEs
===============

The soft-dirty is a bit on a PTE which helps to track which pages a task
writes to. In order to do this tracking one should

  1. Clear soft-dirty bits from the task's PTEs.

     This is done by writing "4" into the ``/proc/PID/clear_refs`` file of the
     task in question.

  2. Wait some time.

  3. Read soft-dirty bits from the PTEs.

     This is done by reading from the ``/proc/PID/pagemap``. The bit 55 of the
     64-bit qword is the soft-dirty one. If set, the respective PTE was
     written to since step 1.

Internally, to do this tracking, the writable bit is cleared from PTEs
when the soft-dirty bit is cleared. So, after this, when the task tries to
modify a page at some virtual address the #PF occurs and the kernel sets
the soft-dirty bit on the respective PTE.

Note, that although all the task's address space is marked as r/o after the
soft-dirty bits clear, the #PF-s that occur after that are processed fast.
This is so, since the pages are still mapped to physical memory, and thus all
the kernel does is finds this fact out and puts both writable and soft-dirty
bits on the PTE.

While in most cases tracking memory changes by #PF-s is more than enough
there is still a scenario when we can lose soft dirty bits -- a task
unmaps a previously mapped memory region and then maps a new one at exactly
the same place. When unmap is called, the kernel internally clears PTE values
including soft dirty bits. To notify user space application about such
memory region renewal the kernel always marks new memory regions (and
expanded regions) as soft dirty.

This feature is actively used by the checkpoint-restore project. You
can find more details about it on http://criu.org

-- Pavel Emelyanov, Apr 9, 2013


code:
.. _swap_numa:

===========================================
Automatically bind swap device to numa node
===========================================

If the system has more than one swap device and swap device has the node
information, we can make use of this information to decide which swap
device to use in get_swap_pages() to get better performance.

How to use this feature
=======================

Swap device has priority and that decides the order of it to be used. To make
use of automatically binding, there is no need to manipulate priority settings
for swap devices. e.g. on a 2 node machine, assume 2 swap devices swapA and
swapB, with swapA attached to node 0 and swapB attached to node 1, are going
to be swapped on. Simply swapping them on by doing::

	# swapon /dev/swapA
	# swapon /dev/swapB

Then node 0 will use the two swap devices in the order of swapA then swapB and
node 1 will use the two swap devices in the order of swapB then swapA. Note
that the order of them being swapped on doesn't matter.

A more complex example on a 4 node machine. Assume 6 swap devices are going to
be swapped on: swapA and swapB are attached to node 0, swapC is attached to
node 1, swapD and swapE are attached to node 2 and swapF is attached to node3.
The way to swap them on is the same as above::

	# swapon /dev/swapA
	# swapon /dev/swapB
	# swapon /dev/swapC
	# swapon /dev/swapD
	# swapon /dev/swapE
	# swapon /dev/swapF

Then node 0 will use them in the order of::

	swapA/swapB -> swapC -> swapD -> swapE -> swapF

swapA and swapB will be used in a round robin mode before any other swap device.

node 1 will use them in the order of::

	swapC -> swapA -> swapB -> swapD -> swapE -> swapF

node 2 will use them in the order of::

	swapD/swapE -> swapA -> swapB -> swapC -> swapF

Similaly, swapD and swapE will be used in a round robin mode before any
other swap devices.

node 3 will use them in the order of::

	swapF -> swapA -> swapB -> swapC -> swapD -> swapE

Implementation details
======================

The current code uses a priority based list, swap_avail_list, to decide
which swap device to use and if multiple swap devices share the same
priority, they are used round robin. This change here replaces the single
global swap_avail_list with a per-numa-node list, i.e. for each numa node,
it sees its own priority based list of available swap devices. Swap
device's priority can be promoted on its matching node's swap_avail_list.

The current swap device's priority is set as: user can set a >=0 value,
or the system will pick one starting from -1 then downwards. The priority
value in the swap_avail_list is the negated value of the swap device's
due to plist being sorted from low to high. The new policy doesn't change
the semantics for priority >=0 cases, the previous starting from -1 then
downwards now becomes starting from -2 then downwards and -1 is reserved
as the promoted value. So if multiple swap devices are attached to the same
node, they will all be promoted to priority -1 on that node's plist and will
be used round robin before any other swap devices.


code:
.. _admin_guide_transhuge:

============================
Transparent Hugepage Support
============================

Objective
=========

Performance critical computing applications dealing with large memory
working sets are already running on top of libhugetlbfs and in turn
hugetlbfs. Transparent HugePage Support (THP) is an alternative mean of
using huge pages for the backing of virtual memory with huge pages
that supports the automatic promotion and demotion of page sizes and
without the shortcomings of hugetlbfs.

Currently THP only works for anonymous memory mappings and tmpfs/shmem.
But in the future it can expand to other filesystems.

.. note::
   in the examples below we presume that the basic page size is 4K and
   the huge page size is 2M, although the actual numbers may vary
   depending on the CPU architecture.

The reason applications are running faster is because of two
factors. The first factor is almost completely irrelevant and it's not
of significant interest because it'll also have the downside of
requiring larger clear-page copy-page in page faults which is a
potentially negative effect. The first factor consists in taking a
single page fault for each 2M virtual region touched by userland (so
reducing the enter/exit kernel frequency by a 512 times factor). This
only matters the first time the memory is accessed for the lifetime of
a memory mapping. The second long lasting and much more important
factor will affect all subsequent accesses to the memory for the whole
runtime of the application. The second factor consist of two
components:

1) the TLB miss will run faster (especially with virtualization using
   nested pagetables but almost always also on bare metal without
   virtualization)

2) a single TLB entry will be mapping a much larger amount of virtual
   memory in turn reducing the number of TLB misses. With
   virtualization and nested pagetables the TLB can be mapped of
   larger size only if both KVM and the Linux guest are using
   hugepages but a significant speedup already happens if only one of
   the two is using hugepages just because of the fact the TLB miss is
   going to run faster.

THP can be enabled system wide or restricted to certain tasks or even
memory ranges inside task's address space. Unless THP is completely
disabled, there is ``khugepaged`` daemon that scans memory and
collapses sequences of basic pages into huge pages.

The THP behaviour is controlled via :ref:`sysfs <thp_sysfs>`
interface and using madvise(2) and prctl(2) system calls.

Transparent Hugepage Support maximizes the usefulness of free memory
if compared to the reservation approach of hugetlbfs by allowing all
unused memory to be used as cache or other movable (or even unmovable
entities). It doesn't require reservation to prevent hugepage
allocation failures to be noticeable from userland. It allows paging
and all other advanced VM features to be available on the
hugepages. It requires no modifications for applications to take
advantage of it.

Applications however can be further optimized to take advantage of
this feature, like for example they've been optimized before to avoid
a flood of mmap system calls for every malloc(4k). Optimizing userland
is by far not mandatory and khugepaged already can take care of long
lived page allocations even for hugepage unaware applications that
deals with large amounts of memory.

In certain cases when hugepages are enabled system wide, application
may end up allocating more memory resources. An application may mmap a
large region but only touch 1 byte of it, in that case a 2M page might
be allocated instead of a 4k page for no good. This is why it's
possible to disable hugepages system-wide and to only have them inside
MADV_HUGEPAGE madvise regions.

Embedded systems should enable hugepages only inside madvise regions
to eliminate any risk of wasting any precious byte of memory and to
only run faster.

Applications that gets a lot of benefit from hugepages and that don't
risk to lose memory by using hugepages, should use
madvise(MADV_HUGEPAGE) on their critical mmapped regions.

.. _thp_sysfs:

sysfs
=====

Global THP controls
-------------------

Transparent Hugepage Support for anonymous memory can be entirely disabled
(mostly for debugging purposes) or only enabled inside MADV_HUGEPAGE
regions (to avoid the risk of consuming more memory resources) or enabled
system wide. This can be achieved with one of::

	echo always >/sys/kernel/mm/transparent_hugepage/enabled
	echo madvise >/sys/kernel/mm/transparent_hugepage/enabled
	echo never >/sys/kernel/mm/transparent_hugepage/enabled

It's also possible to limit defrag efforts in the VM to generate
anonymous hugepages in case they're not immediately free to madvise
regions or to never try to defrag memory and simply fallback to regular
pages unless hugepages are immediately available. Clearly if we spend CPU
time to defrag memory, we would expect to gain even more by the fact we
use hugepages later instead of regular pages. This isn't always
guaranteed, but it may be more likely in case the allocation is for a
MADV_HUGEPAGE region.

::

	echo always >/sys/kernel/mm/transparent_hugepage/defrag
	echo defer >/sys/kernel/mm/transparent_hugepage/defrag
	echo defer+madvise >/sys/kernel/mm/transparent_hugepage/defrag
	echo madvise >/sys/kernel/mm/transparent_hugepage/defrag
	echo never >/sys/kernel/mm/transparent_hugepage/defrag

always
	means that an application requesting THP will stall on
	allocation failure and directly reclaim pages and compact
	memory in an effort to allocate a THP immediately. This may be
	desirable for virtual machines that benefit heavily from THP
	use and are willing to delay the VM start to utilise them.

defer
	means that an application will wake kswapd in the background
	to reclaim pages and wake kcompactd to compact memory so that
	THP is available in the near future. It's the responsibility
	of khugepaged to then install the THP pages later.

defer+madvise
	will enter direct reclaim and compaction like ``always``, but
	only for regions that have used madvise(MADV_HUGEPAGE); all
	other regions will wake kswapd in the background to reclaim
	pages and wake kcompactd to compact memory so that THP is
	available in the near future.

madvise
	will enter direct reclaim like ``always`` but only for regions
	that are have used madvise(MADV_HUGEPAGE). This is the default
	behaviour.

never
	should be self-explanatory.

By default kernel tries to use huge zero page on read page fault to
anonymous mapping. It's possible to disable huge zero page by writing 0
or enable it back by writing 1::

	echo 0 >/sys/kernel/mm/transparent_hugepage/use_zero_page
	echo 1 >/sys/kernel/mm/transparent_hugepage/use_zero_page

Some userspace (such as a test program, or an optimized memory allocation
library) may want to know the size (in bytes) of a transparent hugepage::

	cat /sys/kernel/mm/transparent_hugepage/hpage_pmd_size

khugepaged will be automatically started when
transparent_hugepage/enabled is set to "always" or "madvise, and it'll
be automatically shutdown if it's set to "never".

Khugepaged controls
-------------------

khugepaged runs usually at low frequency so while one may not want to
invoke defrag algorithms synchronously during the page faults, it
should be worth invoking defrag at least in khugepaged. However it's
also possible to disable defrag in khugepaged by writing 0 or enable
defrag in khugepaged by writing 1::

	echo 0 >/sys/kernel/mm/transparent_hugepage/khugepaged/defrag
	echo 1 >/sys/kernel/mm/transparent_hugepage/khugepaged/defrag

You can also control how many pages khugepaged should scan at each
pass::

	/sys/kernel/mm/transparent_hugepage/khugepaged/pages_to_scan

and how many milliseconds to wait in khugepaged between each pass (you
can set this to 0 to run khugepaged at 100% utilization of one core)::

	/sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs

and how many milliseconds to wait in khugepaged if there's an hugepage
allocation failure to throttle the next allocation attempt::

	/sys/kernel/mm/transparent_hugepage/khugepaged/alloc_sleep_millisecs

The khugepaged progress can be seen in the number of pages collapsed (note
that this counter may not be an exact count of the number of pages
collapsed, since "collapsed" could mean multiple things: (1) A PTE mapping
being replaced by a PMD mapping, or (2) All 4K physical pages replaced by
one 2M hugepage. Each may happen independently, or together, depending on
the type of memory and the failures that occur. As such, this value should
be interpreted roughly as a sign of progress, and counters in /proc/vmstat
consulted for more accurate accounting)::

	/sys/kernel/mm/transparent_hugepage/khugepaged/pages_collapsed

for each pass::

	/sys/kernel/mm/transparent_hugepage/khugepaged/full_scans

``max_ptes_none`` specifies how many extra small pages (that are
not already mapped) can be allocated when collapsing a group
of small pages into one large page::

	/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none

A higher value leads to use additional memory for programs.
A lower value leads to gain less thp performance. Value of
max_ptes_none can waste cpu time very little, you can
ignore it.

``max_ptes_swap`` specifies how many pages can be brought in from
swap when collapsing a group of pages into a transparent huge page::

	/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_swap

A higher value can cause excessive swap IO and waste
memory. A lower value can prevent THPs from being
collapsed, resulting fewer pages being collapsed into
THPs, and lower memory access performance.

``max_ptes_shared`` specifies how many pages can be shared across multiple
processes. Exceeding the number would block the collapse::

	/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_shared

A higher value may increase memory footprint for some workloads.

Boot parameter
==============

You can change the sysfs boot time defaults of Transparent Hugepage
Support by passing the parameter ``transparent_hugepage=always`` or
``transparent_hugepage=madvise`` or ``transparent_hugepage=never``
to the kernel command line.

Hugepages in tmpfs/shmem
========================

You can control hugepage allocation policy in tmpfs with mount option
``huge=``. It can have following values:

always
    Attempt to allocate huge pages every time we need a new page;

never
    Do not allocate huge pages;

within_size
    Only allocate huge page if it will be fully within i_size.
    Also respect fadvise()/madvise() hints;

advise
    Only allocate huge pages if requested with fadvise()/madvise();

The default policy is ``never``.

``mount -o remount,huge= /mountpoint`` works fine after mount: remounting
``huge=never`` will not attempt to break up huge pages at all, just stop more
from being allocated.

There's also sysfs knob to control hugepage allocation policy for internal
shmem mount: /sys/kernel/mm/transparent_hugepage/shmem_enabled. The mount
is used for SysV SHM, memfds, shared anonymous mmaps (of /dev/zero or
MAP_ANONYMOUS), GPU drivers' DRM objects, Ashmem.

In addition to policies listed above, shmem_enabled allows two further
values:

deny
    For use in emergencies, to force the huge option off from
    all mounts;
force
    Force the huge option on for all - very useful for testing;

Need of application restart
===========================

The transparent_hugepage/enabled values and tmpfs mount option only affect
future behavior. So to make them effective you need to restart any
application that could have been using hugepages. This also applies to the
regions registered in khugepaged.

Monitoring usage
================

The number of anonymous transparent huge pages currently used by the
system is available by reading the AnonHugePages field in ``/proc/meminfo``.
To identify what applications are using anonymous transparent huge pages,
it is necessary to read ``/proc/PID/smaps`` and count the AnonHugePages fields
for each mapping.

The number of file transparent huge pages mapped to userspace is available
by reading ShmemPmdMapped and ShmemHugePages fields in ``/proc/meminfo``.
To identify what applications are mapping file transparent huge pages, it
is necessary to read ``/proc/PID/smaps`` and count the FileHugeMapped fields
for each mapping.

Note that reading the smaps file is expensive and reading it
frequently will incur overhead.

There are a number of counters in ``/proc/vmstat`` that may be used to
monitor how successfully the system is providing huge pages for use.

thp_fault_alloc
	is incremented every time a huge page is successfully
	allocated to handle a page fault.

thp_collapse_alloc
	is incremented by khugepaged when it has found
	a range of pages to collapse into one huge page and has
	successfully allocated a new huge page to store the data.

thp_fault_fallback
	is incremented if a page fault fails to allocate
	a huge page and instead falls back to using small pages.

thp_fault_fallback_charge
	is incremented if a page fault fails to charge a huge page and
	instead falls back to using small pages even though the
	allocation was successful.

thp_collapse_alloc_failed
	is incremented if khugepaged found a range
	of pages that should be collapsed into one huge page but failed
	the allocation.

thp_file_alloc
	is incremented every time a file huge page is successfully
	allocated.

thp_file_fallback
	is incremented if a file huge page is attempted to be allocated
	but fails and instead falls back to using small pages.

thp_file_fallback_charge
	is incremented if a file huge page cannot be charged and instead
	falls back to using small pages even though the allocation was
	successful.

thp_file_mapped
	is incremented every time a file huge page is mapped into
	user address space.

thp_split_page
	is incremented every time a huge page is split into base
	pages. This can happen for a variety of reasons but a common
	reason is that a huge page is old and is being reclaimed.
	This action implies splitting all PMD the page mapped with.

thp_split_page_failed
	is incremented if kernel fails to split huge
	page. This can happen if the page was pinned by somebody.

thp_deferred_split_page
	is incremented when a huge page is put onto split
	queue. This happens when a huge page is partially unmapped and
	splitting it would free up some memory. Pages on split queue are
	going to be split under memory pressure.

thp_split_pmd
	is incremented every time a PMD split into table of PTEs.
	This can happen, for instance, when application calls mprotect() or
	munmap() on part of huge page. It doesn't split huge page, only
	page table entry.

thp_zero_page_alloc
	is incremented every time a huge zero page used for thp is
	successfully allocated. Note, it doesn't count every map of
	the huge zero page, only its allocation.

thp_zero_page_alloc_failed
	is incremented if kernel fails to allocate
	huge zero page and falls back to using small pages.

thp_swpout
	is incremented every time a huge page is swapout in one
	piece without splitting.

thp_swpout_fallback
	is incremented if a huge page has to be split before swapout.
	Usually because failed to allocate some continuous swap space
	for the huge page.

As the system ages, allocating huge pages may be expensive as the
system uses memory compaction to copy data around memory to free a
huge page for use. There are some counters in ``/proc/vmstat`` to help
monitor this overhead.

compact_stall
	is incremented every time a process stalls to run
	memory compaction so that a huge page is free for use.

compact_success
	is incremented if the system compacted memory and
	freed a huge page for use.

compact_fail
	is incremented if the system tries to compact memory
	but failed.

It is possible to establish how long the stalls were using the function
tracer to record how long was spent in __alloc_pages() and
using the mm_page_alloc tracepoint to identify which allocations were
for huge pages.

Optimizing the applications
===========================

To be guaranteed that the kernel will map a 2M page immediately in any
memory region, the mmap region has to be hugepage naturally
aligned. posix_memalign() can provide that guarantee.

Hugetlbfs
=========

You can use hugetlbfs on a kernel that has transparent hugepage
support enabled just fine as always. No difference can be noted in
hugetlbfs other than there will be less overall fragmentation. All
usual features belonging to hugetlbfs are preserved and
unaffected. libhugetlbfs will also work fine as usual.