code:
.. SPDX-License-Identifier: GPL-2.0
.. include:: <isonum.txt>

.. |struct cpuidle_state| replace:: :c:type:`struct cpuidle_state <cpuidle_state>`
.. |cpufreq| replace:: :doc:`CPU Performance Scaling <cpufreq>`

========================
CPU Idle Time Management
========================

:Copyright: |copy| 2018 Intel Corporation

:Author: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

Concepts
========

Modern processors are generally able to enter states in which the execution of
a program is suspended and instructions belonging to it are not fetched from
memory or executed.  Those states are the *idle* states of the processor.

Since part of the processor hardware is not used in idle states, entering them
generally allows power drawn by the processor to be reduced and, in consequence,
it is an opportunity to save energy.

CPU idle time management is an energy-efficiency feature concerned about using
the idle states of processors for this purpose.

Logical CPUs
------------

CPU idle time management operates on CPUs as seen by the *CPU scheduler* (that
is the part of the kernel responsible for the distribution of computational
work in the system).  In its view, CPUs are *logical* units.  That is, they need
not be separate physical entities and may just be interfaces appearing to
software as individual single-core processors.  In other words, a CPU is an
entity which appears to be fetching instructions that belong to one sequence
(program) from memory and executing them, but it need not work this way
physically.  Generally, three different cases can be consider here.

First, if the whole processor can only follow one sequence of instructions (one
program) at a time, it is a CPU.  In that case, if the hardware is asked to
enter an idle state, that applies to the processor as a whole.

Second, if the processor is multi-core, each core in it is able to follow at
least one program at a time.  The cores need not be entirely independent of each
other (for example, they may share caches), but still most of the time they
work physically in parallel with each other, so if each of them executes only
one program, those programs run mostly independently of each other at the same
time.  The entire cores are CPUs in that case and if the hardware is asked to
enter an idle state, that applies to the core that asked for it in the first
place, but it also may apply to a larger unit (say a "package" or a "cluster")
that the core belongs to (in fact, it may apply to an entire hierarchy of larger
units containing the core).  Namely, if all of the cores in the larger unit
except for one have been put into idle states at the "core level" and the
remaining core asks the processor to enter an idle state, that may trigger it
to put the whole larger unit into an idle state which also will affect the
other cores in that unit.

Finally, each core in a multi-core processor may be able to follow more than one
program in the same time frame (that is, each core may be able to fetch
instructions from multiple locations in memory and execute them in the same time
frame, but not necessarily entirely in parallel with each other).  In that case
the cores present themselves to software as "bundles" each consisting of
multiple individual single-core "processors", referred to as *hardware threads*
(or hyper-threads specifically on Intel hardware), that each can follow one
sequence of instructions.  Then, the hardware threads are CPUs from the CPU idle
time management perspective and if the processor is asked to enter an idle state
by one of them, the hardware thread (or CPU) that asked for it is stopped, but
nothing more happens, unless all of the other hardware threads within the same
core also have asked the processor to enter an idle state.  In that situation,
the core may be put into an idle state individually or a larger unit containing
it may be put into an idle state as a whole (if the other cores within the
larger unit are in idle states already).

Idle CPUs
---------

Logical CPUs, simply referred to as "CPUs" in what follows, are regarded as
*idle* by the Linux kernel when there are no tasks to run on them except for the
special "idle" task.

Tasks are the CPU scheduler's representation of work.  Each task consists of a
sequence of instructions to execute, or code, data to be manipulated while
running that code, and some context information that needs to be loaded into the
processor every time the task's code is run by a CPU.  The CPU scheduler
distributes work by assigning tasks to run to the CPUs present in the system.

Tasks can be in various states.  In particular, they are *runnable* if there are
no specific conditions preventing their code from being run by a CPU as long as
there is a CPU available for that (for example, they are not waiting for any
events to occur or similar).  When a task becomes runnable, the CPU scheduler
assigns it to one of the available CPUs to run and if there are no more runnable
tasks assigned to it, the CPU will load the given task's context and run its
code (from the instruction following the last one executed so far, possibly by
another CPU).  [If there are multiple runnable tasks assigned to one CPU
simultaneously, they will be subject to prioritization and time sharing in order
to allow them to make some progress over time.]

The special "idle" task becomes runnable if there are no other runnable tasks
assigned to the given CPU and the CPU is then regarded as idle.  In other words,
in Linux idle CPUs run the code of the "idle" task called *the idle loop*.  That
code may cause the processor to be put into one of its idle states, if they are
supported, in order to save energy, but if the processor does not support any
idle states, or there is not enough time to spend in an idle state before the
next wakeup event, or there are strict latency constraints preventing any of the
available idle states from being used, the CPU will simply execute more or less
useless instructions in a loop until it is assigned a new task to run.

.. _idle-loop:

The Idle Loop
=============

The idle loop code takes two major steps in every iteration of it.  First, it
calls into a code module referred to as the *governor* that belongs to the CPU
idle time management subsystem called ``CPUIdle`` to select an idle state for
the CPU to ask the hardware to enter.  Second, it invokes another code module
from the ``CPUIdle`` subsystem, called the *driver*, to actually ask the
processor hardware to enter the idle state selected by the governor.

The role of the governor is to find an idle state most suitable for the
conditions at hand.  For this purpose, idle states that the hardware can be
asked to enter by logical CPUs are represented in an abstract way independent of
the platform or the processor architecture and organized in a one-dimensional
(linear) array.  That array has to be prepared and supplied by the ``CPUIdle``
driver matching the platform the kernel is running on at the initialization
time.  This allows ``CPUIdle`` governors to be independent of the underlying
hardware and to work with any platforms that the Linux kernel can run on.

Each idle state present in that array is characterized by two parameters to be
taken into account by the governor, the *target residency* and the (worst-case)
*exit latency*.  The target residency is the minimum time the hardware must
spend in the given state, including the time needed to enter it (which may be
substantial), in order to save more energy than it would save by entering one of
the shallower idle states instead.  [The "depth" of an idle state roughly
corresponds to the power drawn by the processor in that state.]  The exit
latency, in turn, is the maximum time it will take a CPU asking the processor
hardware to enter an idle state to start executing the first instruction after a
wakeup from that state.  Note that in general the exit latency also must cover
the time needed to enter the given state in case the wakeup occurs when the
hardware is entering it and it must be entered completely to be exited in an
ordered manner.

There are two types of information that can influence the governor's decisions.
First of all, the governor knows the time until the closest timer event.  That
time is known exactly, because the kernel programs timers and it knows exactly
when they will trigger, and it is the maximum time the hardware that the given
CPU depends on can spend in an idle state, including the time necessary to enter
and exit it.  However, the CPU may be woken up by a non-timer event at any time
(in particular, before the closest timer triggers) and it generally is not known
when that may happen.  The governor can only see how much time the CPU actually
was idle after it has been woken up (that time will be referred to as the *idle
duration* from now on) and it can use that information somehow along with the
time until the closest timer to estimate the idle duration in future.  How the
governor uses that information depends on what algorithm is implemented by it
and that is the primary reason for having more than one governor in the
``CPUIdle`` subsystem.

There are four ``CPUIdle`` governors available, ``menu``, `TEO <teo-gov_>`_,
``ladder`` and ``haltpoll``.  Which of them is used by default depends on the
configuration of the kernel and in particular on whether or not the scheduler
tick can be `stopped by the idle loop <idle-cpus-and-tick_>`_.  Available
governors can be read from the :file:`available_governors`, and the governor
can be changed at runtime.  The name of the ``CPUIdle`` governor currently
used by the kernel can be read from the :file:`current_governor_ro` or
:file:`current_governor` file under :file:`/sys/devices/system/cpu/cpuidle/`
in ``sysfs``.

Which ``CPUIdle`` driver is used, on the other hand, usually depends on the
platform the kernel is running on, but there are platforms with more than one
matching driver.  For example, there are two drivers that can work with the
majority of Intel platforms, ``intel_idle`` and ``acpi_idle``, one with
hardcoded idle states information and the other able to read that information
from the system's ACPI tables, respectively.  Still, even in those cases, the
driver chosen at the system initialization time cannot be replaced later, so the
decision on which one of them to use has to be made early (on Intel platforms
the ``acpi_idle`` driver will be used if ``intel_idle`` is disabled for some
reason or if it does not recognize the processor).  The name of the ``CPUIdle``
driver currently used by the kernel can be read from the :file:`current_driver`
file under :file:`/sys/devices/system/cpu/cpuidle/` in ``sysfs``.

.. _idle-cpus-and-tick:

Idle CPUs and The Scheduler Tick
================================

The scheduler tick is a timer that triggers periodically in order to implement
the time sharing strategy of the CPU scheduler.  Of course, if there are
multiple runnable tasks assigned to one CPU at the same time, the only way to
allow them to make reasonable progress in a given time frame is to make them
share the available CPU time.  Namely, in rough approximation, each task is
given a slice of the CPU time to run its code, subject to the scheduling class,
prioritization and so on and when that time slice is used up, the CPU should be
switched over to running (the code of) another task.  The currently running task
may not want to give the CPU away voluntarily, however, and the scheduler tick
is there to make the switch happen regardless.  That is not the only role of the
tick, but it is the primary reason for using it.

The scheduler tick is problematic from the CPU idle time management perspective,
because it triggers periodically and relatively often (depending on the kernel
configuration, the length of the tick period is between 1 ms and 10 ms).
Thus, if the tick is allowed to trigger on idle CPUs, it will not make sense
for them to ask the hardware to enter idle states with target residencies above
the tick period length.  Moreover, in that case the idle duration of any CPU
will never exceed the tick period length and the energy used for entering and
exiting idle states due to the tick wakeups on idle CPUs will be wasted.

Fortunately, it is not really necessary to allow the tick to trigger on idle
CPUs, because (by definition) they have no tasks to run except for the special
"idle" one.  In other words, from the CPU scheduler perspective, the only user
of the CPU time on them is the idle loop.  Since the time of an idle CPU need
not be shared between multiple runnable tasks, the primary reason for using the
tick goes away if the given CPU is idle.  Consequently, it is possible to stop
the scheduler tick entirely on idle CPUs in principle, even though that may not
always be worth the effort.

Whether or not it makes sense to stop the scheduler tick in the idle loop
depends on what is expected by the governor.  First, if there is another
(non-tick) timer due to trigger within the tick range, stopping the tick clearly
would be a waste of time, even though the timer hardware may not need to be
reprogrammed in that case.  Second, if the governor is expecting a non-timer
wakeup within the tick range, stopping the tick is not necessary and it may even
be harmful.  Namely, in that case the governor will select an idle state with
the target residency within the time until the expected wakeup, so that state is
going to be relatively shallow.  The governor really cannot select a deep idle
state then, as that would contradict its own expectation of a wakeup in short
order.  Now, if the wakeup really occurs shortly, stopping the tick would be a
waste of time and in this case the timer hardware would need to be reprogrammed,
which is expensive.  On the other hand, if the tick is stopped and the wakeup
does not occur any time soon, the hardware may spend indefinite amount of time
in the shallow idle state selected by the governor, which will be a waste of
energy.  Hence, if the governor is expecting a wakeup of any kind within the
tick range, it is better to allow the tick trigger.  Otherwise, however, the
governor will select a relatively deep idle state, so the tick should be stopped
so that it does not wake up the CPU too early.

In any case, the governor knows what it is expecting and the decision on whether
or not to stop the scheduler tick belongs to it.  Still, if the tick has been
stopped already (in one of the previous iterations of the loop), it is better
to leave it as is and the governor needs to take that into account.

The kernel can be configured to disable stopping the scheduler tick in the idle
loop altogether.  That can be done through the build-time configuration of it
(by unsetting the ``CONFIG_NO_HZ_IDLE`` configuration option) or by passing
``nohz=off`` to it in the command line.  In both cases, as the stopping of the
scheduler tick is disabled, the governor's decisions regarding it are simply
ignored by the idle loop code and the tick is never stopped.

The systems that run kernels configured to allow the scheduler tick to be
stopped on idle CPUs are referred to as *tickless* systems and they are
generally regarded as more energy-efficient than the systems running kernels in
which the tick cannot be stopped.  If the given system is tickless, it will use
the ``menu`` governor by default and if it is not tickless, the default
``CPUIdle`` governor on it will be ``ladder``.

.. _menu-gov:

The ``menu`` Governor
=====================

The ``menu`` governor is the default ``CPUIdle`` governor for tickless systems.
It is quite complex, but the basic principle of its design is straightforward.
Namely, when invoked to select an idle state for a CPU (i.e. an idle state that
the CPU will ask the processor hardware to enter), it attempts to predict the
idle duration and uses the predicted value for idle state selection.

It first obtains the time until the closest timer event with the assumption
that the scheduler tick will be stopped.  That time, referred to as the *sleep
length* in what follows, is the upper bound on the time before the next CPU
wakeup.  It is used to determine the sleep length range, which in turn is needed
to get the sleep length correction factor.

The ``menu`` governor maintains two arrays of sleep length correction factors.
One of them is used when tasks previously running on the given CPU are waiting
for some I/O operations to complete and the other one is used when that is not
the case.  Each array contains several correction factor values that correspond
to different sleep length ranges organized so that each range represented in the
array is approximately 10 times wider than the previous one.

The correction factor for the given sleep length range (determined before
selecting the idle state for the CPU) is updated after the CPU has been woken
up and the closer the sleep length is to the observed idle duration, the closer
to 1 the correction factor becomes (it must fall between 0 and 1 inclusive).
The sleep length is multiplied by the correction factor for the range that it
falls into to obtain the first approximation of the predicted idle duration.

Next, the governor uses a simple pattern recognition algorithm to refine its
idle duration prediction.  Namely, it saves the last 8 observed idle duration
values and, when predicting the idle duration next time, it computes the average
and variance of them.  If the variance is small (smaller than 400 square
milliseconds) or it is small relative to the average (the average is greater
that 6 times the standard deviation), the average is regarded as the "typical
interval" value.  Otherwise, the longest of the saved observed idle duration
values is discarded and the computation is repeated for the remaining ones.
Again, if the variance of them is small (in the above sense), the average is
taken as the "typical interval" value and so on, until either the "typical
interval" is determined or too many data points are disregarded, in which case
the "typical interval" is assumed to equal "infinity" (the maximum unsigned
integer value).  The "typical interval" computed this way is compared with the
sleep length multiplied by the correction factor and the minimum of the two is
taken as the predicted idle duration.

Then, the governor computes an extra latency limit to help "interactive"
workloads.  It uses the observation that if the exit latency of the selected
idle state is comparable with the predicted idle duration, the total time spent
in that state probably will be very short and the amount of energy to save by
entering it will be relatively small, so likely it is better to avoid the
overhead related to entering that state and exiting it.  Thus selecting a
shallower state is likely to be a better option then.   The first approximation
of the extra latency limit is the predicted idle duration itself which
additionally is divided by a value depending on the number of tasks that
previously ran on the given CPU and now they are waiting for I/O operations to
complete.  The result of that division is compared with the latency limit coming
from the power management quality of service, or `PM QoS <cpu-pm-qos_>`_,
framework and the minimum of the two is taken as the limit for the idle states'
exit latency.

Now, the governor is ready to walk the list of idle states and choose one of
them.  For this purpose, it compares the target residency of each state with
the predicted idle duration and the exit latency of it with the computed latency
limit.  It selects the state with the target residency closest to the predicted
idle duration, but still below it, and exit latency that does not exceed the
limit.

In the final step the governor may still need to refine the idle state selection
if it has not decided to `stop the scheduler tick <idle-cpus-and-tick_>`_.  That
happens if the idle duration predicted by it is less than the tick period and
the tick has not been stopped already (in a previous iteration of the idle
loop).  Then, the sleep length used in the previous computations may not reflect
the real time until the closest timer event and if it really is greater than
that time, the governor may need to select a shallower state with a suitable
target residency.

.. _teo-gov:

The Timer Events Oriented (TEO) Governor
========================================

The timer events oriented (TEO) governor is an alternative ``CPUIdle`` governor
for tickless systems.  It follows the same basic strategy as the ``menu`` `one
<menu-gov_>`_: it always tries to find the deepest idle state suitable for the
given conditions.  However, it applies a different approach to that problem.

.. kernel-doc:: drivers/cpuidle/governors/teo.c
   :doc: teo-description

.. _idle-states-representation:

Representation of Idle States
=============================

For the CPU idle time management purposes all of the physical idle states
supported by the processor have to be represented as a one-dimensional array of
|struct cpuidle_state| objects each allowing an individual (logical) CPU to ask
the processor hardware to enter an idle state of certain properties.  If there
is a hierarchy of units in the processor, one |struct cpuidle_state| object can
cover a combination of idle states supported by the units at different levels of
the hierarchy.  In that case, the `target residency and exit latency parameters
of it <idle-loop_>`_, must reflect the properties of the idle state at the
deepest level (i.e. the idle state of the unit containing all of the other
units).

For example, take a processor with two cores in a larger unit referred to as
a "module" and suppose that asking the hardware to enter a specific idle state
(say "X") at the "core" level by one core will trigger the module to try to
enter a specific idle state of its own (say "MX") if the other core is in idle
state "X" already.  In other words, asking for idle state "X" at the "core"
level gives the hardware a license to go as deep as to idle state "MX" at the
"module" level, but there is no guarantee that this is going to happen (the core
asking for idle state "X" may just end up in that state by itself instead).
Then, the target residency of the |struct cpuidle_state| object representing
idle state "X" must reflect the minimum time to spend in idle state "MX" of
the module (including the time needed to enter it), because that is the minimum
time the CPU needs to be idle to save any energy in case the hardware enters
that state.  Analogously, the exit latency parameter of that object must cover
the exit time of idle state "MX" of the module (and usually its entry time too),
because that is the maximum delay between a wakeup signal and the time the CPU
will start to execute the first new instruction (assuming that both cores in the
module will always be ready to execute instructions as soon as the module
becomes operational as a whole).

There are processors without direct coordination between different levels of the
hierarchy of units inside them, however.  In those cases asking for an idle
state at the "core" level does not automatically affect the "module" level, for
example, in any way and the ``CPUIdle`` driver is responsible for the entire
handling of the hierarchy.  Then, the definition of the idle state objects is
entirely up to the driver, but still the physical properties of the idle state
that the processor hardware finally goes into must always follow the parameters
used by the governor for idle state selection (for instance, the actual exit
latency of that idle state must not exceed the exit latency parameter of the
idle state object selected by the governor).

In addition to the target residency and exit latency idle state parameters
discussed above, the objects representing idle states each contain a few other
parameters describing the idle state and a pointer to the function to run in
order to ask the hardware to enter that state.  Also, for each
|struct cpuidle_state| object, there is a corresponding
:c:type:`struct cpuidle_state_usage <cpuidle_state_usage>` one containing usage
statistics of the given idle state.  That information is exposed by the kernel
via ``sysfs``.

For each CPU in the system, there is a :file:`/sys/devices/system/cpu/cpu<N>/cpuidle/`
directory in ``sysfs``, where the number ``<N>`` is assigned to the given
CPU at the initialization time.  That directory contains a set of subdirectories
called :file:`state0`, :file:`state1` and so on, up to the number of idle state
objects defined for the given CPU minus one.  Each of these directories
corresponds to one idle state object and the larger the number in its name, the
deeper the (effective) idle state represented by it.  Each of them contains
a number of files (attributes) representing the properties of the idle state
object corresponding to it, as follows:

``above``
	Total number of times this idle state had been asked for, but the
	observed idle duration was certainly too short to match its target
	residency.

``below``
	Total number of times this idle state had been asked for, but certainly
	a deeper idle state would have been a better match for the observed idle
	duration.

``desc``
	Description of the idle state.

``disable``
	Whether or not this idle state is disabled.

``default_status``
	The default status of this state, "enabled" or "disabled".

``latency``
	Exit latency of the idle state in microseconds.

``name``
	Name of the idle state.

``power``
	Power drawn by hardware in this idle state in milliwatts (if specified,
	0 otherwise).

``residency``
	Target residency of the idle state in microseconds.

``time``
	Total time spent in this idle state by the given CPU (as measured by the
	kernel) in microseconds.

``usage``
	Total number of times the hardware has been asked by the given CPU to
	enter this idle state.

``rejected``
	Total number of times a request to enter this idle state on the given
	CPU was rejected.

The :file:`desc` and :file:`name` files both contain strings.  The difference
between them is that the name is expected to be more concise, while the
description may be longer and it may contain white space or special characters.
The other files listed above contain integer numbers.

The :file:`disable` attribute is the only writeable one.  If it contains 1, the
given idle state is disabled for this particular CPU, which means that the
governor will never select it for this particular CPU and the ``CPUIdle``
driver will never ask the hardware to enter it for that CPU as a result.
However, disabling an idle state for one CPU does not prevent it from being
asked for by the other CPUs, so it must be disabled for all of them in order to
never be asked for by any of them.  [Note that, due to the way the ``ladder``
governor is implemented, disabling an idle state prevents that governor from
selecting any idle states deeper than the disabled one too.]

If the :file:`disable` attribute contains 0, the given idle state is enabled for
this particular CPU, but it still may be disabled for some or all of the other
CPUs in the system at the same time.  Writing 1 to it causes the idle state to
be disabled for this particular CPU and writing 0 to it allows the governor to
take it into consideration for the given CPU and the driver to ask for it,
unless that state was disabled globally in the driver (in which case it cannot
be used at all).

The :file:`power` attribute is not defined very well, especially for idle state
objects representing combinations of idle states at different levels of the
hierarchy of units in the processor, and it generally is hard to obtain idle
state power numbers for complex hardware, so :file:`power` often contains 0 (not
available) and if it contains a nonzero number, that number may not be very
accurate and it should not be relied on for anything meaningful.

The number in the :file:`time` file generally may be greater than the total time
really spent by the given CPU in the given idle state, because it is measured by
the kernel and it may not cover the cases in which the hardware refused to enter
this idle state and entered a shallower one instead of it (or even it did not
enter any idle state at all).  The kernel can only measure the time span between
asking the hardware to enter an idle state and the subsequent wakeup of the CPU
and it cannot say what really happened in the meantime at the hardware level.
Moreover, if the idle state object in question represents a combination of idle
states at different levels of the hierarchy of units in the processor,
the kernel can never say how deep the hardware went down the hierarchy in any
particular case.  For these reasons, the only reliable way to find out how
much time has been spent by the hardware in different idle states supported by
it is to use idle state residency counters in the hardware, if available.

Generally, an interrupt received when trying to enter an idle state causes the
idle state entry request to be rejected, in which case the ``CPUIdle`` driver
may return an error code to indicate that this was the case. The :file:`usage`
and :file:`rejected` files report the number of times the given idle state
was entered successfully or rejected, respectively.

.. _cpu-pm-qos:

Power Management Quality of Service for CPUs
============================================

The power management quality of service (PM QoS) framework in the Linux kernel
allows kernel code and user space processes to set constraints on various
energy-efficiency features of the kernel to prevent performance from dropping
below a required level.

CPU idle time management can be affected by PM QoS in two ways, through the
global CPU latency limit and through the resume latency constraints for
individual CPUs.  Kernel code (e.g. device drivers) can set both of them with
the help of special internal interfaces provided by the PM QoS framework.  User
space can modify the former by opening the :file:`cpu_dma_latency` special
device file under :file:`/dev/` and writing a binary value (interpreted as a
signed 32-bit integer) to it.  In turn, the resume latency constraint for a CPU
can be modified from user space by writing a string (representing a signed
32-bit integer) to the :file:`power/pm_qos_resume_latency_us` file under
:file:`/sys/devices/system/cpu/cpu<N>/` in ``sysfs``, where the CPU number
``<N>`` is allocated at the system initialization time.  Negative values
will be rejected in both cases and, also in both cases, the written integer
number will be interpreted as a requested PM QoS constraint in microseconds.

The requested value is not automatically applied as a new constraint, however,
as it may be less restrictive (greater in this particular case) than another
constraint previously requested by someone else.  For this reason, the PM QoS
framework maintains a list of requests that have been made so far for the
global CPU latency limit and for each individual CPU, aggregates them and
applies the effective (minimum in this particular case) value as the new
constraint.

In fact, opening the :file:`cpu_dma_latency` special device file causes a new
PM QoS request to be created and added to a global priority list of CPU latency
limit requests and the file descriptor coming from the "open" operation
represents that request.  If that file descriptor is then used for writing, the
number written to it will be associated with the PM QoS request represented by
it as a new requested limit value.  Next, the priority list mechanism will be
used to determine the new effective value of the entire list of requests and
that effective value will be set as a new CPU latency limit.  Thus requesting a
new limit value will only change the real limit if the effective "list" value is
affected by it, which is the case if it is the minimum of the requested values
in the list.

The process holding a file descriptor obtained by opening the
:file:`cpu_dma_latency` special device file controls the PM QoS request
associated with that file descriptor, but it controls this particular PM QoS
request only.

Closing the :file:`cpu_dma_latency` special device file or, more precisely, the
file descriptor obtained while opening it, causes the PM QoS request associated
with that file descriptor to be removed from the global priority list of CPU
latency limit requests and destroyed.  If that happens, the priority list
mechanism will be used again, to determine the new effective value for the whole
list and that value will become the new limit.

In turn, for each CPU there is one resume latency PM QoS request associated with
the :file:`power/pm_qos_resume_latency_us` file under
:file:`/sys/devices/system/cpu/cpu<N>/` in ``sysfs`` and writing to it causes
this single PM QoS request to be updated regardless of which user space
process does that.  In other words, this PM QoS request is shared by the entire
user space, so access to the file associated with it needs to be arbitrated
to avoid confusion.  [Arguably, the only legitimate use of this mechanism in
practice is to pin a process to the CPU in question and let it use the
``sysfs`` interface to control the resume latency constraint for it.]  It is
still only a request, however.  It is an entry in a priority list used to
determine the effective value to be set as the resume latency constraint for the
CPU in question every time the list of requests is updated this way or another
(there may be other requests coming from kernel code in that list).

CPU idle time governors are expected to regard the minimum of the global
(effective) CPU latency limit and the effective resume latency constraint for
the given CPU as the upper limit for the exit latency of the idle states that
they are allowed to select for that CPU.  They should never select any idle
states with exit latency beyond that limit.

Idle States Control Via Kernel Command Line
===========================================

In addition to the ``sysfs`` interface allowing individual idle states to be
`disabled for individual CPUs <idle-states-representation_>`_, there are kernel
command line parameters affecting CPU idle time management.

The ``cpuidle.off=1`` kernel command line option can be used to disable the
CPU idle time management entirely.  It does not prevent the idle loop from
running on idle CPUs, but it prevents the CPU idle time governors and drivers
from being invoked.  If it is added to the kernel command line, the idle loop
will ask the hardware to enter idle states on idle CPUs via the CPU architecture
support code that is expected to provide a default mechanism for this purpose.
That default mechanism usually is the least common denominator for all of the
processors implementing the architecture (i.e. CPU instruction set) in question,
however, so it is rather crude and not very energy-efficient.  For this reason,
it is not recommended for production use.

The ``cpuidle.governor=`` kernel command line switch allows the ``CPUIdle``
governor to use to be specified.  It has to be appended with a string matching
the name of an available governor (e.g. ``cpuidle.governor=menu``) and that
governor will be used instead of the default one.  It is possible to force
the ``menu`` governor to be used on the systems that use the ``ladder`` governor
by default this way, for example.

The other kernel command line parameters controlling CPU idle time management
described below are only relevant for the *x86* architecture and references
to ``intel_idle`` affect Intel processors only.

The *x86* architecture support code recognizes three kernel command line
options related to CPU idle time management: ``idle=poll``, ``idle=halt``,
and ``idle=nomwait``.  The first two of them disable the ``acpi_idle`` and
``intel_idle`` drivers altogether, which effectively causes the entire
``CPUIdle`` subsystem to be disabled and makes the idle loop invoke the
architecture support code to deal with idle CPUs.  How it does that depends on
which of the two parameters is added to the kernel command line.  In the
``idle=halt`` case, the architecture support code will use the ``HLT``
instruction of the CPUs (which, as a rule, suspends the execution of the program
and causes the hardware to attempt to enter the shallowest available idle state)
for this purpose, and if ``idle=poll`` is used, idle CPUs will execute a
more or less "lightweight" sequence of instructions in a tight loop.  [Note
that using ``idle=poll`` is somewhat drastic in many cases, as preventing idle
CPUs from saving almost any energy at all may not be the only effect of it.
For example, on Intel hardware it effectively prevents CPUs from using
P-states (see |cpufreq|) that require any number of CPUs in a package to be
idle, so it very well may hurt single-thread computations performance as well as
energy-efficiency.  Thus using it for performance reasons may not be a good idea
at all.]

The ``idle=nomwait`` option prevents the use of ``MWAIT`` instruction of
the CPU to enter idle states. When this option is used, the ``acpi_idle``
driver will use the ``HLT`` instruction instead of ``MWAIT``. On systems
running Intel processors, this option disables the ``intel_idle`` driver
and forces the use of the ``acpi_idle`` driver instead. Note that in either
case, ``acpi_idle`` driver will function only if all the information needed
by it is in the system's ACPI tables.

In addition to the architecture-level kernel command line options affecting CPU
idle time management, there are parameters affecting individual ``CPUIdle``
drivers that can be passed to them via the kernel command line.  Specifically,
the ``intel_idle.max_cstate=<n>`` and ``processor.max_cstate=<n>`` parameters,
where ``<n>`` is an idle state index also used in the name of the given
state's directory in ``sysfs`` (see
`Representation of Idle States <idle-states-representation_>`_), causes the
``intel_idle`` and ``acpi_idle`` drivers, respectively, to discard all of the
idle states deeper than idle state ``<n>``.  In that case, they will never ask
for any of those idle states or expose them to the governor.  [The behavior of
the two drivers is different for ``<n>`` equal to ``0``.  Adding
``intel_idle.max_cstate=0`` to the kernel command line disables the
``intel_idle`` driver and allows ``acpi_idle`` to be used, whereas
``processor.max_cstate=0`` is equivalent to ``processor.max_cstate=1``.
Also, the ``acpi_idle`` driver is part of the ``processor`` kernel module that
can be loaded separately and ``max_cstate=<n>`` can be passed to it as a module
parameter when it is loaded.]


code:
.. SPDX-License-Identifier: GPL-2.0

================
Power Management
================

.. toctree::
   :maxdepth: 2

   strategies
   system-wide
   working-state

repo_name: angery/linux


code:
.. SPDX-License-Identifier: GPL-2.0

============================================================
Intel(R) Speed Select Technology User Guide
============================================================

The Intel(R) Speed Select Technology (Intel(R) SST) provides a powerful new
collection of features that give more granular control over CPU performance.
With Intel(R) SST, one server can be configured for power and performance for a
variety of diverse workload requirements.

Refer to the links below for an overview of the technology:

- https://www.intel.com/content/www/us/en/architecture-and-technology/speed-select-technology-article.html
- https://builders.intel.com/docs/networkbuilders/intel-speed-select-technology-base-frequency-enhancing-performance.pdf

These capabilities are further enhanced in some of the newer generations of
server platforms where these features can be enumerated and controlled
dynamically without pre-configuring via BIOS setup options. This dynamic
configuration is done via mailbox commands to the hardware. One way to enumerate
and configure these features is by using the Intel Speed Select utility.

This document explains how to use the Intel Speed Select tool to enumerate and
control Intel(R) SST features. This document gives example commands and explains
how these commands change the power and performance profile of the system under
test. Using this tool as an example, customers can replicate the messaging
implemented in the tool in their production software.

intel-speed-select configuration tool
======================================

Most Linux distribution packages may include the "intel-speed-select" tool. If not,
it can be built by downloading the Linux kernel tree from kernel.org. Once
downloaded, the tool can be built without building the full kernel.

From the kernel tree, run the following commands::

# cd tools/power/x86/intel-speed-select/
# make
# make install

Getting Help
------------

To get help with the tool, execute the command below::

# intel-speed-select --help

The top-level help describes arguments and features. Notice that there is a
multi-level help structure in the tool. For example, to get help for the feature "perf-profile"::

# intel-speed-select perf-profile --help

To get help on a command, another level of help is provided. For example for the command info "info"::

# intel-speed-select perf-profile info --help

Summary of platform capability
------------------------------
To check the current platform and driver capabilities, execute::

#intel-speed-select --info

For example on a test system::

 # intel-speed-select --info
 Intel(R) Speed Select Technology
 Executing on CPU model: X
 Platform: API version : 1
 Platform: Driver version : 1
 Platform: mbox supported : 1
 Platform: mmio supported : 1
 Intel(R) SST-PP (feature perf-profile) is supported
 TDP level change control is unlocked, max level: 4
 Intel(R) SST-TF (feature turbo-freq) is supported
 Intel(R) SST-BF (feature base-freq) is not supported
 Intel(R) SST-CP (feature core-power) is supported

Intel(R) Speed Select Technology - Performance Profile (Intel(R) SST-PP)
------------------------------------------------------------------------

This feature allows configuration of a server dynamically based on workload
performance requirements. This helps users during deployment as they do not have
to choose a specific server configuration statically.  This Intel(R) Speed Select
Technology - Performance Profile (Intel(R) SST-PP) feature introduces a mechanism
that allows multiple optimized performance profiles per system. Each profile
defines a set of CPUs that need to be online and rest offline to sustain a
guaranteed base frequency. Once the user issues a command to use a specific
performance profile and meet CPU online/offline requirement, the user can expect
a change in the base frequency dynamically. This feature is called
"perf-profile" when using the Intel Speed Select tool.

Number or performance levels
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

There can be multiple performance profiles on a system. To get the number of
profiles, execute the command below::

 # intel-speed-select perf-profile get-config-levels
 Intel(R) Speed Select Technology
 Executing on CPU model: X
 package-0
  die-0
    cpu-0
        get-config-levels:4
 package-1
  die-0
    cpu-14
        get-config-levels:4

On this system under test, there are 4 performance profiles in addition to the
base performance profile (which is performance level 0).

Lock/Unlock status
~~~~~~~~~~~~~~~~~~

Even if there are multiple performance profiles, it is possible that they
are locked. If they are locked, users cannot issue a command to change the
performance state. It is possible that there is a BIOS setup to unlock or check
with your system vendor.

To check if the system is locked, execute the following command::

 # intel-speed-select perf-profile get-lock-status
 Intel(R) Speed Select Technology
 Executing on CPU model: X
 package-0
  die-0
    cpu-0
        get-lock-status:0
 package-1
  die-0
    cpu-14
        get-lock-status:0

In this case, lock status is 0, which means that the system is unlocked.

Properties of a performance level
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To get properties of a specific performance level (For example for the level 0, below), execute the command below::

 # intel-speed-select perf-profile info -l 0
 Intel(R) Speed Select Technology
 Executing on CPU model: X
 package-0
  die-0
    cpu-0
      perf-profile-level-0
        cpu-count:28
        enable-cpu-mask:000003ff,f0003fff
        enable-cpu-list:0,1,2,3,4,5,6,7,8,9,10,11,12,13,28,29,30,31,32,33,34,35,36,37,38,39,40,41
        thermal-design-power-ratio:26
        base-frequency(MHz):2600
        speed-select-turbo-freq:disabled
        speed-select-base-freq:disabled
	...
	...

Here -l option is used to specify a performance level.

If the option -l is omitted, then this command will print information about all
the performance levels. The above command is printing properties of the
performance level 0.

For this performance profile, the list of CPUs displayed by the
"enable-cpu-mask/enable-cpu-list" at the max can be "online." When that
condition is met, then base frequency of 2600 MHz can be maintained. To
understand more, execute "intel-speed-select perf-profile info" for performance
level 4::

 # intel-speed-select perf-profile info -l 4
 Intel(R) Speed Select Technology
 Executing on CPU model: X
 package-0
  die-0
    cpu-0
      perf-profile-level-4
        cpu-count:28
        enable-cpu-mask:000000fa,f0000faf
        enable-cpu-list:0,1,2,3,5,7,8,9,10,11,28,29,30,31,33,35,36,37,38,39
        thermal-design-power-ratio:28
        base-frequency(MHz):2800
        speed-select-turbo-freq:disabled
        speed-select-base-freq:unsupported
	...
	...

There are fewer CPUs in the "enable-cpu-mask/enable-cpu-list". Consequently, if
the user only keeps these CPUs online and the rest "offline," then the base
frequency is increased to 2.8 GHz compared to 2.6 GHz at performance level 0.

Get current performance level
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To get the current performance level, execute::

 # intel-speed-select perf-profile get-config-current-level
 Intel(R) Speed Select Technology
 Executing on CPU model: X
 package-0
  die-0
    cpu-0
        get-config-current_level:0

First verify that the base_frequency displayed by the cpufreq sysfs is correct::

 # cat /sys/devices/system/cpu/cpu0/cpufreq/base_frequency
 2600000

This matches the base-frequency (MHz) field value displayed from the
"perf-profile info" command for performance level 0(cpufreq frequency is in
KHz).

To check if the average frequency is equal to the base frequency for a 100% busy
workload, disable turbo::

# echo 1 > /sys/devices/system/cpu/intel_pstate/no_turbo

Then runs a busy workload on all CPUs, for example::

#stress -c 64

To verify the base frequency, run turbostat::

 #turbostat -c 0-13 --show Package,Core,CPU,Bzy_MHz -i 1

  Package	Core	CPU	Bzy_MHz
		-	-	2600
  0		0	0	2600
  0		1	1	2600
  0		2	2	2600
  0		3	3	2600
  0		4	4	2600
  .		.	.	.

Changing performance level
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To the change the performance level to 4, execute::

 # intel-speed-select -d perf-profile set-config-level -l 4 -o
 Intel(R) Speed Select Technology
 Executing on CPU model: X
 package-0
  die-0
    cpu-0
      perf-profile
        set_tdp_level:success

In the command above, "-o" is optional. If it is specified, then it will also
offline CPUs which are not present in the enable_cpu_mask for this performance
level.

Now if the base_frequency is checked::

 #cat /sys/devices/system/cpu/cpu0/cpufreq/base_frequency
 2800000

Which shows that the base frequency now increased from 2600 MHz at performance
level 0 to 2800 MHz at performance level 4. As a result, any workload, which can
use fewer CPUs, can see a boost of 200 MHz compared to performance level 0.

Changing performance level via BMC Interface
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

It is possible to change SST-PP level using out of band (OOB) agent (Via some
remote management console, through BMC "Baseboard Management Controller"
interface). This mode is supported from the Sapphire Rapids processor
generation. The kernel and tool change to support this mode is added to Linux
kernel version 5.18. To enable this feature, kernel config
"CONFIG_INTEL_HFI_THERMAL" is required. The minimum version of the tool
is "v1.12" to support this feature, which is part of Linux kernel version 5.18.

To support such configuration, this tool can be used as a daemon. Add
a command line option --oob::

 # intel-speed-select --oob
 Intel(R) Speed Select Technology
 Executing on CPU model:143[0x8f]
 OOB mode is enabled and will run as daemon

In this mode the tool will online/offline CPUs based on the new performance
level.

Check presence of other Intel(R) SST features
---------------------------------------------

Each of the performance profiles also specifies weather there is support of
other two Intel(R) SST features (Intel(R) Speed Select Technology - Base Frequency
(Intel(R) SST-BF) and Intel(R) Speed Select Technology - Turbo Frequency (Intel
SST-TF)).

For example, from the output of "perf-profile info" above, for level 0 and level
4:

For level 0::
       speed-select-turbo-freq:disabled
       speed-select-base-freq:disabled

For level 4::
       speed-select-turbo-freq:disabled
       speed-select-base-freq:unsupported

Given these results, the "speed-select-base-freq" (Intel(R) SST-BF) in level 4
changed from "disabled" to "unsupported" compared to performance level 0.

This means that at performance level 4, the "speed-select-base-freq" feature is
not supported. However, at performance level 0, this feature is "supported", but
currently "disabled", meaning the user has not activated this feature. Whereas
"speed-select-turbo-freq" (Intel(R) SST-TF) is supported at both performance
levels, but currently not activated by the user.

The Intel(R) SST-BF and the Intel(R) SST-TF features are built on a foundation
technology called Intel(R) Speed Select Technology - Core Power (Intel(R) SST-CP).
The platform firmware enables this feature when Intel(R) SST-BF or Intel(R) SST-TF
is supported on a platform.

Intel(R) Speed Select Technology Core Power (Intel(R) SST-CP)
---------------------------------------------------------------

Intel(R) Speed Select Technology Core Power (Intel(R) SST-CP) is an interface that
allows users to define per core priority. This defines a mechanism to distribute
power among cores when there is a power constrained scenario. This defines a
class of service (CLOS) configuration.

The user can configure up to 4 class of service configurations. Each CLOS group
configuration allows definitions of parameters, which affects how the frequency
can be limited and power is distributed. Each CPU core can be tied to a class of
service and hence an associated priority. The granularity is at core level not
at per CPU level.

Enable CLOS based prioritization
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To use CLOS based prioritization feature, firmware must be informed to enable
and use a priority type. There is a default per platform priority type, which
can be changed with optional command line parameter.

To enable and check the options, execute::

 # intel-speed-select core-power enable --help
 Intel(R) Speed Select Technology
 Executing on CPU model: X
 Enable core-power for a package/die
	Clos Enable: Specify priority type with [--priority|-p]
		 0: Proportional, 1: Ordered

There are two types of priority types:

- Ordered

Priority for ordered throttling is defined based on the index of the assigned
CLOS group. Where CLOS0 gets highest priority (throttled last).

Priority order is:
CLOS0 > CLOS1 > CLOS2 > CLOS3.

- Proportional

When proportional priority is used, there is an additional parameter called
frequency_weight, which can be specified per CLOS group. The goal of
proportional priority is to provide each core with the requested min., then
distribute all remaining (excess/deficit) budgets in proportion to a defined
weight. This proportional priority can be configured using "core-power config"
command.

To enable with the platform default priority type, execute::

 # intel-speed-select core-power enable
 Intel(R) Speed Select Technology
 Executing on CPU model: X
 package-0
  die-0
    cpu-0
      core-power
        enable:success
 package-1
  die-0
    cpu-6
      core-power
        enable:success

The scope of this enable is per package or die scoped when a package contains
multiple dies. To check if CLOS is enabled and get priority type, "core-power
info" command can be used. For example to check the status of core-power feature
on CPU 0, execute::

 # intel-speed-select -c 0 core-power info
 Intel(R) Speed Select Technology
 Executing on CPU model: X
 package-0
  die-0
    cpu-0
      core-power
        support-status:supported
        enable-status:enabled
        clos-enable-status:enabled
        priority-type:proportional
 package-1
  die-0
    cpu-24
      core-power
        support-status:supported
        enable-status:enabled
        clos-enable-status:enabled
        priority-type:proportional

Configuring CLOS groups
~~~~~~~~~~~~~~~~~~~~~~~

Each CLOS group has its own attributes including min, max, freq_weight and
desired. These parameters can be configured with "core-power config" command.
Defaults will be used if user skips setting a parameter except clos id, which is
mandatory. To check core-power config options, execute::

 # intel-speed-select core-power config --help
 Intel(R) Speed Select Technology
 Executing on CPU model: X
 Set core-power configuration for one of the four clos ids
	Specify targeted clos id with [--clos|-c]
	Specify clos Proportional Priority [--weight|-w]
	Specify clos min in MHz with [--min|-n]
	Specify clos max in MHz with [--max|-m]

For example::

 # intel-speed-select core-power config -c 0
 Intel(R) Speed Select Technology
 Executing on CPU model: X
 clos epp is not specified, default: 0
 clos frequency weight is not specified, default: 0
 clos min is not specified, default: 0 MHz
 clos max is not specified, default: 25500 MHz
 clos desired is not specified, default: 0
 package-0
  die-0
    cpu-0
      core-power
        config:success
 package-1
  die-0
    cpu-6
      core-power
        config:success

The user has the option to change defaults. For example, the user can change the
"min" and set the base frequency to always get guaranteed base frequency.

Get the current CLOS configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To check the current configuration, "core-power get-config" can be used. For
example, to get the configuration of CLOS 0::

 # intel-speed-select core-power get-config -c 0
 Intel(R) Speed Select Technology
 Executing on CPU model: X
 package-0
  die-0
    cpu-0
      core-power
        clos:0
        epp:0
        clos-proportional-priority:0
        clos-min:0 MHz
        clos-max:Max Turbo frequency
        clos-desired:0 MHz
 package-1
  die-0
    cpu-24
      core-power
        clos:0
        epp:0
        clos-proportional-priority:0
        clos-min:0 MHz
        clos-max:Max Turbo frequency
        clos-desired:0 MHz

Associating a CPU with a CLOS group
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To associate a CPU to a CLOS group "core-power assoc" command can be used::

 # intel-speed-select core-power assoc --help
 Intel(R) Speed Select Technology
 Executing on CPU model: X
 Associate a clos id to a CPU
	Specify targeted clos id with [--clos|-c]

For example to associate CPU 10 to CLOS group 3, execute::

 # intel-speed-select -c 10 core-power assoc -c 3
 Intel(R) Speed Select Technology
 Executing on CPU model: X
 package-0
  die-0
    cpu-10
      core-power
        assoc:success

Once a CPU is associated, its sibling CPUs are also associated to a CLOS group.
Once associated, avoid changing Linux "cpufreq" subsystem scaling frequency
limits.

To check the existing association for a CPU, "core-power get-assoc" command can
be used. For example, to get association of CPU 10, execute::

 # intel-speed-select -c 10 core-power get-assoc
 Intel(R) Speed Select Technology
 Executing on CPU model: X
 package-1
  die-0
    cpu-10
      get-assoc
        clos:3

This shows that CPU 10 is part of a CLOS group 3.

Disable CLOS based prioritization
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To disable, execute::

# intel-speed-select core-power disable

Some features like Intel(R) SST-TF can only be enabled when CLOS based prioritization
is enabled. For this reason, disabling while Intel(R) SST-TF is enabled can cause
Intel(R) SST-TF to fail. This will cause the "disable" command to display an error
if Intel(R) SST-TF is already enabled. In turn, to disable, the Intel(R) SST-TF
feature must be disabled first.

Intel(R) Speed Select Technology - Base Frequency (Intel(R) SST-BF)
-------------------------------------------------------------------

The Intel(R) Speed Select Technology - Base Frequency (Intel(R) SST-BF) feature lets
the user control base frequency. If some critical workload threads demand
constant high guaranteed performance, then this feature can be used to execute
the thread at higher base frequency on specific sets of CPUs (high priority
CPUs) at the cost of lower base frequency (low priority CPUs) on other CPUs.
This feature does not require offline of the low priority CPUs.

The support of Intel(R) SST-BF depends on the Intel(R) Speed Select Technology -
Performance Profile (Intel(R) SST-PP) performance level configuration. It is
possible that only certain performance levels support Intel(R) SST-BF. It is also
possible that only base performance level (level = 0) has support of Intel
SST-BF. Consequently, first select the desired performance level to enable this
feature.

In the system under test here, Intel(R) SST-BF is supported at the base
performance level 0, but currently disabled. For example for the level 0::

 # intel-speed-select -c 0 perf-profile info -l 0
 Intel(R) Speed Select Technology
 Executing on CPU model: X
 package-0
  die-0
    cpu-0
      perf-profile-level-0
        ...

        speed-select-base-freq:disabled
	...

Before enabling Intel(R) SST-BF and measuring its impact on a workload
performance, execute some workload and measure performance and get a baseline
performance to compare against.

Here the user wants more guaranteed performance. For this reason, it is likely
that turbo is disabled. To disable turbo, execute::

#echo 1 > /sys/devices/system/cpu/intel_pstate/no_turbo

Based on the output of the "intel-speed-select perf-profile info -l 0" base
frequency of guaranteed frequency 2600 MHz.

Measure baseline performance for comparison
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To compare, pick a multi-threaded workload where each thread can be scheduled on
separate CPUs. "Hackbench pipe" test is a good example on how to improve
performance using Intel(R) SST-BF.

Below, the workload is measuring average scheduler wakeup latency, so a lower
number means better performance::

 # taskset -c 3,4 perf bench -r 100 sched pipe
 # Running 'sched/pipe' benchmark:
 # Executed 1000000 pipe operations between two processes
     Total time: 6.102 [sec]
       6.102445 usecs/op
         163868 ops/sec

While running the above test, if we take turbostat output, it will show us that
2 of the CPUs are busy and reaching max. frequency (which would be the base
frequency as the turbo is disabled). The turbostat output::

 #turbostat -c 0-13 --show Package,Core,CPU,Bzy_MHz -i 1
 Package	Core	CPU	Bzy_MHz
 0		0	0	1000
 0		1	1	1005
 0		2	2	1000
 0		3	3	2600
 0		4	4	2600
 0		5	5	1000
 0		6	6	1000
 0		7	7	1005
 0		8	8	1005
 0		9	9	1000
 0		10	10	1000
 0		11	11	995
 0		12	12	1000
 0		13	13	1000

From the above turbostat output, both CPU 3 and 4 are very busy and reaching
full guaranteed frequency of 2600 MHz.

Intel(R) SST-BF Capabilities
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To get capabilities of Intel(R) SST-BF for the current performance level 0,
execute::

 # intel-speed-select base-freq info -l 0
 Intel(R) Speed Select Technology
 Executing on CPU model: X
 package-0
  die-0
    cpu-0
      speed-select-base-freq
        high-priority-base-frequency(MHz):3000
        high-priority-cpu-mask:00000216,00002160
        high-priority-cpu-list:5,6,8,13,33,34,36,41
        low-priority-base-frequency(MHz):2400
        tjunction-temperature(C):125
        thermal-design-power(W):205

The above capabilities show that there are some CPUs on this system that can
offer base frequency of 3000 MHz compared to the standard base frequency at this
performance levels. Nevertheless, these CPUs are fixed, and they are presented
via high-priority-cpu-list/high-priority-cpu-mask. But if this Intel(R) SST-BF
feature is selected, the low priorities CPUs (which are not in
high-priority-cpu-list) can only offer up to 2400 MHz. As a result, if this
clipping of low priority CPUs is acceptable, then the user can enable Intel
SST-BF feature particularly for the above "sched pipe" workload since only two
CPUs are used, they can be scheduled on high priority CPUs and can get boost of
400 MHz.

Enable Intel(R) SST-BF
~~~~~~~~~~~~~~~~~~~~~~

To enable Intel(R) SST-BF feature, execute::

 # intel-speed-select base-freq enable -a
 Intel(R) Speed Select Technology
 Executing on CPU model: X
 package-0
  die-0
    cpu-0
      base-freq
        enable:success
 package-1
  die-0
    cpu-14
      base-freq
        enable:success

In this case, -a option is optional. This not only enables Intel(R) SST-BF, but it
also adjusts the priority of cores using Intel(R) Speed Select Technology Core
Power (Intel(R) SST-CP) features. This option sets the minimum performance of each
Intel(R) Speed Select Technology - Performance Profile (Intel(R) SST-PP) class to
maximum performance so that the hardware will give maximum performance possible
for each CPU.

If -a option is not used, then the following steps are required before enabling
Intel(R) SST-BF:

- Discover Intel(R) SST-BF and note low and high priority base frequency
- Note the high priority CPU list
- Enable CLOS using core-power feature set
- Configure CLOS parameters. Use CLOS.min to set to minimum performance
- Subscribe desired CPUs to CLOS groups

With this configuration, if the same workload is executed by pinning the
workload to high priority CPUs (CPU 5 and 6 in this case)::

 #taskset -c 5,6 perf bench -r 100 sched pipe
 # Running 'sched/pipe' benchmark:
 # Executed 1000000 pipe operations between two processes
     Total time: 5.627 [sec]
       5.627922 usecs/op
         177685 ops/sec

This way, by enabling Intel(R) SST-BF, the performance of this benchmark is
improved (latency reduced) by 7.79%. From the turbostat output, it can be
observed that the high priority CPUs reached 3000 MHz compared to 2600 MHz.
The turbostat output::

 #turbostat -c 0-13 --show Package,Core,CPU,Bzy_MHz -i 1
 Package	Core	CPU	Bzy_MHz
 0		0	0	2151
 0		1	1	2166
 0		2	2	2175
 0		3	3	2175
 0		4	4	2175
 0		5	5	3000
 0		6	6	3000
 0		7	7	2180
 0		8	8	2662
 0		9	9	2176
 0		10	10	2175
 0		11	11	2176
 0		12	12	2176
 0		13	13	2661

Disable Intel(R) SST-BF
~~~~~~~~~~~~~~~~~~~~~~~

To disable the Intel(R) SST-BF feature, execute::

# intel-speed-select base-freq disable -a

Intel(R) Speed Select Technology - Turbo Frequency (Intel(R) SST-TF)
--------------------------------------------------------------------

This feature enables the ability to set different "All core turbo ratio limits"
to cores based on the priority. By using this feature, some cores can be
configured to get higher turbo frequency by designating them as high priority at
the cost of lower or no turbo frequency on the low priority cores.

For this reason, this feature is only useful when system is busy utilizing all
CPUs, but the user wants some configurable option to get high performance on
some CPUs.

The support of Intel(R) Speed Select Technology - Turbo Frequency (Intel(R) SST-TF)
depends on the Intel(R) Speed Select Technology - Performance Profile (Intel
SST-PP) performance level configuration. It is possible that only a certain
performance level supports Intel(R) SST-TF. It is also possible that only the base
performance level (level = 0) has the support of Intel(R) SST-TF. Hence, first
select the desired performance level to enable this feature.

In the system under test here, Intel(R) SST-TF is supported at the base
performance level 0, but currently disabled::

 # intel-speed-select -c 0 perf-profile info -l 0
 Intel(R) Speed Select Technology
 package-0
  die-0
    cpu-0
      perf-profile-level-0
        ...
        ...
        speed-select-turbo-freq:disabled
        ...
        ...

To check if performance can be improved using Intel(R) SST-TF feature, get the turbo
frequency properties with Intel(R) SST-TF enabled and compare to the base turbo
capability of this system.

Get Base turbo capability
~~~~~~~~~~~~~~~~~~~~~~~~~

To get the base turbo capability of performance level 0, execute::

 # intel-speed-select perf-profile info -l 0
 Intel(R) Speed Select Technology
 Executing on CPU model: X
 package-0
  die-0
    cpu-0
      perf-profile-level-0
        ...
        ...
        turbo-ratio-limits-sse
          bucket-0
            core-count:2
            max-turbo-frequency(MHz):3200
          bucket-1
            core-count:4
            max-turbo-frequency(MHz):3100
          bucket-2
            core-count:6
            max-turbo-frequency(MHz):3100
          bucket-3
            core-count:8
            max-turbo-frequency(MHz):3100
          bucket-4
            core-count:10
            max-turbo-frequency(MHz):3100
          bucket-5
            core-count:12
            max-turbo-frequency(MHz):3100
          bucket-6
            core-count:14
            max-turbo-frequency(MHz):3100
          bucket-7
            core-count:16
            max-turbo-frequency(MHz):3100

Based on the data above, when all the CPUS are busy, the max. frequency of 3100
MHz can be achieved. If there is some busy workload on cpu 0 - 11 (e.g. stress)
and on CPU 12 and 13, execute "hackbench pipe" workload::

 # taskset -c 12,13 perf bench -r 100 sched pipe
 # Running 'sched/pipe' benchmark:
 # Executed 1000000 pipe operations between two processes
     Total time: 5.705 [sec]
       5.705488 usecs/op
         175269 ops/sec

The turbostat output::

 #turbostat -c 0-13 --show Package,Core,CPU,Bzy_MHz -i 1
 Package	Core	CPU	Bzy_MHz
 0		0	0	3000
 0		1	1	3000
 0		2	2	3000
 0		3	3	3000
 0		4	4	3000
 0		5	5	3100
 0		6	6	3100
 0		7	7	3000
 0		8	8	3100
 0		9	9	3000
 0		10	10	3000
 0		11	11	3000
 0		12	12	3100
 0		13	13	3100

Based on turbostat output, the performance is limited by frequency cap of 3100
MHz. To check if the hackbench performance can be improved for CPU 12 and CPU
13, first check the capability of the Intel(R) SST-TF feature for this performance
level.

Get Intel(R) SST-TF Capability
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To get the capability, the "turbo-freq info" command can be used::

 # intel-speed-select turbo-freq info -l 0
 Intel(R) Speed Select Technology
 Executing on CPU model: X
 package-0
  die-0
    cpu-0
      speed-select-turbo-freq
          bucket-0
            high-priority-cores-count:2
            high-priority-max-frequency(MHz):3200
            high-priority-max-avx2-frequency(MHz):3200
            high-priority-max-avx512-frequency(MHz):3100
          bucket-1
            high-priority-cores-count:4
            high-priority-max-frequency(MHz):3100
            high-priority-max-avx2-frequency(MHz):3000
            high-priority-max-avx512-frequency(MHz):2900
          bucket-2
            high-priority-cores-count:6
            high-priority-max-frequency(MHz):3100
            high-priority-max-avx2-frequency(MHz):3000
            high-priority-max-avx512-frequency(MHz):2900
          speed-select-turbo-freq-clip-frequencies
            low-priority-max-frequency(MHz):2600
            low-priority-max-avx2-frequency(MHz):2400
            low-priority-max-avx512-frequency(MHz):2100

Based on the output above, there is an Intel(R) SST-TF bucket for which there are
two high priority cores. If only two high priority cores are set, then max.
turbo frequency on those cores can be increased to 3200 MHz. This is 100 MHz
more than the base turbo capability for all cores.

In turn, for the hackbench workload, two CPUs can be set as high priority and
rest as low priority. One side effect is that once enabled, the low priority
cores will be clipped to a lower frequency of 2600 MHz.

Enable Intel(R) SST-TF
~~~~~~~~~~~~~~~~~~~~~~

To enable Intel(R) SST-TF, execute::

 # intel-speed-select -c 12,13 turbo-freq enable -a
 Intel(R) Speed Select Technology
 Executing on CPU model: X
 package-0
  die-0
    cpu-12
      turbo-freq
        enable:success
 package-0
  die-0
    cpu-13
      turbo-freq
        enable:success
 package--1
  die-0
    cpu-63
      turbo-freq --auto
        enable:success

In this case, the option "-a" is optional. If set, it enables Intel(R) SST-TF
feature and also sets the CPUs to high and low priority using Intel Speed
Select Technology Core Power (Intel(R) SST-CP) features. The CPU numbers passed
with "-c" arguments are marked as high priority, including its siblings.

If -a option is not used, then the following steps are required before enabling
Intel(R) SST-TF:

- Discover Intel(R) SST-TF and note buckets of high priority cores and maximum frequency

- Enable CLOS using core-power feature set - Configure CLOS parameters

- Subscribe desired CPUs to CLOS groups making sure that high priority cores are set to the maximum frequency

If the same hackbench workload is executed, schedule hackbench threads on high
priority CPUs::

 #taskset -c 12,13 perf bench -r 100 sched pipe
 # Running 'sched/pipe' benchmark:
 # Executed 1000000 pipe operations between two processes
     Total time: 5.510 [sec]
       5.510165 usecs/op
         180826 ops/sec

This improved performance by around 3.3% improvement on a busy system. Here the
turbostat output will show that the CPU 12 and CPU 13 are getting 100 MHz boost.
The turbostat output::

 #turbostat -c 0-13 --show Package,Core,CPU,Bzy_MHz -i 1
 Package	Core	CPU	Bzy_MHz
 ...
 0		12	12	3200
 0		13	13	3200


code:
.. SPDX-License-Identifier: GPL-2.0
.. include:: <isonum.txt>

======================================
Intel Performance and Energy Bias Hint
======================================

:Copyright: |copy| 2019 Intel Corporation

:Author: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

.. kernel-doc:: arch/x86/kernel/cpu/intel_epb.c
   :doc: overview

Intel Performance and Energy Bias Attribute in ``sysfs``
========================================================

The Intel Performance and Energy Bias Hint (EPB) value for a given (logical) CPU
can be checked or updated through a ``sysfs`` attribute (file) under
:file:`/sys/devices/system/cpu/cpu<N>/power/`, where the CPU number ``<N>``
is allocated at the system initialization time:

``energy_perf_bias``
	Shows the current EPB value for the CPU in a sliding scale 0 - 15, where
	a value of 0 corresponds to a hint preference for highest performance
	and a value of 15 corresponds to the maximum energy savings.

	In order to update the EPB value for the CPU, this attribute can be
	written to, either with a number in the 0 - 15 sliding scale above, or
	with one of the strings: "performance", "balance-performance", "normal",
	"balance-power", "power" that represent values reflected by their
	meaning.

	This attribute is present for all online CPUs supporting the EPB
	feature.

Note that while the EPB interface to the processor is defined at the logical CPU
level, the physical register backing it may be shared by multiple CPUs (for
example, SMT siblings or cores in one package).  For this reason, updating the
EPB value for one CPU may cause the EPB values for other CPUs to change.


code:
.. SPDX-License-Identifier: GPL-2.0
.. include:: <isonum.txt>

==============================================
``intel_idle`` CPU Idle Time Management Driver
==============================================

:Copyright: |copy| 2020 Intel Corporation

:Author: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

General Information
===================

``intel_idle`` is a part of the
:doc:`CPU idle time management subsystem <cpuidle>` in the Linux kernel
(``CPUIdle``).  It is the default CPU idle time management driver for the
Nehalem and later generations of Intel processors, but the level of support for
a particular processor model in it depends on whether or not it recognizes that
processor model and may also depend on information coming from the platform
firmware.  [To understand ``intel_idle`` it is necessary to know how ``CPUIdle``
works in general, so this is the time to get familiar with
Documentation/admin-guide/pm/cpuidle.rst if you have not done that yet.]

``intel_idle`` uses the ``MWAIT`` instruction to inform the processor that the
logical CPU executing it is idle and so it may be possible to put some of the
processor's functional blocks into low-power states.  That instruction takes two
arguments (passed in the ``EAX`` and ``ECX`` registers of the target CPU), the
first of which, referred to as a *hint*, can be used by the processor to
determine what can be done (for details refer to Intel Software Developer’s
Manual [1]_).  Accordingly, ``intel_idle`` refuses to work with processors in
which the support for the ``MWAIT`` instruction has been disabled (for example,
via the platform firmware configuration menu) or which do not support that
instruction at all.

``intel_idle`` is not modular, so it cannot be unloaded, which means that the
only way to pass early-configuration-time parameters to it is via the kernel
command line.

.. _intel-idle-enumeration-of-states:

Enumeration of Idle States
==========================

Each ``MWAIT`` hint value is interpreted by the processor as a license to
reconfigure itself in a certain way in order to save energy.  The processor
configurations (with reduced power draw) resulting from that are referred to
as C-states (in the ACPI terminology) or idle states.  The list of meaningful
``MWAIT`` hint values and idle states (i.e. low-power configurations of the
processor) corresponding to them depends on the processor model and it may also
depend on the configuration of the platform.

In order to create a list of available idle states required by the ``CPUIdle``
subsystem (see :ref:`idle-states-representation` in
Documentation/admin-guide/pm/cpuidle.rst),
``intel_idle`` can use two sources of information: static tables of idle states
for different processor models included in the driver itself and the ACPI tables
of the system.  The former are always used if the processor model at hand is
recognized by ``intel_idle`` and the latter are used if that is required for
the given processor model (which is the case for all server processor models
recognized by ``intel_idle``) or if the processor model is not recognized.
[There is a module parameter that can be used to make the driver use the ACPI
tables with any processor model recognized by it; see
`below <intel-idle-parameters_>`_.]

If the ACPI tables are going to be used for building the list of available idle
states, ``intel_idle`` first looks for a ``_CST`` object under one of the ACPI
objects corresponding to the CPUs in the system (refer to the ACPI specification
[2]_ for the description of ``_CST`` and its output package).  Because the
``CPUIdle`` subsystem expects that the list of idle states supplied by the
driver will be suitable for all of the CPUs handled by it and ``intel_idle`` is
registered as the ``CPUIdle`` driver for all of the CPUs in the system, the
driver looks for the first ``_CST`` object returning at least one valid idle
state description and such that all of the idle states included in its return
package are of the FFH (Functional Fixed Hardware) type, which means that the
``MWAIT`` instruction is expected to be used to tell the processor that it can
enter one of them.  The return package of that ``_CST`` is then assumed to be
applicable to all of the other CPUs in the system and the idle state
descriptions extracted from it are stored in a preliminary list of idle states
coming from the ACPI tables.  [This step is skipped if ``intel_idle`` is
configured to ignore the ACPI tables; see `below <intel-idle-parameters_>`_.]

Next, the first (index 0) entry in the list of available idle states is
initialized to represent a "polling idle state" (a pseudo-idle state in which
the target CPU continuously fetches and executes instructions), and the
subsequent (real) idle state entries are populated as follows.

If the processor model at hand is recognized by ``intel_idle``, there is a
(static) table of idle state descriptions for it in the driver.  In that case,
the "internal" table is the primary source of information on idle states and the
information from it is copied to the final list of available idle states.  If
using the ACPI tables for the enumeration of idle states is not required
(depending on the processor model), all of the listed idle state are enabled by
default (so all of them will be taken into consideration by ``CPUIdle``
governors during CPU idle state selection).  Otherwise, some of the listed idle
states may not be enabled by default if there are no matching entries in the
preliminary list of idle states coming from the ACPI tables.  In that case user
space still can enable them later (on a per-CPU basis) with the help of
the ``disable`` idle state attribute in ``sysfs`` (see
:ref:`idle-states-representation` in
Documentation/admin-guide/pm/cpuidle.rst).  This basically means that
the idle states "known" to the driver may not be enabled by default if they have
not been exposed by the platform firmware (through the ACPI tables).

If the given processor model is not recognized by ``intel_idle``, but it
supports ``MWAIT``, the preliminary list of idle states coming from the ACPI
tables is used for building the final list that will be supplied to the
``CPUIdle`` core during driver registration.  For each idle state in that list,
the description, ``MWAIT`` hint and exit latency are copied to the corresponding
entry in the final list of idle states.  The name of the idle state represented
by it (to be returned by the ``name`` idle state attribute in ``sysfs``) is
"CX_ACPI", where X is the index of that idle state in the final list (note that
the minimum value of X is 1, because 0 is reserved for the "polling" state), and
its target residency is based on the exit latency value.  Specifically, for
C1-type idle states the exit latency value is also used as the target residency
(for compatibility with the majority of the "internal" tables of idle states for
various processor models recognized by ``intel_idle``) and for the other idle
state types (C2 and C3) the target residency value is 3 times the exit latency
(again, that is because it reflects the target residency to exit latency ratio
in the majority of cases for the processor models recognized by ``intel_idle``).
All of the idle states in the final list are enabled by default in this case.

.. _intel-idle-initialization:

Initialization
==============

The initialization of ``intel_idle`` starts with checking if the kernel command
line options forbid the use of the ``MWAIT`` instruction.  If that is the case,
an error code is returned right away.

The next step is to check whether or not the processor model is known to the
driver, which determines the idle states enumeration method (see
`above <intel-idle-enumeration-of-states_>`_), and whether or not the processor
supports ``MWAIT`` (the initialization fails if that is not the case).  Then,
the ``MWAIT`` support in the processor is enumerated through ``CPUID`` and the
driver initialization fails if the level of support is not as expected (for
example, if the total number of ``MWAIT`` substates returned is 0).

Next, if the driver is not configured to ignore the ACPI tables (see
`below <intel-idle-parameters_>`_), the idle states information provided by the
platform firmware is extracted from them.

Then, ``CPUIdle`` device objects are allocated for all CPUs and the list of
available idle states is created as explained
`above <intel-idle-enumeration-of-states_>`_.

Finally, ``intel_idle`` is registered with the help of cpuidle_register_driver()
as the ``CPUIdle`` driver for all CPUs in the system and a CPU online callback
for configuring individual CPUs is registered via cpuhp_setup_state(), which
(among other things) causes the callback routine to be invoked for all of the
CPUs present in the system at that time (each CPU executes its own instance of
the callback routine).  That routine registers a ``CPUIdle`` device for the CPU
running it (which enables the ``CPUIdle`` subsystem to operate that CPU) and
optionally performs some CPU-specific initialization actions that may be
required for the given processor model.

.. _intel-idle-parameters:

Kernel Command Line Options and Module Parameters
=================================================

The *x86* architecture support code recognizes three kernel command line
options related to CPU idle time management: ``idle=poll``, ``idle=halt``,
and ``idle=nomwait``.  If any of them is present in the kernel command line, the
``MWAIT`` instruction is not allowed to be used, so the initialization of
``intel_idle`` will fail.

Apart from that there are four module parameters recognized by ``intel_idle``
itself that can be set via the kernel command line (they cannot be updated via
sysfs, so that is the only way to change their values).

The ``max_cstate`` parameter value is the maximum idle state index in the list
of idle states supplied to the ``CPUIdle`` core during the registration of the
driver.  It is also the maximum number of regular (non-polling) idle states that
can be used by ``intel_idle``, so the enumeration of idle states is terminated
after finding that number of usable idle states (the other idle states that
potentially might have been used if ``max_cstate`` had been greater are not
taken into consideration at all).  Setting ``max_cstate`` can prevent
``intel_idle`` from exposing idle states that are regarded as "too deep" for
some reason to the ``CPUIdle`` core, but it does so by making them effectively
invisible until the system is shut down and started again which may not always
be desirable.  In practice, it is only really necessary to do that if the idle
states in question cannot be enabled during system startup, because in the
working state of the system the CPU power management quality of service (PM
QoS) feature can be used to prevent ``CPUIdle`` from touching those idle states
even if they have been enumerated (see :ref:`cpu-pm-qos` in
Documentation/admin-guide/pm/cpuidle.rst).
Setting ``max_cstate`` to 0 causes the ``intel_idle`` initialization to fail.

The ``no_acpi`` and ``use_acpi`` module parameters (recognized by ``intel_idle``
if the kernel has been configured with ACPI support) can be set to make the
driver ignore the system's ACPI tables entirely or use them for all of the
recognized processor models, respectively (they both are unset by default and
``use_acpi`` has no effect if ``no_acpi`` is set).

The value of the ``states_off`` module parameter (0 by default) represents a
list of idle states to be disabled by default in the form of a bitmask.

Namely, the positions of the bits that are set in the ``states_off`` value are
the indices of idle states to be disabled by default (as reflected by the names
of the corresponding idle state directories in ``sysfs``, :file:`state0`,
:file:`state1` ... :file:`state<i>` ..., where ``<i>`` is the index of the given
idle state; see :ref:`idle-states-representation` in
Documentation/admin-guide/pm/cpuidle.rst).

For example, if ``states_off`` is equal to 3, the driver will disable idle
states 0 and 1 by default, and if it is equal to 8, idle state 3 will be
disabled by default and so on (bit positions beyond the maximum idle state index
are ignored).

The idle states disabled this way can be enabled (on a per-CPU basis) from user
space via ``sysfs``.

.. _intel-idle-core-and-package-idle-states:

Core and Package Levels of Idle States
======================================

Typically, in a processor supporting the ``MWAIT`` instruction there are (at
least) two levels of idle states (or C-states).  One level, referred to as
"core C-states", covers individual cores in the processor, whereas the other
level, referred to as "package C-states", covers the entire processor package
and it may also involve other components of the system (GPUs, memory
controllers, I/O hubs etc.).

Some of the ``MWAIT`` hint values allow the processor to use core C-states only
(most importantly, that is the case for the ``MWAIT`` hint value corresponding
to the ``C1`` idle state), but the majority of them give it a license to put
the target core (i.e. the core containing the logical CPU executing ``MWAIT``
with the given hint value) into a specific core C-state and then (if possible)
to enter a specific package C-state at the deeper level.  For example, the
``MWAIT`` hint value representing the ``C3`` idle state allows the processor to
put the target core into the low-power state referred to as "core ``C3``" (or
``CC3``), which happens if all of the logical CPUs (SMT siblings) in that core
have executed ``MWAIT`` with the ``C3`` hint value (or with a hint value
representing a deeper idle state), and in addition to that (in the majority of
cases) it gives the processor a license to put the entire package (possibly
including some non-CPU components such as a GPU or a memory controller) into the
low-power state referred to as "package ``C3``" (or ``PC3``), which happens if
all of the cores have gone into the ``CC3`` state and (possibly) some additional
conditions are satisfied (for instance, if the GPU is covered by ``PC3``, it may
be required to be in a certain GPU-specific low-power state for ``PC3`` to be
reachable).

As a rule, there is no simple way to make the processor use core C-states only
if the conditions for entering the corresponding package C-states are met, so
the logical CPU executing ``MWAIT`` with a hint value that is not core-level
only (like for ``C1``) must always assume that this may cause the processor to
enter a package C-state.  [That is why the exit latency and target residency
values corresponding to the majority of ``MWAIT`` hint values in the "internal"
tables of idle states in ``intel_idle`` reflect the properties of package
C-states.]  If using package C-states is not desirable at all, either
:ref:`PM QoS <cpu-pm-qos>` or the ``max_cstate`` module parameter of
``intel_idle`` described `above <intel-idle-parameters_>`_ must be used to
restrict the range of permissible idle states to the ones with core-level only
``MWAIT`` hint values (like ``C1``).

References
==========

.. [1] *Intel® 64 and IA-32 Architectures Software Developer’s Manual Volume 2B*,
       https://www.intel.com/content/www/us/en/architecture-and-technology/64-ia-32-architectures-software-developer-vol-2b-manual.html

.. [2] *Advanced Configuration and Power Interface (ACPI) Specification*,
       https://uefi.org/specifications