code:
===========================================
Seccomp BPF (SECure COMPuting with filters)
===========================================

Introduction
============

A large number of system calls are exposed to every userland process
with many of them going unused for the entire lifetime of the process.
As system calls change and mature, bugs are found and eradicated.  A
certain subset of userland applications benefit by having a reduced set
of available system calls.  The resulting set reduces the total kernel
surface exposed to the application.  System call filtering is meant for
use with those applications.

Seccomp filtering provides a means for a process to specify a filter for
incoming system calls.  The filter is expressed as a Berkeley Packet
Filter (BPF) program, as with socket filters, except that the data
operated on is related to the system call being made: system call
number and the system call arguments.  This allows for expressive
filtering of system calls using a filter program language with a long
history of being exposed to userland and a straightforward data set.

Additionally, BPF makes it impossible for users of seccomp to fall prey
to time-of-check-time-of-use (TOCTOU) attacks that are common in system
call interposition frameworks.  BPF programs may not dereference
pointers which constrains all filters to solely evaluating the system
call arguments directly.

What it isn't
=============

System call filtering isn't a sandbox.  It provides a clearly defined
mechanism for minimizing the exposed kernel surface.  It is meant to be
a tool for sandbox developers to use.  Beyond that, policy for logical
behavior and information flow should be managed with a combination of
other system hardening techniques and, potentially, an LSM of your
choosing.  Expressive, dynamic filters provide further options down this
path (avoiding pathological sizes or selecting which of the multiplexed
system calls in socketcall() is allowed, for instance) which could be
construed, incorrectly, as a more complete sandboxing solution.

Usage
=====

An additional seccomp mode is added and is enabled using the same
prctl(2) call as the strict seccomp.  If the architecture has
``CONFIG_HAVE_ARCH_SECCOMP_FILTER``, then filters may be added as below:

``PR_SET_SECCOMP``:
	Now takes an additional argument which specifies a new filter
	using a BPF program.
	The BPF program will be executed over struct seccomp_data
	reflecting the system call number, arguments, and other
	metadata.  The BPF program must then return one of the
	acceptable values to inform the kernel which action should be
	taken.

	Usage::

		prctl(PR_SET_SECCOMP, SECCOMP_MODE_FILTER, prog);

	The 'prog' argument is a pointer to a struct sock_fprog which
	will contain the filter program.  If the program is invalid, the
	call will return -1 and set errno to ``EINVAL``.

	If ``fork``/``clone`` and ``execve`` are allowed by @prog, any child
	processes will be constrained to the same filters and system
	call ABI as the parent.

	Prior to use, the task must call ``prctl(PR_SET_NO_NEW_PRIVS, 1)`` or
	run with ``CAP_SYS_ADMIN`` privileges in its namespace.  If these are not
	true, ``-EACCES`` will be returned.  This requirement ensures that filter
	programs cannot be applied to child processes with greater privileges
	than the task that installed them.

	Additionally, if ``prctl(2)`` is allowed by the attached filter,
	additional filters may be layered on which will increase evaluation
	time, but allow for further decreasing the attack surface during
	execution of a process.

The above call returns 0 on success and non-zero on error.

Return values
=============

A seccomp filter may return any of the following values. If multiple
filters exist, the return value for the evaluation of a given system
call will always use the highest precedent value. (For example,
``SECCOMP_RET_KILL_PROCESS`` will always take precedence.)

In precedence order, they are:

``SECCOMP_RET_KILL_PROCESS``:
	Results in the entire process exiting immediately without executing
	the system call.  The exit status of the task (``status & 0x7f``)
	will be ``SIGSYS``, not ``SIGKILL``.

``SECCOMP_RET_KILL_THREAD``:
	Results in the task exiting immediately without executing the
	system call.  The exit status of the task (``status & 0x7f``) will
	be ``SIGSYS``, not ``SIGKILL``.

``SECCOMP_RET_TRAP``:
	Results in the kernel sending a ``SIGSYS`` signal to the triggering
	task without executing the system call. ``siginfo->si_call_addr``
	will show the address of the system call instruction, and
	``siginfo->si_syscall`` and ``siginfo->si_arch`` will indicate which
	syscall was attempted.  The program counter will be as though
	the syscall happened (i.e. it will not point to the syscall
	instruction).  The return value register will contain an arch-
	dependent value -- if resuming execution, set it to something
	sensible.  (The architecture dependency is because replacing
	it with ``-ENOSYS`` could overwrite some useful information.)

	The ``SECCOMP_RET_DATA`` portion of the return value will be passed
	as ``si_errno``.

	``SIGSYS`` triggered by seccomp will have a si_code of ``SYS_SECCOMP``.

``SECCOMP_RET_ERRNO``:
	Results in the lower 16-bits of the return value being passed
	to userland as the errno without executing the system call.

``SECCOMP_RET_USER_NOTIF``:
	Results in a ``struct seccomp_notif`` message sent on the userspace
	notification fd, if it is attached, or ``-ENOSYS`` if it is not. See
	below on discussion of how to handle user notifications.

``SECCOMP_RET_TRACE``:
	When returned, this value will cause the kernel to attempt to
	notify a ``ptrace()``-based tracer prior to executing the system
	call.  If there is no tracer present, ``-ENOSYS`` is returned to
	userland and the system call is not executed.

	A tracer will be notified if it requests ``PTRACE_O_TRACESECCOMP``
	using ``ptrace(PTRACE_SETOPTIONS)``.  The tracer will be notified
	of a ``PTRACE_EVENT_SECCOMP`` and the ``SECCOMP_RET_DATA`` portion of
	the BPF program return value will be available to the tracer
	via ``PTRACE_GETEVENTMSG``.

	The tracer can skip the system call by changing the syscall number
	to -1.  Alternatively, the tracer can change the system call
	requested by changing the system call to a valid syscall number.  If
	the tracer asks to skip the system call, then the system call will
	appear to return the value that the tracer puts in the return value
	register.

	The seccomp check will not be run again after the tracer is
	notified.  (This means that seccomp-based sandboxes MUST NOT
	allow use of ptrace, even of other sandboxed processes, without
	extreme care; ptracers can use this mechanism to escape.)

``SECCOMP_RET_LOG``:
	Results in the system call being executed after it is logged. This
	should be used by application developers to learn which syscalls their
	application needs without having to iterate through multiple test and
	development cycles to build the list.

	This action will only be logged if "log" is present in the
	actions_logged sysctl string.

``SECCOMP_RET_ALLOW``:
	Results in the system call being executed.

If multiple filters exist, the return value for the evaluation of a
given system call will always use the highest precedent value.

Precedence is only determined using the ``SECCOMP_RET_ACTION`` mask.  When
multiple filters return values of the same precedence, only the
``SECCOMP_RET_DATA`` from the most recently installed filter will be
returned.

Pitfalls
========

The biggest pitfall to avoid during use is filtering on system call
number without checking the architecture value.  Why?  On any
architecture that supports multiple system call invocation conventions,
the system call numbers may vary based on the specific invocation.  If
the numbers in the different calling conventions overlap, then checks in
the filters may be abused.  Always check the arch value!

Example
=======

The ``samples/seccomp/`` directory contains both an x86-specific example
and a more generic example of a higher level macro interface for BPF
program generation.

Userspace Notification
======================

The ``SECCOMP_RET_USER_NOTIF`` return code lets seccomp filters pass a
particular syscall to userspace to be handled. This may be useful for
applications like container managers, which wish to intercept particular
syscalls (``mount()``, ``finit_module()``, etc.) and change their behavior.

To acquire a notification FD, use the ``SECCOMP_FILTER_FLAG_NEW_LISTENER``
argument to the ``seccomp()`` syscall:

.. code-block:: c

    fd = seccomp(SECCOMP_SET_MODE_FILTER, SECCOMP_FILTER_FLAG_NEW_LISTENER, &prog);

which (on success) will return a listener fd for the filter, which can then be
passed around via ``SCM_RIGHTS`` or similar. Note that filter fds correspond to
a particular filter, and not a particular task. So if this task then forks,
notifications from both tasks will appear on the same filter fd. Reads and
writes to/from a filter fd are also synchronized, so a filter fd can safely
have many readers.

The interface for a seccomp notification fd consists of two structures:

.. code-block:: c

    struct seccomp_notif_sizes {
        __u16 seccomp_notif;
        __u16 seccomp_notif_resp;
        __u16 seccomp_data;
    };

    struct seccomp_notif {
        __u64 id;
        __u32 pid;
        __u32 flags;
        struct seccomp_data data;
    };

    struct seccomp_notif_resp {
        __u64 id;
        __s64 val;
        __s32 error;
        __u32 flags;
    };

The ``struct seccomp_notif_sizes`` structure can be used to determine the size
of the various structures used in seccomp notifications. The size of ``struct
seccomp_data`` may change in the future, so code should use:

.. code-block:: c

    struct seccomp_notif_sizes sizes;
    seccomp(SECCOMP_GET_NOTIF_SIZES, 0, &sizes);

to determine the size of the various structures to allocate. See
samples/seccomp/user-trap.c for an example.

Users can read via ``ioctl(SECCOMP_IOCTL_NOTIF_RECV)``  (or ``poll()``) on a
seccomp notification fd to receive a ``struct seccomp_notif``, which contains
five members: the input length of the structure, a unique-per-filter ``id``,
the ``pid`` of the task which triggered this request (which may be 0 if the
task is in a pid ns not visible from the listener's pid namespace). The
notification also contains the ``data`` passed to seccomp, and a filters flag.
The structure should be zeroed out prior to calling the ioctl.

Userspace can then make a decision based on this information about what to do,
and ``ioctl(SECCOMP_IOCTL_NOTIF_SEND)`` a response, indicating what should be
returned to userspace. The ``id`` member of ``struct seccomp_notif_resp`` should
be the same ``id`` as in ``struct seccomp_notif``.

Userspace can also add file descriptors to the notifying process via
``ioctl(SECCOMP_IOCTL_NOTIF_ADDFD)``. The ``id`` member of
``struct seccomp_notif_addfd`` should be the same ``id`` as in
``struct seccomp_notif``. The ``newfd_flags`` flag may be used to set flags
like O_CLOEXEC on the file descriptor in the notifying process. If the supervisor
wants to inject the file descriptor with a specific number, the
``SECCOMP_ADDFD_FLAG_SETFD`` flag can be used, and set the ``newfd`` member to
the specific number to use. If that file descriptor is already open in the
notifying process it will be replaced. The supervisor can also add an FD, and
respond atomically by using the ``SECCOMP_ADDFD_FLAG_SEND`` flag and the return
value will be the injected file descriptor number.

The notifying process can be preempted, resulting in the notification being
aborted. This can be problematic when trying to take actions on behalf of the
notifying process that are long-running and typically retryable (mounting a
filesytem). Alternatively, at filter installation time, the
``SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV`` flag can be set. This flag makes it
such that when a user notification is received by the supervisor, the notifying
process will ignore non-fatal signals until the response is sent. Signals that
are sent prior to the notification being received by userspace are handled
normally.

It is worth noting that ``struct seccomp_data`` contains the values of register
arguments to the syscall, but does not contain pointers to memory. The task's
memory is accessible to suitably privileged traces via ``ptrace()`` or
``/proc/pid/mem``. However, care should be taken to avoid the TOCTOU mentioned
above in this document: all arguments being read from the tracee's memory
should be read into the tracer's memory before any policy decisions are made.
This allows for an atomic decision on syscall arguments.

Sysctls
=======

Seccomp's sysctl files can be found in the ``/proc/sys/kernel/seccomp/``
directory. Here's a description of each file in that directory:

``actions_avail``:
	A read-only ordered list of seccomp return values (refer to the
	``SECCOMP_RET_*`` macros above) in string form. The ordering, from
	left-to-right, is the least permissive return value to the most
	permissive return value.

	The list represents the set of seccomp return values supported
	by the kernel. A userspace program may use this list to
	determine if the actions found in the ``seccomp.h``, when the
	program was built, differs from the set of actions actually
	supported in the current running kernel.

``actions_logged``:
	A read-write ordered list of seccomp return values (refer to the
	``SECCOMP_RET_*`` macros above) that are allowed to be logged. Writes
	to the file do not need to be in ordered form but reads from the file
	will be ordered in the same way as the actions_avail sysctl.

	The ``allow`` string is not accepted in the ``actions_logged`` sysctl
	as it is not possible to log ``SECCOMP_RET_ALLOW`` actions. Attempting
	to write ``allow`` to the sysctl will result in an EINVAL being
	returned.

Adding architecture support
===========================

See ``arch/Kconfig`` for the authoritative requirements.  In general, if an
architecture supports both ptrace_event and seccomp, it will be able to
support seccomp filter with minor fixup: ``SIGSYS`` support and seccomp return
value checking.  Then it must just add ``CONFIG_HAVE_ARCH_SECCOMP_FILTER``
to its arch-specific Kconfig.

Caveats
=======

The vDSO can cause some system calls to run entirely in userspace,
leading to surprises when you run programs on different machines that
fall back to real syscalls.  To minimize these surprises on x86, make
sure you test with
``/sys/devices/system/clocksource/clocksource0/current_clocksource`` set to
something like ``acpi_pm``.

On x86-64, vsyscall emulation is enabled by default.  (vsyscalls are
legacy variants on vDSO calls.)  Currently, emulated vsyscalls will
honor seccomp, with a few oddities:

- A return value of ``SECCOMP_RET_TRAP`` will set a ``si_call_addr`` pointing to
  the vsyscall entry for the given call and not the address after the
  'syscall' instruction.  Any code which wants to restart the call
  should be aware that (a) a ret instruction has been emulated and (b)
  trying to resume the syscall will again trigger the standard vsyscall
  emulation security checks, making resuming the syscall mostly
  pointless.

- A return value of ``SECCOMP_RET_TRACE`` will signal the tracer as usual,
  but the syscall may not be changed to another system call using the
  orig_rax register. It may only be changed to -1 order to skip the
  currently emulated call. Any other change MAY terminate the process.
  The rip value seen by the tracer will be the syscall entry address;
  this is different from normal behavior.  The tracer MUST NOT modify
  rip or rsp.  (Do not rely on other changes terminating the process.
  They might work.  For example, on some kernels, choosing a syscall
  that only exists in future kernels will be correctly emulated (by
  returning ``-ENOSYS``).

To detect this quirky behavior, check for ``addr & ~0x0C00 ==
0xFFFFFFFFFF600000``.  (For ``SECCOMP_RET_TRACE``, use rip.  For
``SECCOMP_RET_TRAP``, use ``siginfo->si_call_addr``.)  Do not check any other
condition: future kernels may improve vsyscall emulation and current
kernels in vsyscall=native mode will behave differently, but the
instructions at ``0xF...F600{0,4,8,C}00`` will not be system calls in these
cases.

Note that modern systems are unlikely to use vsyscalls at all -- they
are a legacy feature and they are considerably slower than standard
syscalls.  New code will use the vDSO, and vDSO-issued system calls
are indistinguishable from normal system calls.


code:
===================
Speculation Control
===================

Quite some CPUs have speculation-related misfeatures which are in
fact vulnerabilities causing data leaks in various forms even across
privilege domains.

The kernel provides mitigation for such vulnerabilities in various
forms. Some of these mitigations are compile-time configurable and some
can be supplied on the kernel command line.

There is also a class of mitigations which are very expensive, but they can
be restricted to a certain set of processes or tasks in controlled
environments. The mechanism to control these mitigations is via
:manpage:`prctl(2)`.

There are two prctl options which are related to this:

 * PR_GET_SPECULATION_CTRL

 * PR_SET_SPECULATION_CTRL

PR_GET_SPECULATION_CTRL
-----------------------

PR_GET_SPECULATION_CTRL returns the state of the speculation misfeature
which is selected with arg2 of prctl(2). The return value uses bits 0-3 with
the following meaning:

==== ====================== ==================================================
Bit  Define                 Description
==== ====================== ==================================================
0    PR_SPEC_PRCTL          Mitigation can be controlled per task by
                            PR_SET_SPECULATION_CTRL.
1    PR_SPEC_ENABLE         The speculation feature is enabled, mitigation is
                            disabled.
2    PR_SPEC_DISABLE        The speculation feature is disabled, mitigation is
                            enabled.
3    PR_SPEC_FORCE_DISABLE  Same as PR_SPEC_DISABLE, but cannot be undone. A
                            subsequent prctl(..., PR_SPEC_ENABLE) will fail.
4    PR_SPEC_DISABLE_NOEXEC Same as PR_SPEC_DISABLE, but the state will be
                            cleared on :manpage:`execve(2)`.
==== ====================== ==================================================

If all bits are 0 the CPU is not affected by the speculation misfeature.

If PR_SPEC_PRCTL is set, then the per-task control of the mitigation is
available. If not set, prctl(PR_SET_SPECULATION_CTRL) for the speculation
misfeature will fail.

.. _set_spec_ctrl:

PR_SET_SPECULATION_CTRL
-----------------------

PR_SET_SPECULATION_CTRL allows to control the speculation misfeature, which
is selected by arg2 of :manpage:`prctl(2)` per task. arg3 is used to hand
in the control value, i.e. either PR_SPEC_ENABLE or PR_SPEC_DISABLE or
PR_SPEC_FORCE_DISABLE.

Common error codes
------------------
======= =================================================================
Value   Meaning
======= =================================================================
EINVAL  The prctl is not implemented by the architecture or unused
        prctl(2) arguments are not 0.

ENODEV  arg2 is selecting a not supported speculation misfeature.
======= =================================================================

PR_SET_SPECULATION_CTRL error codes
-----------------------------------
======= =================================================================
Value   Meaning
======= =================================================================
0       Success

ERANGE  arg3 is incorrect, i.e. it's neither PR_SPEC_ENABLE nor
        PR_SPEC_DISABLE nor PR_SPEC_FORCE_DISABLE.

ENXIO   Control of the selected speculation misfeature is not possible.
        See PR_GET_SPECULATION_CTRL.

EPERM   Speculation was disabled with PR_SPEC_FORCE_DISABLE and caller
        tried to enable it again.
======= =================================================================

Speculation misfeature controls
-------------------------------
- PR_SPEC_STORE_BYPASS: Speculative Store Bypass

  Invocations:
   * prctl(PR_GET_SPECULATION_CTRL, PR_SPEC_STORE_BYPASS, 0, 0, 0);
   * prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_STORE_BYPASS, PR_SPEC_ENABLE, 0, 0);
   * prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_STORE_BYPASS, PR_SPEC_DISABLE, 0, 0);
   * prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_STORE_BYPASS, PR_SPEC_FORCE_DISABLE, 0, 0);
   * prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_STORE_BYPASS, PR_SPEC_DISABLE_NOEXEC, 0, 0);

- PR_SPEC_INDIR_BRANCH: Indirect Branch Speculation in User Processes
                        (Mitigate Spectre V2 style attacks against user processes)

  Invocations:
   * prctl(PR_GET_SPECULATION_CTRL, PR_SPEC_INDIRECT_BRANCH, 0, 0, 0);
   * prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_INDIRECT_BRANCH, PR_SPEC_ENABLE, 0, 0);
   * prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_INDIRECT_BRANCH, PR_SPEC_DISABLE, 0, 0);
   * prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_INDIRECT_BRANCH, PR_SPEC_FORCE_DISABLE, 0, 0);

- PR_SPEC_L1D_FLUSH: Flush L1D Cache on context switch out of the task
                        (works only when tasks run on non SMT cores)

  Invocations:
   * prctl(PR_GET_SPECULATION_CTRL, PR_SPEC_L1D_FLUSH, 0, 0, 0);
   * prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_L1D_FLUSH, PR_SPEC_ENABLE, 0, 0);
   * prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_L1D_FLUSH, PR_SPEC_DISABLE, 0, 0);

repo_name: angery/linux


code:
=====================================================================
Platform Profile Selection (e.g. /sys/firmware/acpi/platform_profile)
=====================================================================

On modern systems the platform performance, temperature, fan and other
hardware related characteristics are often dynamically configurable. The
platform configuration is often automatically adjusted to the current
conditions by some automatic mechanism (which may very well live outside
the kernel).

These auto platform adjustment mechanisms often can be configured with
one of several platform profiles, with either a bias towards low power
operation or towards performance.

The purpose of the platform_profile attribute is to offer a generic sysfs
API for selecting the platform profile of these automatic mechanisms.

Note that this API is only for selecting the platform profile, it is
NOT a goal of this API to allow monitoring the resulting performance
characteristics. Monitoring performance is best done with device/vendor
specific tools such as e.g. turbostat.

Specifically when selecting a high performance profile the actual achieved
performance may be limited by various factors such as: the heat generated
by other components, room temperature, free air flow at the bottom of a
laptop, etc. It is explicitly NOT a goal of this API to let userspace know
about any sub-optimal conditions which are impeding reaching the requested
performance level.

Since numbers on their own cannot represent the multiple variables that a
profile will adjust (power consumption, heat generation, etc) this API
uses strings to describe the various profiles. To make sure that userspace
gets a consistent experience the sysfs-platform_profile ABI document defines
a fixed set of profile names. Drivers *must* map their internal profile
representation onto this fixed set.

If there is no good match when mapping then a new profile name may be
added. Drivers which wish to introduce new profile names must:

 1. Explain why the existing profile names canot be used.
 2. Add the new profile name, along with a clear description of the
    expected behaviour, to the sysfs-platform_profile ABI documentation.


code:
unshare system call
===================

This document describes the new system call, unshare(). The document
provides an overview of the feature, why it is needed, how it can
be used, its interface specification, design, implementation and
how it can be tested.

Change Log
----------
version 0.1  Initial document, Janak Desai (janak@us.ibm.com), Jan 11, 2006

Contents
--------
	1) Overview
	2) Benefits
	3) Cost
	4) Requirements
	5) Functional Specification
	6) High Level Design
	7) Low Level Design
	8) Test Specification
	9) Future Work

1) Overview
-----------

Most legacy operating system kernels support an abstraction of threads
as multiple execution contexts within a process. These kernels provide
special resources and mechanisms to maintain these "threads". The Linux
kernel, in a clever and simple manner, does not make distinction
between processes and "threads". The kernel allows processes to share
resources and thus they can achieve legacy "threads" behavior without
requiring additional data structures and mechanisms in the kernel. The
power of implementing threads in this manner comes not only from
its simplicity but also from allowing application programmers to work
outside the confinement of all-or-nothing shared resources of legacy
threads. On Linux, at the time of thread creation using the clone system
call, applications can selectively choose which resources to share
between threads.

unshare() system call adds a primitive to the Linux thread model that
allows threads to selectively 'unshare' any resources that were being
shared at the time of their creation. unshare() was conceptualized by
Al Viro in the August of 2000, on the Linux-Kernel mailing list, as part
of the discussion on POSIX threads on Linux.  unshare() augments the
usefulness of Linux threads for applications that would like to control
shared resources without creating a new process. unshare() is a natural
addition to the set of available primitives on Linux that implement
the concept of process/thread as a virtual machine.

2) Benefits
-----------

unshare() would be useful to large application frameworks such as PAM
where creating a new process to control sharing/unsharing of process
resources is not possible. Since namespaces are shared by default
when creating a new process using fork or clone, unshare() can benefit
even non-threaded applications if they have a need to disassociate
from default shared namespace. The following lists two use-cases
where unshare() can be used.

2.1 Per-security context namespaces
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

unshare() can be used to implement polyinstantiated directories using
the kernel's per-process namespace mechanism. Polyinstantiated directories,
such as per-user and/or per-security context instance of /tmp, /var/tmp or
per-security context instance of a user's home directory, isolate user
processes when working with these directories. Using unshare(), a PAM
module can easily setup a private namespace for a user at login.
Polyinstantiated directories are required for Common Criteria certification
with Labeled System Protection Profile, however, with the availability
of shared-tree feature in the Linux kernel, even regular Linux systems
can benefit from setting up private namespaces at login and
polyinstantiating /tmp, /var/tmp and other directories deemed
appropriate by system administrators.

2.2 unsharing of virtual memory and/or open files
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Consider a client/server application where the server is processing
client requests by creating processes that share resources such as
virtual memory and open files. Without unshare(), the server has to
decide what needs to be shared at the time of creating the process
which services the request. unshare() allows the server an ability to
disassociate parts of the context during the servicing of the
request. For large and complex middleware application frameworks, this
ability to unshare() after the process was created can be very
useful.

3) Cost
-------

In order to not duplicate code and to handle the fact that unshare()
works on an active task (as opposed to clone/fork working on a newly
allocated inactive task) unshare() had to make minor reorganizational
changes to copy_* functions utilized by clone/fork system call.
There is a cost associated with altering existing, well tested and
stable code to implement a new feature that may not get exercised
extensively in the beginning. However, with proper design and code
review of the changes and creation of an unshare() test for the LTP
the benefits of this new feature can exceed its cost.

4) Requirements
---------------

unshare() reverses sharing that was done using clone(2) system call,
so unshare() should have a similar interface as clone(2). That is,
since flags in clone(int flags, void \*stack) specifies what should
be shared, similar flags in unshare(int flags) should specify
what should be unshared. Unfortunately, this may appear to invert
the meaning of the flags from the way they are used in clone(2).
However, there was no easy solution that was less confusing and that
allowed incremental context unsharing in future without an ABI change.

unshare() interface should accommodate possible future addition of
new context flags without requiring a rebuild of old applications.
If and when new context flags are added, unshare() design should allow
incremental unsharing of those resources on an as needed basis.

5) Functional Specification
---------------------------

NAME
	unshare - disassociate parts of the process execution context

SYNOPSIS
	#include <sched.h>

	int unshare(int flags);

DESCRIPTION
	unshare() allows a process to disassociate parts of its execution
	context that are currently being shared with other processes. Part
	of execution context, such as the namespace, is shared by default
	when a new process is created using fork(2), while other parts,
	such as the virtual memory, open file descriptors, etc, may be
	shared by explicit request to share them when creating a process
	using clone(2).

	The main use of unshare() is to allow a process to control its
	shared execution context without creating a new process.

	The flags argument specifies one or bitwise-or'ed of several of
	the following constants.

	CLONE_FS
		If CLONE_FS is set, file system information of the caller
		is disassociated from the shared file system information.

	CLONE_FILES
		If CLONE_FILES is set, the file descriptor table of the
		caller is disassociated from the shared file descriptor
		table.

	CLONE_NEWNS
		If CLONE_NEWNS is set, the namespace of the caller is
		disassociated from the shared namespace.

	CLONE_VM
		If CLONE_VM is set, the virtual memory of the caller is
		disassociated from the shared virtual memory.

RETURN VALUE
	On success, zero returned. On failure, -1 is returned and errno is

ERRORS
	EPERM	CLONE_NEWNS was specified by a non-root process (process
		without CAP_SYS_ADMIN).

	ENOMEM	Cannot allocate sufficient memory to copy parts of caller's
		context that need to be unshared.

	EINVAL	Invalid flag was specified as an argument.

CONFORMING TO
	The unshare() call is Linux-specific and  should  not be used
	in programs intended to be portable.

SEE ALSO
	clone(2), fork(2)

6) High Level Design
--------------------

Depending on the flags argument, the unshare() system call allocates
appropriate process context structures, populates it with values from
the current shared version, associates newly duplicated structures
with the current task structure and releases corresponding shared
versions. Helper functions of clone (copy_*) could not be used
directly by unshare() because of the following two reasons.

  1) clone operates on a newly allocated not-yet-active task
     structure, where as unshare() operates on the current active
     task. Therefore unshare() has to take appropriate task_lock()
     before associating newly duplicated context structures

  2) unshare() has to allocate and duplicate all context structures
     that are being unshared, before associating them with the
     current task and releasing older shared structures. Failure
     do so will create race conditions and/or oops when trying
     to backout due to an error. Consider the case of unsharing
     both virtual memory and namespace. After successfully unsharing
     vm, if the system call encounters an error while allocating
     new namespace structure, the error return code will have to
     reverse the unsharing of vm. As part of the reversal the
     system call will have to go back to older, shared, vm
     structure, which may not exist anymore.

Therefore code from copy_* functions that allocated and duplicated
current context structure was moved into new dup_* functions. Now,
copy_* functions call dup_* functions to allocate and duplicate
appropriate context structures and then associate them with the
task structure that is being constructed. unshare() system call on
the other hand performs the following:

  1) Check flags to force missing, but implied, flags

  2) For each context structure, call the corresponding unshare()
     helper function to allocate and duplicate a new context
     structure, if the appropriate bit is set in the flags argument.

  3) If there is no error in allocation and duplication and there
     are new context structures then lock the current task structure,
     associate new context structures with the current task structure,
     and release the lock on the current task structure.

  4) Appropriately release older, shared, context structures.

7) Low Level Design
-------------------

Implementation of unshare() can be grouped in the following 4 different
items:

  a) Reorganization of existing copy_* functions

  b) unshare() system call service function

  c) unshare() helper functions for each different process context

  d) Registration of system call number for different architectures

7.1) Reorganization of copy_* functions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Each copy function such as copy_mm, copy_namespace, copy_files,
etc, had roughly two components. The first component allocated
and duplicated the appropriate structure and the second component
linked it to the task structure passed in as an argument to the copy
function. The first component was split into its own function.
These dup_* functions allocated and duplicated the appropriate
context structure. The reorganized copy_* functions invoked
their corresponding dup_* functions and then linked the newly
duplicated structures to the task structure with which the
copy function was called.

7.2) unshare() system call service function
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

       * Check flags
	 Force implied flags. If CLONE_THREAD is set force CLONE_VM.
	 If CLONE_VM is set, force CLONE_SIGHAND. If CLONE_SIGHAND is
	 set and signals are also being shared, force CLONE_THREAD. If
	 CLONE_NEWNS is set, force CLONE_FS.

       * For each context flag, invoke the corresponding unshare_*
	 helper routine with flags passed into the system call and a
	 reference to pointer pointing the new unshared structure

       * If any new structures are created by unshare_* helper
	 functions, take the task_lock() on the current task,
	 modify appropriate context pointers, and release the
         task lock.

       * For all newly unshared structures, release the corresponding
         older, shared, structures.

7.3) unshare_* helper functions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

For unshare_* helpers corresponding to CLONE_SYSVSEM, CLONE_SIGHAND,
and CLONE_THREAD, return -EINVAL since they are not implemented yet.
For others, check the flag value to see if the unsharing is
required for that structure. If it is, invoke the corresponding
dup_* function to allocate and duplicate the structure and return
a pointer to it.

7.4) Finally
~~~~~~~~~~~~

Appropriately modify architecture specific code to register the
new system call.

8) Test Specification
---------------------

The test for unshare() should test the following:

  1) Valid flags: Test to check that clone flags for signal and
     signal handlers, for which unsharing is not implemented
     yet, return -EINVAL.

  2) Missing/implied flags: Test to make sure that if unsharing
     namespace without specifying unsharing of filesystem, correctly
     unshares both namespace and filesystem information.

  3) For each of the four (namespace, filesystem, files and vm)
     supported unsharing, verify that the system call correctly
     unshares the appropriate structure. Verify that unsharing
     them individually as well as in combination with each
     other works as expected.

  4) Concurrent execution: Use shared memory segments and futex on
     an address in the shm segment to synchronize execution of
     about 10 threads. Have a couple of threads execute execve,
     a couple _exit and the rest unshare with different combination
     of flags. Verify that unsharing is performed as expected and
     that there are no oops or hangs.

9) Future Work
--------------

The current implementation of unshare() does not allow unsharing of
signals and signal handlers. Signals are complex to begin with and
to unshare signals and/or signal handlers of a currently running
process is even more complex. If in the future there is a specific
need to allow unsharing of signals and/or signal handlers, it can
be incrementally added to unshare() without affecting legacy
applications using unshare().


code:
==================================
VDUSE - "vDPA Device in Userspace"
==================================

vDPA (virtio data path acceleration) device is a device that uses a
datapath which complies with the virtio specifications with vendor
specific control path. vDPA devices can be both physically located on
the hardware or emulated by software. VDUSE is a framework that makes it
possible to implement software-emulated vDPA devices in userspace. And
to make the device emulation more secure, the emulated vDPA device's
control path is handled in the kernel and only the data path is
implemented in the userspace.

Note that only virtio block device is supported by VDUSE framework now,
which can reduce security risks when the userspace process that implements
the data path is run by an unprivileged user. The support for other device
types can be added after the security issue of corresponding device driver
is clarified or fixed in the future.

Create/Destroy VDUSE devices
----------------------------

VDUSE devices are created as follows:

1. Create a new VDUSE instance with ioctl(VDUSE_CREATE_DEV) on
   /dev/vduse/control.

2. Setup each virtqueue with ioctl(VDUSE_VQ_SETUP) on /dev/vduse/$NAME.

3. Begin processing VDUSE messages from /dev/vduse/$NAME. The first
   messages will arrive while attaching the VDUSE instance to vDPA bus.

4. Send the VDPA_CMD_DEV_NEW netlink message to attach the VDUSE
   instance to vDPA bus.

VDUSE devices are destroyed as follows:

1. Send the VDPA_CMD_DEV_DEL netlink message to detach the VDUSE
   instance from vDPA bus.

2. Close the file descriptor referring to /dev/vduse/$NAME.

3. Destroy the VDUSE instance with ioctl(VDUSE_DESTROY_DEV) on
   /dev/vduse/control.

The netlink messages can be sent via vdpa tool in iproute2 or use the
below sample codes:

.. code-block:: c

	static int netlink_add_vduse(const char *name, enum vdpa_command cmd)
	{
		struct nl_sock *nlsock;
		struct nl_msg *msg;
		int famid;

		nlsock = nl_socket_alloc();
		if (!nlsock)
			return -ENOMEM;

		if (genl_connect(nlsock))
			goto free_sock;

		famid = genl_ctrl_resolve(nlsock, VDPA_GENL_NAME);
		if (famid < 0)
			goto close_sock;

		msg = nlmsg_alloc();
		if (!msg)
			goto close_sock;

		if (!genlmsg_put(msg, NL_AUTO_PORT, NL_AUTO_SEQ, famid, 0, 0, cmd, 0))
			goto nla_put_failure;

		NLA_PUT_STRING(msg, VDPA_ATTR_DEV_NAME, name);
		if (cmd == VDPA_CMD_DEV_NEW)
			NLA_PUT_STRING(msg, VDPA_ATTR_MGMTDEV_DEV_NAME, "vduse");

		if (nl_send_sync(nlsock, msg))
			goto close_sock;

		nl_close(nlsock);
		nl_socket_free(nlsock);

		return 0;
	nla_put_failure:
		nlmsg_free(msg);
	close_sock:
		nl_close(nlsock);
	free_sock:
		nl_socket_free(nlsock);
		return -1;
	}

How VDUSE works
---------------

As mentioned above, a VDUSE device is created by ioctl(VDUSE_CREATE_DEV) on
/dev/vduse/control. With this ioctl, userspace can specify some basic configuration
such as device name (uniquely identify a VDUSE device), virtio features, virtio
configuration space, the number of virtqueues and so on for this emulated device.
Then a char device interface (/dev/vduse/$NAME) is exported to userspace for device
emulation. Userspace can use the VDUSE_VQ_SETUP ioctl on /dev/vduse/$NAME to
add per-virtqueue configuration such as the max size of virtqueue to the device.

After the initialization, the VDUSE device can be attached to vDPA bus via
the VDPA_CMD_DEV_NEW netlink message. Userspace needs to read()/write() on
/dev/vduse/$NAME to receive/reply some control messages from/to VDUSE kernel
module as follows:

.. code-block:: c

	static int vduse_message_handler(int dev_fd)
	{
		int len;
		struct vduse_dev_request req;
		struct vduse_dev_response resp;

		len = read(dev_fd, &req, sizeof(req));
		if (len != sizeof(req))
			return -1;

		resp.request_id = req.request_id;

		switch (req.type) {

		/* handle different types of messages */

		}

		len = write(dev_fd, &resp, sizeof(resp));
		if (len != sizeof(resp))
			return -1;

		return 0;
	}

There are now three types of messages introduced by VDUSE framework:

- VDUSE_GET_VQ_STATE: Get the state for virtqueue, userspace should return
  avail index for split virtqueue or the device/driver ring wrap counters and
  the avail and used index for packed virtqueue.

- VDUSE_SET_STATUS: Set the device status, userspace should follow
  the virtio spec: https://docs.oasis-open.org/virtio/virtio/v1.1/virtio-v1.1.html
  to process this message. For example, fail to set the FEATURES_OK device
  status bit if the device can not accept the negotiated virtio features
  get from the VDUSE_DEV_GET_FEATURES ioctl.

- VDUSE_UPDATE_IOTLB: Notify userspace to update the memory mapping for specified
  IOVA range, userspace should firstly remove the old mapping, then setup the new
  mapping via the VDUSE_IOTLB_GET_FD ioctl.

After DRIVER_OK status bit is set via the VDUSE_SET_STATUS message, userspace is
able to start the dataplane processing as follows:

1. Get the specified virtqueue's information with the VDUSE_VQ_GET_INFO ioctl,
   including the size, the IOVAs of descriptor table, available ring and used ring,
   the state and the ready status.

2. Pass the above IOVAs to the VDUSE_IOTLB_GET_FD ioctl so that those IOVA regions
   can be mapped into userspace. Some sample codes is shown below:

.. code-block:: c

	static int perm_to_prot(uint8_t perm)
	{
		int prot = 0;

		switch (perm) {
		case VDUSE_ACCESS_WO:
			prot |= PROT_WRITE;
			break;
		case VDUSE_ACCESS_RO:
			prot |= PROT_READ;
			break;
		case VDUSE_ACCESS_RW:
			prot |= PROT_READ | PROT_WRITE;
			break;
		}

		return prot;
	}

	static void *iova_to_va(int dev_fd, uint64_t iova, uint64_t *len)
	{
		int fd;
		void *addr;
		size_t size;
		struct vduse_iotlb_entry entry;

		entry.start = iova;
		entry.last = iova;

		/*
		 * Find the first IOVA region that overlaps with the specified
		 * range [start, last] and return the corresponding file descriptor.
		 */
		fd = ioctl(dev_fd, VDUSE_IOTLB_GET_FD, &entry);
		if (fd < 0)
			return NULL;

		size = entry.last - entry.start + 1;
		*len = entry.last - iova + 1;
		addr = mmap(0, size, perm_to_prot(entry.perm), MAP_SHARED,
			    fd, entry.offset);
		close(fd);
		if (addr == MAP_FAILED)
			return NULL;

		/*
		 * Using some data structures such as linked list to store
		 * the iotlb mapping. The munmap(2) should be called for the
		 * cached mapping when the corresponding VDUSE_UPDATE_IOTLB
		 * message is received or the device is reset.
		 */

		return addr + iova - entry.start;
	}

3. Setup the kick eventfd for the specified virtqueues with the VDUSE_VQ_SETUP_KICKFD
   ioctl. The kick eventfd is used by VDUSE kernel module to notify userspace to
   consume the available ring. This is optional since userspace can choose to poll the
   available ring instead.

4. Listen to the kick eventfd (optional) and consume the available ring. The buffer
   described by the descriptors in the descriptor table should be also mapped into
   userspace via the VDUSE_IOTLB_GET_FD ioctl before accessing.

5. Inject an interrupt for specific virtqueue with the VDUSE_INJECT_VQ_IRQ ioctl
   after the used ring is filled.

For more details on the uAPI, please see include/uapi/linux/vduse.h.


code:
.. SPDX-License-Identifier: GPL-2.0

===============
ACRN CPUID bits
===============

A guest VM running on an ACRN hypervisor can check some of its features using
CPUID.

ACRN cpuid functions are:

function: 0x40000000

returns::

   eax = 0x40000010
   ebx = 0x4e524341
   ecx = 0x4e524341
   edx = 0x4e524341

Note that this value in ebx, ecx and edx corresponds to the string
"ACRNACRNACRN". The value in eax corresponds to the maximum cpuid function
present in this leaf, and will be updated if more functions are added in the
future.

function: define ACRN_CPUID_FEATURES (0x40000001)

returns::

          ebx, ecx, edx
          eax = an OR'ed group of (1 << flag)

where ``flag`` is defined as below:

================================= =========== ================================
flag                              value       meaning
================================= =========== ================================
ACRN_FEATURE_PRIVILEGED_VM        0           guest VM is a privileged VM
================================= =========== ================================

function: 0x40000010

returns::

          ebx, ecx, edx
          eax = (Virtual) TSC frequency in kHz.


code:
.. SPDX-License-Identifier: GPL-2.0

===============
ACRN Hypervisor
===============

.. toctree::
   :maxdepth: 1

   introduction
   io-request
   cpuid


code:
.. SPDX-License-Identifier: GPL-2.0

ACRN Hypervisor Introduction
============================

The ACRN Hypervisor is a Type 1 hypervisor, running directly on bare-metal
hardware. It has a privileged management VM, called Service VM, to manage User
VMs and do I/O emulation.

ACRN userspace is an application running in the Service VM that emulates
devices for a User VM based on command line configurations. ACRN Hypervisor
Service Module (HSM) is a kernel module in the Service VM which provides
hypervisor services to the ACRN userspace.

Below figure shows the architecture.

::

                Service VM                    User VM
      +----------------------------+  |  +------------------+
      |        +--------------+    |  |  |                  |
      |        |ACRN userspace|    |  |  |                  |
      |        +--------------+    |  |  |                  |
      |-----------------ioctl------|  |  |                  |   ...
      |kernel space   +----------+ |  |  |                  |
      |               |   HSM    | |  |  | Drivers          |
      |               +----------+ |  |  |                  |
      +--------------------|-------+  |  +------------------+
  +---------------------hypercall----------------------------------------+
  |                         ACRN Hypervisor                              |
  +----------------------------------------------------------------------+
  |                          Hardware                                    |
  +----------------------------------------------------------------------+

ACRN userspace allocates memory for the User VM, configures and initializes the
devices used by the User VM, loads the virtual bootloader, initializes the
virtual CPU state and handles I/O request accesses from the User VM. It uses
ioctls to communicate with the HSM. HSM implements hypervisor services by
interacting with the ACRN Hypervisor via hypercalls. HSM exports a char device
interface (/dev/acrn_hsm) to userspace.

The ACRN hypervisor is open for contribution from anyone. The source repo is
available at https://github.com/projectacrn/acrn-hypervisor.


code:
.. SPDX-License-Identifier: GPL-2.0

I/O request handling
====================

An I/O request of a User VM, which is constructed by the hypervisor, is
distributed by the ACRN Hypervisor Service Module to an I/O client
corresponding to the address range of the I/O request. Details of I/O request
handling are described in the following sections.

1. I/O request
--------------

For each User VM, there is a shared 4-KByte memory region used for I/O requests
communication between the hypervisor and Service VM. An I/O request is a
256-byte structure buffer, which is 'struct acrn_io_request', that is filled by
an I/O handler of the hypervisor when a trapped I/O access happens in a User
VM. ACRN userspace in the Service VM first allocates a 4-KByte page and passes
the GPA (Guest Physical Address) of the buffer to the hypervisor. The buffer is
used as an array of 16 I/O request slots with each I/O request slot being 256
bytes. This array is indexed by vCPU ID.

2. I/O clients
--------------

An I/O client is responsible for handling User VM I/O requests whose accessed
GPA falls in a certain range. Multiple I/O clients can be associated with each
User VM. There is a special client associated with each User VM, called the
default client, that handles all I/O requests that do not fit into the range of
any other clients. The ACRN userspace acts as the default client for each User
VM.

Below illustration shows the relationship between I/O requests shared buffer,
I/O requests and I/O clients.

::

     +------------------------------------------------------+
     |                                       Service VM     |
     |+--------------------------------------------------+  |
     ||      +----------------------------------------+  |  |
     ||      | shared page            ACRN userspace  |  |  |
     ||      |    +-----------------+  +------------+ |  |  |
     ||   +----+->| acrn_io_request |<-+  default   | |  |  |
     ||   |  | |  +-----------------+  | I/O client | |  |  |
     ||   |  | |  |       ...       |  +------------+ |  |  |
     ||   |  | |  +-----------------+                 |  |  |
     ||   |  +-|--------------------------------------+  |  |
     ||---|----|-----------------------------------------|  |
     ||   |    |                             kernel      |  |
     ||   |    |            +----------------------+     |  |
     ||   |    |            | +-------------+  HSM |     |  |
     ||   |    +--------------+             |      |     |  |
     ||   |                 | | I/O clients |      |     |  |
     ||   |                 | |             |      |     |  |
     ||   |                 | +-------------+      |     |  |
     ||   |                 +----------------------+     |  |
     |+---|----------------------------------------------+  |
     +----|-------------------------------------------------+
          |
     +----|-------------------------------------------------+
     |  +-+-----------+                                     |
     |  | I/O handler |              ACRN Hypervisor        |
     |  +-------------+                                     |
     +------------------------------------------------------+

3. I/O request state transition
-------------------------------

The state transitions of an ACRN I/O request are as follows.

::

   FREE -> PENDING -> PROCESSING -> COMPLETE -> FREE -> ...

- FREE: this I/O request slot is empty
- PENDING: a valid I/O request is pending in this slot
- PROCESSING: the I/O request is being processed
- COMPLETE: the I/O request has been processed

An I/O request in COMPLETE or FREE state is owned by the hypervisor. HSM and
ACRN userspace are in charge of processing the others.

4. Processing flow of I/O requests
----------------------------------

a. The I/O handler of the hypervisor will fill an I/O request with PENDING
   state when a trapped I/O access happens in a User VM.
b. The hypervisor makes an upcall, which is a notification interrupt, to
   the Service VM.
c. The upcall handler schedules a worker to dispatch I/O requests.
d. The worker looks for the PENDING I/O requests, assigns them to different
   registered clients based on the address of the I/O accesses, updates
   their state to PROCESSING, and notifies the corresponding client to handle.
e. The notified client handles the assigned I/O requests.
f. The HSM updates I/O requests states to COMPLETE and notifies the hypervisor
   of the completion via hypercalls.


code:
.. SPDX-License-Identifier: GPL-2.0

===================================================================
The Definitive SEV Guest API Documentation
===================================================================

1. General description
======================

The SEV API is a set of ioctls that are used by the guest or hypervisor
to get or set a certain aspect of the SEV virtual machine. The ioctls belong
to the following classes:

 - Hypervisor ioctls: These query and set global attributes which affect the
   whole SEV firmware.  These ioctl are used by platform provisioning tools.

 - Guest ioctls: These query and set attributes of the SEV virtual machine.

2. API description
==================

This section describes ioctls that is used for querying the SEV guest report
from the SEV firmware. For each ioctl, the following information is provided
along with a description:

  Technology:
      which SEV technology provides this ioctl. SEV, SEV-ES, SEV-SNP or all.

  Type:
      hypervisor or guest. The ioctl can be used inside the guest or the
      hypervisor.

  Parameters:
      what parameters are accepted by the ioctl.

  Returns:
      the return value.  General error numbers (-ENOMEM, -EINVAL)
      are not detailed, but errors with specific meanings are.

The guest ioctl should be issued on a file descriptor of the /dev/sev-guest device.
The ioctl accepts struct snp_user_guest_request. The input and output structure is
specified through the req_data and resp_data field respectively. If the ioctl fails
to execute due to a firmware error, then fw_err code will be set otherwise the
fw_err will be set to 0x00000000000000ff.

The firmware checks that the message sequence counter is one greater than
the guests message sequence counter. If guest driver fails to increment message
counter (e.g. counter overflow), then -EIO will be returned.

::

        struct snp_guest_request_ioctl {
                /* Message version number */
                __u32 msg_version;

                /* Request and response structure address */
                __u64 req_data;
                __u64 resp_data;

                /* firmware error code on failure (see psp-sev.h) */
                __u64 fw_err;
        };

2.1 SNP_GET_REPORT
------------------

:Technology: sev-snp
:Type: guest ioctl
:Parameters (in): struct snp_report_req
:Returns (out): struct snp_report_resp on success, -negative on error

The SNP_GET_REPORT ioctl can be used to query the attestation report from the
SEV-SNP firmware. The ioctl uses the SNP_GUEST_REQUEST (MSG_REPORT_REQ) command
provided by the SEV-SNP firmware to query the attestation report.

On success, the snp_report_resp.data will contains the report. The report
contain the format described in the SEV-SNP specification. See the SEV-SNP
specification for further details.

2.2 SNP_GET_DERIVED_KEY
-----------------------
:Technology: sev-snp
:Type: guest ioctl
:Parameters (in): struct snp_derived_key_req
:Returns (out): struct snp_derived_key_resp on success, -negative on error

The SNP_GET_DERIVED_KEY ioctl can be used to get a key derive from a root key.
The derived key can be used by the guest for any purpose, such as sealing keys
or communicating with external entities.

The ioctl uses the SNP_GUEST_REQUEST (MSG_KEY_REQ) command provided by the
SEV-SNP firmware to derive the key. See SEV-SNP specification for further details
on the various fields passed in the key derivation request.

On success, the snp_derived_key_resp.data contains the derived key value. See
the SEV-SNP specification for further details.

2.3 SNP_GET_EXT_REPORT
----------------------
:Technology: sev-snp
:Type: guest ioctl
:Parameters (in/out): struct snp_ext_report_req
:Returns (out): struct snp_report_resp on success, -negative on error

The SNP_GET_EXT_REPORT ioctl is similar to the SNP_GET_REPORT. The difference is
related to the additional certificate data that is returned with the report.
The certificate data returned is being provided by the hypervisor through the
SNP_SET_EXT_CONFIG.

The ioctl uses the SNP_GUEST_REQUEST (MSG_REPORT_REQ) command provided by the SEV-SNP
firmware to get the attestation report.

On success, the snp_ext_report_resp.data will contain the attestation report
and snp_ext_report_req.certs_address will contain the certificate blob. If the
length of the blob is smaller than expected then snp_ext_report_req.certs_len will
be updated with the expected value.

See GHCB specification for further detail on how to parse the certificate blob.

3. SEV-SNP CPUID Enforcement
============================

SEV-SNP guests can access a special page that contains a table of CPUID values
that have been validated by the PSP as part of the SNP_LAUNCH_UPDATE firmware
command. It provides the following assurances regarding the validity of CPUID
values:

 - Its address is obtained via bootloader/firmware (via CC blob), and those
   binaries will be measured as part of the SEV-SNP attestation report.
 - Its initial state will be encrypted/pvalidated, so attempts to modify
   it during run-time will result in garbage being written, or #VC exceptions
   being generated due to changes in validation state if the hypervisor tries
   to swap the backing page.
 - Attempts to bypass PSP checks by the hypervisor by using a normal page, or
   a non-CPUID encrypted page will change the measurement provided by the
   SEV-SNP attestation report.
 - The CPUID page contents are *not* measured, but attempts to modify the
   expected contents of a CPUID page as part of guest initialization will be
   gated by the PSP CPUID enforcement policy checks performed on the page
   during SNP_LAUNCH_UPDATE, and noticeable later if the guest owner
   implements their own checks of the CPUID values.

It is important to note that this last assurance is only useful if the kernel
has taken care to make use of the SEV-SNP CPUID throughout all stages of boot.
Otherwise, guest owner attestation provides no assurance that the kernel wasn't
fed incorrect values at some point during boot.

Reference
---------

SEV-SNP and GHCB specification: developer.amd.com/sev

The driver is based on SEV-SNP firmware spec 0.9 and GHCB spec version 2.0.


code:
.. SPDX-License-Identifier: GPL-2.0

===================================================================
TDX Guest API Documentation
===================================================================

1. General description
======================

The TDX guest driver exposes IOCTL interfaces via the /dev/tdx-guest misc
device to allow userspace to get certain TDX guest-specific details.

2. API description
==================

In this section, for each supported IOCTL, the following information is
provided along with a generic description.

:Input parameters: Parameters passed to the IOCTL and related details.
:Output: Details about output data and return value (with details about
         the non common error values).

2.1 TDX_CMD_GET_REPORT0
-----------------------

:Input parameters: struct tdx_report_req
:Output: Upon successful execution, TDREPORT data is copied to
         tdx_report_req.tdreport and return 0. Return -EINVAL for invalid
         operands, -EIO on TDCALL failure or standard error number on other
         common failures.

The TDX_CMD_GET_REPORT0 IOCTL can be used by the attestation software to get
the TDREPORT0 (a.k.a. TDREPORT subtype 0) from the TDX module using
TDCALL[TDG.MR.REPORT].

A subtype index is added at the end of this IOCTL CMD to uniquely identify the
subtype-specific TDREPORT request. Although the subtype option is mentioned in
the TDX Module v1.0 specification, section titled "TDG.MR.REPORT", it is not
currently used, and it expects this value to be 0. So to keep the IOCTL
implementation simple, the subtype option was not included as part of the input
ABI. However, in the future, if the TDX Module supports more than one subtype,
a new IOCTL CMD will be created to handle it. To keep the IOCTL naming
consistent, a subtype index is added as part of the IOCTL CMD.

Reference
---------

TDX reference material is collected here:

https://www.intel.com/content/www/us/en/developer/articles/technical/intel-trust-domain-extensions.html

The driver is based on TDX module specification v1.0 and TDX GHCI specification v1.0.


code:
==================
Guest halt polling
==================

The cpuidle_haltpoll driver, with the haltpoll governor, allows
the guest vcpus to poll for a specified amount of time before
halting.

This provides the following benefits to host side polling:

	1) The POLL flag is set while polling is performed, which allows
	   a remote vCPU to avoid sending an IPI (and the associated
	   cost of handling the IPI) when performing a wakeup.

	2) The VM-exit cost can be avoided.

The downside of guest side polling is that polling is performed
even with other runnable tasks in the host.

The basic logic as follows: A global value, guest_halt_poll_ns,
is configured by the user, indicating the maximum amount of
time polling is allowed. This value is fixed.

Each vcpu has an adjustable guest_halt_poll_ns
("per-cpu guest_halt_poll_ns"), which is adjusted by the algorithm
in response to events (explained below).

Module Parameters
=================

The haltpoll governor has 5 tunable module parameters:

1) guest_halt_poll_ns:

Maximum amount of time, in nanoseconds, that polling is
performed before halting.

Default: 200000

2) guest_halt_poll_shrink:

Division factor used to shrink per-cpu guest_halt_poll_ns when
wakeup event occurs after the global guest_halt_poll_ns.

Default: 2

3) guest_halt_poll_grow:

Multiplication factor used to grow per-cpu guest_halt_poll_ns
when event occurs after per-cpu guest_halt_poll_ns
but before global guest_halt_poll_ns.

Default: 2

4) guest_halt_poll_grow_start:

The per-cpu guest_halt_poll_ns eventually reaches zero
in case of an idle system. This value sets the initial
per-cpu guest_halt_poll_ns when growing. This can
be increased from 10000, to avoid misses during the initial
growth stage:

10k, 20k, 40k, ... (example assumes guest_halt_poll_grow=2).

Default: 50000

5) guest_halt_poll_allow_shrink:

Bool parameter which allows shrinking. Set to N
to avoid it (per-cpu guest_halt_poll_ns will remain
high once achieves global guest_halt_poll_ns value).

Default: Y

The module parameters can be set from the debugfs files in::

	/sys/module/haltpoll/parameters/

Further Notes
=============

- Care should be taken when setting the guest_halt_poll_ns parameter as a
  large value has the potential to drive the cpu usage to 100% on a machine
  which would be almost entirely idle otherwise.


code:
.. SPDX-License-Identifier: GPL-2.0

Clocks and Timers
=================

arm64
-----
On arm64, Hyper-V virtualizes the ARMv8 architectural system counter
and timer. Guest VMs use this virtualized hardware as the Linux
clocksource and clockevents via the standard arm_arch_timer.c
driver, just as they would on bare metal. Linux vDSO support for the
architectural system counter is functional in guest VMs on Hyper-V.
While Hyper-V also provides a synthetic system clock and four synthetic
per-CPU timers as described in the TLFS, they are not used by the
Linux kernel in a Hyper-V guest on arm64.  However, older versions
of Hyper-V for arm64 only partially virtualize the ARMv8
architectural timer, such that the timer does not generate
interrupts in the VM. Because of this limitation, running current
Linux kernel versions on these older Hyper-V versions requires an
out-of-tree patch to use the Hyper-V synthetic clocks/timers instead.

x86/x64
-------
On x86/x64, Hyper-V provides guest VMs with a synthetic system clock
and four synthetic per-CPU timers as described in the TLFS. Hyper-V
also provides access to the virtualized TSC via the RDTSC and
related instructions. These TSC instructions do not trap to
the hypervisor and so provide excellent performance in a VM.
Hyper-V performs TSC calibration, and provides the TSC frequency
to the guest VM via a synthetic MSR.  Hyper-V initialization code
in Linux reads this MSR to get the frequency, so it skips TSC
calibration and sets tsc_reliable. Hyper-V provides virtualized
versions of the PIT (in Hyper-V  Generation 1 VMs only), local
APIC timer, and RTC. Hyper-V does not provide a virtualized HPET in
guest VMs.

The Hyper-V synthetic system clock can be read via a synthetic MSR,
but this access traps to the hypervisor. As a faster alternative,
the guest can configure a memory page to be shared between the guest
and the hypervisor.  Hyper-V populates this memory page with a
64-bit scale value and offset value. To read the synthetic clock
value, the guest reads the TSC and then applies the scale and offset
as described in the Hyper-V TLFS. The resulting value advances
at a constant 10 MHz frequency. In the case of a live migration
to a host with a different TSC frequency, Hyper-V adjusts the
scale and offset values in the shared page so that the 10 MHz
frequency is maintained.

Starting with Windows Server 2022 Hyper-V, Hyper-V uses hardware
support for TSC frequency scaling to enable live migration of VMs
across Hyper-V hosts where the TSC frequency may be different.
When a Linux guest detects that this Hyper-V functionality is
available, it prefers to use Linux's standard TSC-based clocksource.
Otherwise, it uses the clocksource for the Hyper-V synthetic system
clock implemented via the shared page (identified as
"hyperv_clocksource_tsc_page").

The Hyper-V synthetic system clock is available to user space via
vDSO, and gettimeofday() and related system calls can execute
entirely in user space.  The vDSO is implemented by mapping the
shared page with scale and offset values into user space.  User
space code performs the same algorithm of reading the TSC and
appying the scale and offset to get the constant 10 MHz clock.

Linux clockevents are based on Hyper-V synthetic timer 0. While
Hyper-V offers 4 synthetic timers for each CPU, Linux only uses
timer 0. Interrupts from stimer0 are recorded on the "HVS" line in
/proc/interrupts.  Clockevents based on the virtualized PIT and
local APIC timer also work, but the Hyper-V synthetic timer is
preferred.

The driver for the Hyper-V synthetic system clock and timers is
drivers/clocksource/hyperv_timer.c.


code:
.. SPDX-License-Identifier: GPL-2.0

======================
Hyper-V Enlightenments
======================

.. toctree::
   :maxdepth: 1

   overview
   vmbus
   clocks


code:
.. SPDX-License-Identifier: GPL-2.0

Overview
========
The Linux kernel contains a variety of code for running as a fully
enlightened guest on Microsoft's Hyper-V hypervisor.  Hyper-V
consists primarily of a bare-metal hypervisor plus a virtual machine
management service running in the parent partition (roughly
equivalent to KVM and QEMU, for example).  Guest VMs run in child
partitions.  In this documentation, references to Hyper-V usually
encompass both the hypervisor and the VMM service without making a
distinction about which functionality is provided by which
component.

Hyper-V runs on x86/x64 and arm64 architectures, and Linux guests
are supported on both.  The functionality and behavior of Hyper-V is
generally the same on both architectures unless noted otherwise.

Linux Guest Communication with Hyper-V
--------------------------------------
Linux guests communicate with Hyper-V in four different ways:

* Implicit traps: As defined by the x86/x64 or arm64 architecture,
  some guest actions trap to Hyper-V.  Hyper-V emulates the action and
  returns control to the guest.  This behavior is generally invisible
  to the Linux kernel.

* Explicit hypercalls: Linux makes an explicit function call to
  Hyper-V, passing parameters.  Hyper-V performs the requested action
  and returns control to the caller.  Parameters are passed in
  processor registers or in memory shared between the Linux guest and
  Hyper-V.   On x86/x64, hypercalls use a Hyper-V specific calling
  sequence.  On arm64, hypercalls use the ARM standard SMCCC calling
  sequence.

* Synthetic register access: Hyper-V implements a variety of
  synthetic registers.  On x86/x64 these registers appear as MSRs in
  the guest, and the Linux kernel can read or write these MSRs using
  the normal mechanisms defined by the x86/x64 architecture.  On
  arm64, these synthetic registers must be accessed using explicit
  hypercalls.

* VMbus: VMbus is a higher-level software construct that is built on
  the other 3 mechanisms.  It is a message passing interface between
  the Hyper-V host and the Linux guest.  It uses memory that is shared
  between Hyper-V and the guest, along with various signaling
  mechanisms.

The first three communication mechanisms are documented in the
`Hyper-V Top Level Functional Spec (TLFS)`_.  The TLFS describes
general Hyper-V functionality and provides details on the hypercalls
and synthetic registers.  The TLFS is currently written for the
x86/x64 architecture only.

.. _Hyper-V Top Level Functional Spec (TLFS): https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/tlfs/tlfs

VMbus is not documented.  This documentation provides a high-level
overview of VMbus and how it works, but the details can be discerned
only from the code.

Sharing Memory
--------------
Many aspects are communication between Hyper-V and Linux are based
on sharing memory.  Such sharing is generally accomplished as
follows:

* Linux allocates memory from its physical address space using
  standard Linux mechanisms.

* Linux tells Hyper-V the guest physical address (GPA) of the
  allocated memory.  Many shared areas are kept to 1 page so that a
  single GPA is sufficient.   Larger shared areas require a list of
  GPAs, which usually do not need to be contiguous in the guest
  physical address space.  How Hyper-V is told about the GPA or list
  of GPAs varies.  In some cases, a single GPA is written to a
  synthetic register.  In other cases, a GPA or list of GPAs is sent
  in a VMbus message.

* Hyper-V translates the GPAs into "real" physical memory addresses,
  and creates a virtual mapping that it can use to access the memory.

* Linux can later revoke sharing it has previously established by
  telling Hyper-V to set the shared GPA to zero.

Hyper-V operates with a page size of 4 Kbytes. GPAs communicated to
Hyper-V may be in the form of page numbers, and always describe a
range of 4 Kbytes.  Since the Linux guest page size on x86/x64 is
also 4 Kbytes, the mapping from guest page to Hyper-V page is 1-to-1.
On arm64, Hyper-V supports guests with 4/16/64 Kbyte pages as
defined by the arm64 architecture.   If Linux is using 16 or 64
Kbyte pages, Linux code must be careful to communicate with Hyper-V
only in terms of 4 Kbyte pages.  HV_HYP_PAGE_SIZE and related macros
are used in code that communicates with Hyper-V so that it works
correctly in all configurations.

As described in the TLFS, a few memory pages shared between Hyper-V
and the Linux guest are "overlay" pages.  With overlay pages, Linux
uses the usual approach of allocating guest memory and telling
Hyper-V the GPA of the allocated memory.  But Hyper-V then replaces
that physical memory page with a page it has allocated, and the
original physical memory page is no longer accessible in the guest
VM.  Linux may access the memory normally as if it were the memory
that it originally allocated.  The "overlay" behavior is visible
only because the contents of the page (as seen by Linux) change at
the time that Linux originally establishes the sharing and the
overlay page is inserted.  Similarly, the contents change if Linux
revokes the sharing, in which case Hyper-V removes the overlay page,
and the guest page originally allocated by Linux becomes visible
again.

Before Linux does a kexec to a kdump kernel or any other kernel,
memory shared with Hyper-V should be revoked.  Hyper-V could modify
a shared page or remove an overlay page after the new kernel is
using the page for a different purpose, corrupting the new kernel.
Hyper-V does not provide a single "set everything" operation to
guest VMs, so Linux code must individually revoke all sharing before
doing kexec.   See hv_kexec_handler() and hv_crash_handler().  But
the crash/panic path still has holes in cleanup because some shared
pages are set using per-CPU synthetic registers and there's no
mechanism to revoke the shared pages for CPUs other than the CPU
running the panic path.

CPU Management
--------------
Hyper-V does not have a ability to hot-add or hot-remove a CPU
from a running VM.  However, Windows Server 2019 Hyper-V and
earlier versions may provide guests with ACPI tables that indicate
more CPUs than are actually present in the VM.  As is normal, Linux
treats these additional CPUs as potential hot-add CPUs, and reports
them as such even though Hyper-V will never actually hot-add them.
Starting in Windows Server 2022 Hyper-V, the ACPI tables reflect
only the CPUs actually present in the VM, so Linux does not report
any hot-add CPUs.

A Linux guest CPU may be taken offline using the normal Linux
mechanisms, provided no VMbus channel interrupts are assigned to
the CPU.  See the section on VMbus Interrupts for more details
on how VMbus channel interrupts can be re-assigned to permit
taking a CPU offline.

32-bit and 64-bit
-----------------
On x86/x64, Hyper-V supports 32-bit and 64-bit guests, and Linux
will build and run in either version. While the 32-bit version is
expected to work, it is used rarely and may suffer from undetected
regressions.

On arm64, Hyper-V supports only 64-bit guests.

Endian-ness
-----------
All communication between Hyper-V and guest VMs uses Little-Endian
format on both x86/x64 and arm64.  Big-endian format on arm64 is not
supported by Hyper-V, and Linux code does not use endian-ness macros
when accessing data shared with Hyper-V.

Versioning
----------
Current Linux kernels operate correctly with older versions of
Hyper-V back to Windows Server 2012 Hyper-V. Support for running
on the original Hyper-V release in Windows Server 2008/2008 R2
has been removed.

A Linux guest on Hyper-V outputs in dmesg the version of Hyper-V
it is running on.  This version is in the form of a Windows build
number and is for display purposes only. Linux code does not
test this version number at runtime to determine available features
and functionality. Hyper-V indicates feature/function availability
via flags in synthetic MSRs that Hyper-V provides to the guest,
and the guest code tests these flags.

VMbus has its own protocol version that is negotiated during the
initial VMbus connection from the guest to Hyper-V. This version
number is also output to dmesg during boot.  This version number
is checked in a few places in the code to determine if specific
functionality is present.

Furthermore, each synthetic device on VMbus also has a protocol
version that is separate from the VMbus protocol version. Device
drivers for these synthetic devices typically negotiate the device
protocol version, and may test that protocol version to determine
if specific device functionality is present.

Code Packaging
--------------
Hyper-V related code appears in the Linux kernel code tree in three
main areas:

1. drivers/hv

2. arch/x86/hyperv and arch/arm64/hyperv

3. individual device driver areas such as drivers/scsi, drivers/net,
   drivers/clocksource, etc.

A few miscellaneous files appear elsewhere. See the full list under
"Hyper-V/Azure CORE AND DRIVERS" and "DRM DRIVER FOR HYPERV
SYNTHETIC VIDEO DEVICE" in the MAINTAINERS file.

The code in #1 and #2 is built only when CONFIG_HYPERV is set.
Similarly, the code for most Hyper-V related drivers is built only
when CONFIG_HYPERV is set.

Most Hyper-V related code in #1 and #3 can be built as a module.
The architecture specific code in #2 must be built-in.  Also,
drivers/hv/hv_common.c is low-level code that is common across
architectures and must be built-in.


code:
.. SPDX-License-Identifier: GPL-2.0

VMbus
=====
VMbus is a software construct provided by Hyper-V to guest VMs.  It
consists of a control path and common facilities used by synthetic
devices that Hyper-V presents to guest VMs.   The control path is
used to offer synthetic devices to the guest VM and, in some cases,
to rescind those devices.   The common facilities include software
channels for communicating between the device driver in the guest VM
and the synthetic device implementation that is part of Hyper-V, and
signaling primitives to allow Hyper-V and the guest to interrupt
each other.

VMbus is modeled in Linux as a bus, with the expected /sys/bus/vmbus
entry in a running Linux guest.  The VMbus driver (drivers/hv/vmbus_drv.c)
establishes the VMbus control path with the Hyper-V host, then
registers itself as a Linux bus driver.  It implements the standard
bus functions for adding and removing devices to/from the bus.

Most synthetic devices offered by Hyper-V have a corresponding Linux
device driver.  These devices include:

* SCSI controller
* NIC
* Graphics frame buffer
* Keyboard
* Mouse
* PCI device pass-thru
* Heartbeat
* Time Sync
* Shutdown
* Memory balloon
* Key/Value Pair (KVP) exchange with Hyper-V
* Hyper-V online backup (a.k.a. VSS)

Guest VMs may have multiple instances of the synthetic SCSI
controller, synthetic NIC, and PCI pass-thru devices.  Other
synthetic devices are limited to a single instance per VM.  Not
listed above are a small number of synthetic devices offered by
Hyper-V that are used only by Windows guests and for which Linux
does not have a driver.

Hyper-V uses the terms "VSP" and "VSC" in describing synthetic
devices.  "VSP" refers to the Hyper-V code that implements a
particular synthetic device, while "VSC" refers to the driver for
the device in the guest VM.  For example, the Linux driver for the
synthetic NIC is referred to as "netvsc" and the Linux driver for
the synthetic SCSI controller is "storvsc".  These drivers contain
functions with names like "storvsc_connect_to_vsp".

VMbus channels
--------------
An instance of a synthetic device uses VMbus channels to communicate
between the VSP and the VSC.  Channels are bi-directional and used
for passing messages.   Most synthetic devices use a single channel,
but the synthetic SCSI controller and synthetic NIC may use multiple
channels to achieve higher performance and greater parallelism.

Each channel consists of two ring buffers.  These are classic ring
buffers from a university data structures textbook.  If the read
and writes pointers are equal, the ring buffer is considered to be
empty, so a full ring buffer always has at least one byte unused.
The "in" ring buffer is for messages from the Hyper-V host to the
guest, and the "out" ring buffer is for messages from the guest to
the Hyper-V host.  In Linux, the "in" and "out" designations are as
viewed by the guest side.  The ring buffers are memory that is
shared between the guest and the host, and they follow the standard
paradigm where the memory is allocated by the guest, with the list
of GPAs that make up the ring buffer communicated to the host.  Each
ring buffer consists of a header page (4 Kbytes) with the read and
write indices and some control flags, followed by the memory for the
actual ring.  The size of the ring is determined by the VSC in the
guest and is specific to each synthetic device.   The list of GPAs
making up the ring is communicated to the Hyper-V host over the
VMbus control path as a GPA Descriptor List (GPADL).  See function
vmbus_establish_gpadl().

Each ring buffer is mapped into contiguous Linux kernel virtual
space in three parts:  1) the 4 Kbyte header page, 2) the memory
that makes up the ring itself, and 3) a second mapping of the memory
that makes up the ring itself.  Because (2) and (3) are contiguous
in kernel virtual space, the code that copies data to and from the
ring buffer need not be concerned with ring buffer wrap-around.
Once a copy operation has completed, the read or write index may
need to be reset to point back into the first mapping, but the
actual data copy does not need to be broken into two parts.  This
approach also allows complex data structures to be easily accessed
directly in the ring without handling wrap-around.

On arm64 with page sizes > 4 Kbytes, the header page must still be
passed to Hyper-V as a 4 Kbyte area.  But the memory for the actual
ring must be aligned to PAGE_SIZE and have a size that is a multiple
of PAGE_SIZE so that the duplicate mapping trick can be done.  Hence
a portion of the header page is unused and not communicated to
Hyper-V.  This case is handled by vmbus_establish_gpadl().

Hyper-V enforces a limit on the aggregate amount of guest memory
that can be shared with the host via GPADLs.  This limit ensures
that a rogue guest can't force the consumption of excessive host
resources.  For Windows Server 2019 and later, this limit is
approximately 1280 Mbytes.  For versions prior to Windows Server
2019, the limit is approximately 384 Mbytes.

VMbus messages
--------------
All VMbus messages have a standard header that includes the message
length, the offset of the message payload, some flags, and a
transactionID.  The portion of the message after the header is
unique to each VSP/VSC pair.

Messages follow one of two patterns:

* Unidirectional:  Either side sends a message and does not
  expect a response message
* Request/response:  One side (usually the guest) sends a message
  and expects a response

The transactionID (a.k.a. "requestID") is for matching requests &
responses.  Some synthetic devices allow multiple requests to be in-
flight simultaneously, so the guest specifies a transactionID when
sending a request.  Hyper-V sends back the same transactionID in the
matching response.

Messages passed between the VSP and VSC are control messages.  For
example, a message sent from the storvsc driver might be "execute
this SCSI command".   If a message also implies some data transfer
between the guest and the Hyper-V host, the actual data to be
transferred may be embedded with the control message, or it may be
specified as a separate data buffer that the Hyper-V host will
access as a DMA operation.  The former case is used when the size of
the data is small and the cost of copying the data to and from the
ring buffer is minimal.  For example, time sync messages from the
Hyper-V host to the guest contain the actual time value.  When the
data is larger, a separate data buffer is used.  In this case, the
control message contains a list of GPAs that describe the data
buffer.  For example, the storvsc driver uses this approach to
specify the data buffers to/from which disk I/O is done.

Three functions exist to send VMbus messages:

1. vmbus_sendpacket():  Control-only messages and messages with
   embedded data -- no GPAs
2. vmbus_sendpacket_pagebuffer(): Message with list of GPAs
   identifying data to transfer.  An offset and length is
   associated with each GPA so that multiple discontinuous areas
   of guest memory can be targeted.
3. vmbus_sendpacket_mpb_desc(): Message with list of GPAs
   identifying data to transfer.  A single offset and length is
   associated with a list of GPAs.  The GPAs must describe a
   single logical area of guest memory to be targeted.

Historically, Linux guests have trusted Hyper-V to send well-formed
and valid messages, and Linux drivers for synthetic devices did not
fully validate messages.  With the introduction of processor
technologies that fully encrypt guest memory and that allow the
guest to not trust the hypervisor (AMD SNP-SEV, Intel TDX), trusting
the Hyper-V host is no longer a valid assumption.  The drivers for
VMbus synthetic devices are being updated to fully validate any
values read from memory that is shared with Hyper-V, which includes
messages from VMbus devices.  To facilitate such validation,
messages read by the guest from the "in" ring buffer are copied to a
temporary buffer that is not shared with Hyper-V.  Validation is
performed in this temporary buffer without the risk of Hyper-V
maliciously modifying the message after it is validated but before
it is used.

VMbus interrupts
----------------
VMbus provides a mechanism for the guest to interrupt the host when
the guest has queued new messages in a ring buffer.  The host
expects that the guest will send an interrupt only when an "out"
ring buffer transitions from empty to non-empty.  If the guest sends
interrupts at other times, the host deems such interrupts to be
unnecessary.  If a guest sends an excessive number of unnecessary
interrupts, the host may throttle that guest by suspending its
execution for a few seconds to prevent a denial-of-service attack.

Similarly, the host will interrupt the guest when it sends a new
message on the VMbus control path, or when a VMbus channel "in" ring
buffer transitions from empty to non-empty.  Each CPU in the guest
may receive VMbus interrupts, so they are best modeled as per-CPU
interrupts in Linux.  This model works well on arm64 where a single
per-CPU IRQ is allocated for VMbus.  Since x86/x64 lacks support for
per-CPU IRQs, an x86 interrupt vector is statically allocated (see
HYPERVISOR_CALLBACK_VECTOR) across all CPUs and explicitly coded to
call the VMbus interrupt service routine.  These interrupts are
visible in /proc/interrupts on the "HYP" line.

The guest CPU that a VMbus channel will interrupt is selected by the
guest when the channel is created, and the host is informed of that
selection.  VMbus devices are broadly grouped into two categories:

1. "Slow" devices that need only one VMbus channel.  The devices
   (such as keyboard, mouse, heartbeat, and timesync) generate
   relatively few interrupts.  Their VMbus channels are all
   assigned to interrupt the VMBUS_CONNECT_CPU, which is always
   CPU 0.

2. "High speed" devices that may use multiple VMbus channels for
   higher parallelism and performance.  These devices include the
   synthetic SCSI controller and synthetic NIC.  Their VMbus
   channels interrupts are assigned to CPUs that are spread out
   among the available CPUs in the VM so that interrupts on
   multiple channels can be processed in parallel.

The assignment of VMbus channel interrupts to CPUs is done in the
function init_vp_index().  This assignment is done outside of the
normal Linux interrupt affinity mechanism, so the interrupts are
neither "unmanaged" nor "managed" interrupts.

The CPU that a VMbus channel will interrupt can be seen in
/sys/bus/vmbus/devices/<deviceGUID>/ channels/<channelRelID>/cpu.
When running on later versions of Hyper-V, the CPU can be changed
by writing a new value to this sysfs entry.  Because the interrupt
assignment is done outside of the normal Linux affinity mechanism,
there are no entries in /proc/irq corresponding to individual
VMbus channel interrupts.

An online CPU in a Linux guest may not be taken offline if it has
VMbus channel interrupts assigned to it.  Any such channel
interrupts must first be manually reassigned to another CPU as
described above.  When no channel interrupts are assigned to the
CPU, it can be taken offline.

When a guest CPU receives a VMbus interrupt from the host, the
function vmbus_isr() handles the interrupt.  It first checks for
channel interrupts by calling vmbus_chan_sched(), which looks at a
bitmap setup by the host to determine which channels have pending
interrupts on this CPU.  If multiple channels have pending
interrupts for this CPU, they are processed sequentially.  When all
channel interrupts have been processed, vmbus_isr() checks for and
processes any message received on the VMbus control path.

The VMbus channel interrupt handling code is designed to work
correctly even if an interrupt is received on a CPU other than the
CPU assigned to the channel.  Specifically, the code does not use
CPU-based exclusion for correctness.  In normal operation, Hyper-V
will interrupt the assigned CPU.  But when the CPU assigned to a
channel is being changed via sysfs, the guest doesn't know exactly
when Hyper-V will make the transition.  The code must work correctly
even if there is a time lag before Hyper-V starts interrupting the
new CPU.  See comments in target_cpu_store().

VMbus device creation/deletion
------------------------------
Hyper-V and the Linux guest have a separate message-passing path
that is used for synthetic device creation and deletion. This
path does not use a VMbus channel.  See vmbus_post_msg() and
vmbus_on_msg_dpc().

The first step is for the guest to connect to the generic
Hyper-V VMbus mechanism.  As part of establishing this connection,
the guest and Hyper-V agree on a VMbus protocol version they will
use.  This negotiation allows newer Linux kernels to run on older
Hyper-V versions, and vice versa.

The guest then tells Hyper-V to "send offers".  Hyper-V sends an
offer message to the guest for each synthetic device that the VM
is configured to have. Each VMbus device type has a fixed GUID
known as the "class ID", and each VMbus device instance is also
identified by a GUID. The offer message from Hyper-V contains
both GUIDs to uniquely (within the VM) identify the device.
There is one offer message for each device instance, so a VM with
two synthetic NICs will get two offers messages with the NIC
class ID. The ordering of offer messages can vary from boot-to-boot
and must not be assumed to be consistent in Linux code. Offer
messages may also arrive long after Linux has initially booted
because Hyper-V supports adding devices, such as synthetic NICs,
to running VMs. A new offer message is processed by
vmbus_process_offer(), which indirectly invokes vmbus_add_channel_work().

Upon receipt of an offer message, the guest identifies the device
type based on the class ID, and invokes the correct driver to set up
the device.  Driver/device matching is performed using the standard
Linux mechanism.

The device driver probe function opens the primary VMbus channel to
the corresponding VSP. It allocates guest memory for the channel
ring buffers and shares the ring buffer with the Hyper-V host by
giving the host a list of GPAs for the ring buffer memory.  See
vmbus_establish_gpadl().

Once the ring buffer is set up, the device driver and VSP exchange
setup messages via the primary channel.  These messages may include
negotiating the device protocol version to be used between the Linux
VSC and the VSP on the Hyper-V host.  The setup messages may also
include creating additional VMbus channels, which are somewhat
mis-named as "sub-channels" since they are functionally
equivalent to the primary channel once they are created.

Finally, the device driver may create entries in /dev as with
any device driver.

The Hyper-V host can send a "rescind" message to the guest to
remove a device that was previously offered. Linux drivers must
handle such a rescind message at any time. Rescinding a device
invokes the device driver "remove" function to cleanly shut
down the device and remove it. Once a synthetic device is
rescinded, neither Hyper-V nor Linux retains any state about
its previous existence. Such a device might be re-added later,
in which case it is treated as an entirely new device. See
vmbus_onoffer_rescind().


code:
.. SPDX-License-Identifier: GPL-2.0

============================
Linux Virtualization Support
============================

.. toctree::
   :maxdepth: 2

   kvm/index
   uml/user_mode_linux_howto_v2
   paravirt_ops
   guest-halt-polling
   ne_overview
   acrn/index
   coco/sev-guest
   coco/tdx-guest
   hyperv/index

.. only:: html and subproject

   Indices
   =======

   * :ref:`genindex`